Buffer not displaying response when gptel-stream is t using curl

yorisilo commented 11 months ago

Hi!

In the latest version of gptel-curl.el, when the gptel-stream is set to t, the buffer doesn't display the API response. This issue appears to be caused by the way the gptel-curl--stream-filter function processes the data: response chunks from the API when using curl.

The original implementation of the function parsed the data: chunks and combined the content values before passing them to the gptel-curl--stream-insert-response function. However, in the newer version, this logic was altered, causing the described issue.

To fix this problem, the function should reintroduce the logic to parse and combine the data: chunks from the API response before passing the combined content to the gptel-curl--stream-insert-response function.

The relevant section can be found here: (when-let ((http-msg (plist-get proc-info :status)) ... https://github.com/karthink/gptel/blob/63027083cd5aef95ca6601d81c03069e8da2a667/gptel-curl.el#L229-L277

Modified Code:

(defun gptel-curl--stream-filter (process output)
  (let* ((proc-info (alist-get process gptel-curl--process-alist)))
    (with-current-buffer (process-buffer process)
      ;; Insert output
      (save-excursion
        (goto-char (process-mark process))
        (insert output)
        (set-marker (process-mark process) (point)))

      ;; Find HTTP status
      (unless (plist-get proc-info :http-status)
        (save-excursion
          (goto-char (point-min))
          (when-let* (((not (= (line-end-position) (point-max))))
                      (http-msg (buffer-substring (line-beginning-position)
                                                  (line-end-position)))
                      (http-status
                       (save-match-data
                         (and (string-match "HTTP/[.0-9]+ +\\([0-9]+\\)" http-msg)
                              (match-string 1 http-msg)))))
            (plist-put proc-info :http-status http-status)
            (plist-put proc-info :status (string-trim http-msg))))
        ;; Handle read-only gptel buffer
        (when (with-current-buffer (plist-get proc-info :buffer)
                (or buffer-read-only
                    (get-char-property (plist-get proc-info :position) 'read-only)))
          (message "Buffer is read only, displaying reply in buffer \"*ChatGPT response*\"")
          (display-buffer
           (with-current-buffer (get-buffer-create "*ChatGPT response*")
             (goto-char (point-max))
             (move-marker (plist-get proc-info :position) (point) (current-buffer))
             (current-buffer))
           '((display-buffer-reuse-window
              display-buffer-pop-up-window)
             (reusable-frames . visible))))
        ;; Run pre-response hook
        (when (and (equal (plist-get proc-info :http-status) "200")
                   gptel-pre-response-hook)
          (with-current-buffer (marker-buffer (plist-get proc-info :position))
            (run-hooks 'gptel-pre-response-hook))))

      (when-let ((http-msg (plist-get proc-info :status))
                 (http-status (plist-get proc-info :http-status)))
        ;; Find data chunk(s) and run callback
        (when (equal http-status "200")
          (funcall (or (plist-get proc-info :callback)
                       #'gptel-curl--stream-insert-response)
                   (let* ((json-object-type 'plist)
                          (content-strs))
                     (condition-case nil
                         (while (re-search-forward "^data:" nil t)
                           (save-match-data
                             (unless (looking-at " *\\[DONE\\]")
                               (when-let* ((response (json-read))
                                           (delta (map-nested-elt
                                                   response '(:choices 0 :delta)))
                                           (content (plist-get delta :content)))
                                 (push content content-strs)))))
                       (error
                        (goto-char (match-beginning 0))))
                     (apply #'concat (nreverse content-strs)))
                   proc-info))))))

thx

Additionally, I'll include my init.el configuration here. Is there any other configuration required besides this?(I'm using leaf instead of use-package, but that shouldn't be an issue.)

(leaf gptel
  :ensure t
  :config
  (funcall gptel-api-key "api.openai.com" "apikey")
)

karthink commented 11 months ago

@yorisilo Your init.el configuration is fine, but calling gptel-api-key in the :config block does nothing useful. You just need

(leaf gptel :ensure t)

The new logic is unchanged from before, the JSON parsing code was just refactored to a different function. I'm guessing the problem is something else, not parsing the stream. The organization of the package changed in the last update, could you try with a clean install? If deleting and reinstalling gptel does not fix the problem, try setting gpte--debug to t and using it. You can paste the HTTP request buffer that pops up here (after removing your API key from it!)

yorisilo commented 11 months ago

@karthink Thank you for the response. I now understand the config.

Regarding .authinfo, if I wanted to use a different name other than apikey, would it be better to set it up like this?

~/.authinfo

machine api.openai.com login hogehoge password .......

init.el

(leaf gptel
  :ensure t
  :custom
  (gptel-api-key . (lambda () (gptel-api-key-from-auth-source "api.openai.com" "hogehoge"))))

I might be sidetracking a bit. Let's get back to the main topic.

I initialized the package by:

Deleting eln-cache
Removing gptel from the elpa directory
Reinstalling gptel

To cut to the chase, it still didn't work as expected... Below are the logs: (Thanks also for the note about removing the API key!)

When `(setq gptel-stream t)`

https://github.com/karthink/gptel/assets/10104705/39ccdc95-50db-4eb7-ba3f-9f3e8417c078

request

Querying ChatGPT...
("--location" "--silent" "--compressed" "--disable" "-XPOST" "-w(bc9ffaefec2c39719565927dbb16f561 . %{size_header})" "-m60" "-D-" "-d{\"model\":\"gpt-3.5-turbo\",\"messages\":[{\"role\":\"system\",\"content\":\"You are a large language model living in Emacs and a helpful assistant. Respond concisely.\"},{\"role\":\"user\",\"content\":\"hello\\n\\n\\n\\n### hello\\n\\n\\n\\n### hello\\n\\n\\n\\n### hello\\n\\n\\n\\n### hello\\n\\n\\n\\n### hello\\n\\n\\n\\n### hello\\n\\n\\n\\n### hi!\\n\\n\\n\\n### hello\\n\\n\\n\\n### hello\\n\\n\\n\\n### hi!\\n\\n\\n\\n### hello\\n\\n\\n\\n### hi!!!\"}],\"stream\":true,\"temperature\":1.0}" "-HContent-Type: application/json" "-HAuthorization: Bearer ......" "https://api.openai.com/v1/chat/completions")

response

HTTP/2 200 
date: Mon, 30 Oct 2023 15:25:49 GMT
content-type: text/event-stream
access-control-allow-origin: *
cache-control: no-cache, must-revalidate
openai-model: gpt-3.5-turbo-0613
openai-organization: yorisilo
openai-processing-ms: 170
openai-version: 2020-10-01
strict-transport-security: max-age=15724800; includeSubDomains
x-ratelimit-limit-requests: 5000
x-ratelimit-limit-tokens: 160000
x-ratelimit-remaining-requests: 4999
x-ratelimit-remaining-tokens: 159920
x-ratelimit-reset-requests: 12ms
x-ratelimit-reset-tokens: 30ms
x-request-id: 88dabe575a6e9c041a9602d213e5e960
cf-cache-status: DYNAMIC
server: cloudflare
cf-ray: 81e4ac4bfc7db006-NRT
alt-svc: h3=":443"; ma=86400

data: {"id":"chatcmpl-8FORAFestHSSLiRuJiaGz9nY6FSkC","object":"chat.completion.chunk","created":1698679548,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-8FORAFestHSSLiRuJiaGz9nY6FSkC","object":"chat.completion.chunk","created":1698679548,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

data: {"id":"chatcmpl-8FORAFestHSSLiRuJiaGz9nY6FSkC","object":"chat.completion.chunk","created":1698679548,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":"!"},"finish_reason":null}]}

data: {"id":"chatcmpl-8FORAFestHSSLiRuJiaGz9nY6FSkC","object":"chat.completion.chunk","created":1698679548,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":" How"},"finish_reason":null}]}

data: {"id":"chatcmpl-8FORAFestHSSLiRuJiaGz9nY6FSkC","object":"chat.completion.chunk","created":1698679548,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":" can"},"finish_reason":null}]}

data: {"id":"chatcmpl-8FORAFestHSSLiRuJiaGz9nY6FSkC","object":"chat.completion.chunk","created":1698679548,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":" I"},"finish_reason":null}]}

data: {"id":"chatcmpl-8FORAFestHSSLiRuJiaGz9nY6FSkC","object":"chat.completion.chunk","created":1698679548,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":" assist"},"finish_reason":null}]}

data: {"id":"chatcmpl-8FORAFestHSSLiRuJiaGz9nY6FSkC","object":"chat.completion.chunk","created":1698679548,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":" you"},"finish_reason":null}]}

data: {"id":"chatcmpl-8FORAFestHSSLiRuJiaGz9nY6FSkC","object":"chat.completion.chunk","created":1698679548,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":" today"},"finish_reason":null}]}

data: {"id":"chatcmpl-8FORAFestHSSLiRuJiaGz9nY6FSkC","object":"chat.completion.chunk","created":1698679548,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":"?"},"finish_reason":null}]}

data: {"id":"chatcmpl-8FORAFestHSSLiRuJiaGz9nY6FSkC","object":"chat.completion.chunk","created":1698679548,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

(bc9ffaefec2c39719565927dbb16f561 . 707)

When `(setq gptel-stream nil)`

https://github.com/karthink/gptel/assets/10104705/c55b1c1e-4d8b-471a-ae94-e39ce762d643

request

Querying ChatGPT...
("--location" "--silent" "--compressed" "--disable" "-XPOST" "-w(1896e58d335001a483d8fb7670d6ebcf . %{size_header})" "-m60" "-D-" "-d{\"model\":\"gpt-3.5-turbo\",\"messages\":[{\"role\":\"system\",\"content\":\"You are a large language model living in Emacs and a helpful assistant. Respond concisely.\"},{\"role\":\"user\",\"content\":\"hello\\n\\n\\n\\n### hello\\n\\n\\n\\n### hello\\n\\n\\n\\n### hello\\n\\n\\n\\n### hello\\n\\n\\n\\n### hello\\n\\n\\n\\n### hello\\n\\n\\n\\n### hi!\\n\\n\\n\\n### hello\\n\\n\\n\\n### hello\\n\\n\\n\\n### hi!\\n\\n\\n\\n### hello\\n\\n\\n\\n### hi!!!\\n\\n\\n\\n### hogehoge\\n\\n\\n\\n### hello\"},{\"role\":\"assistant\",\"content\":\"Hello! How can I assist you today?\"},{\"role\":\"user\",\"content\":\"hello!!\"},{\"role\":\"assistant\",\"content\":\"Hello! How can I assist you today?\"},{\"role\":\"user\",\"content\":\"yeah!\"}],\"stream\":false,\"temperature\":1.0}" "-HContent-Type: application/json" "-HAuthorization: Bearer ......" "https://api.openai.com/v1/chat/completions")

response

HTTP/2 200 
date: Mon, 30 Oct 2023 15:30:17 GMT
content-type: application/json
access-control-allow-origin: *
cache-control: no-cache, must-revalidate
openai-model: gpt-3.5-turbo-0613
openai-organization: yorisilo
openai-processing-ms: 736
openai-version: 2020-10-01
strict-transport-security: max-age=15724800; includeSubDomains
x-ratelimit-limit-requests: 5000
x-ratelimit-limit-tokens: 160000
x-ratelimit-remaining-requests: 4999
x-ratelimit-remaining-tokens: 159889
x-ratelimit-reset-requests: 12ms
x-ratelimit-reset-tokens: 41ms
x-request-id: a40b343fc221dd2b82952ba61570dd77
cf-cache-status: DYNAMIC
server: cloudflare
cf-ray: 81e4b2d67f253499-NRT
content-encoding: br
alt-svc: h3=":443"; ma=86400

{
  "id": "chatcmpl-8FOVV9Mkc2oTY04pNOi91nf0O79g6",
  "object": "chat.completion",
  "created": 1698679817,
  "model": "gpt-3.5-turbo-0613",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Great! How can I assist you today?"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 114,
    "completion_tokens": 9,
    "total_tokens": 123
  }
}
(1896e58d335001a483d8fb7670d6ebcf . 728)

yorisilo commented 11 months ago

I've made a bit of progress in my investigation. It seems that the issue might be that gptel-curl--parse-stream is not being set as expected.

When using chatgpt, it seems that the following gptel-curl--parse-stream is expected to be chosen dynamically:

https://github.com/karthink/gptel/blob/63027083cd5aef95ca6601d81c03069e8da2a667/gptel-openai.el#L40-L54

However, it appears to be applied while still being empty: https://github.com/karthink/gptel/blob/63027083cd5aef95ca6601d81c03069e8da2a667/gptel-curl.el#L279-L289

karthink commented 11 months ago

Thanks for the extensive report -- this rules out a bunch of possibilities.

I tested with emacs -q by installing a fresh gptel from MELPA, and I cannot reproduce the streaming error -- it works fine. So I'm somewhat stumped by this. :thinking:

Your modified init config looks fine to me, BTW.

karthink commented 11 months ago

However, it appears to be applied while still being empty:

The implementation of the parser is moved to gptel-openai.el, in the first block you pasted. What you included after is just the declaration of the generic function, which is supposed to be empty.

karthink commented 11 months ago

I tested with emacs -q by installing a fresh gptel from MELPA, and I cannot reproduce the streaming error

Could you try testing with emacs -q to be sure?

Run emacs -q

Add MELPA to the package archives:

(add-to-list 'package-archives '("melpa" . "http://melpa.org/packages/"))
(package-refresh-contents)

M-x package-install ⮐ gptel ⮐ (ignore the byte-compile warnings for now)
M-x gptel, paste in your API key when it asks for it

Try to use gptel as usual.

yorisilo commented 11 months ago

@karthink Thank you!

I tried using gptel after initializing as follows:

Moved the existing ~/.emacs.d outside of ~/.
Ran emacs -q, wrote the following in the scratch buffer, and executed each line with C-j (I also installed gnu because the compat package was needed to install gptel).

scratch-buffer

(customize-set-variable
   'package-archives '(("melpa" . "https://melpa.org/packages/")  ("gnu" . "https://elpa.gnu.org/packages/")))

(package-refresh-contents)

M-x package-install compat
M-x package-install gptel

Behavior of M-x gptel

To get straight to the point, it didn't work... When the default gptel-stream is set to t, the response is not rendered. If I set gptel-stream to nil, it gets rendered.

https://github.com/karthink/gptel/assets/10104705/594cba65-22d2-46f2-8687-956a90f55a51

Could the version of Emacs be related to this issue?

M-x version GNU Emacs 29.1 (build 1, aarch64-apple-darwin21.6.0, Carbon Version 165 AppKit 2113.6) of 2023-08-08

Or the part related to "compat" seems suspicious...

karthink commented 11 months ago

Thank you. This is very strange.

Does the problem go away if you use the previous version of gptel-curl--stream-filter (from your opening post)?

A couple of notes:

Moved the existing ~/.emacs.d outside of ~/.

emacs -q ignores your init file -- that's the point of -q, so you don't need to do this.

(I also installed gnu because the compat package was needed to install gptel).

The ELPA repo is included by package.el by default in Emacs, you don't need to add it yourself.

M-x package-install compat

Compat should be installed automatically as a dependency of transient, you don't need to install it.

Could the version of Emacs be related to this issue? Or the part related to "compat" seems suspicious...

It's not these things.

yorisilo commented 11 months ago

@karthink

Does the problem go away if you use the previous version of gptel-curl--stream-filter (from your opening post)?

Yes, with the initial changes I posted, it works. Additionally, if you modify gptel-curl--parse-stream in gptel-curl.el to what's written in gptel-openai.el, it also works.

Change this:

https://github.com/karthink/gptel/blob/63027083cd5aef95ca6601d81c03069e8da2a667/gptel-curl.el#L279-L289

to this:

https://github.com/karthink/gptel/blob/63027083cd5aef95ca6601d81c03069e8da2a667/gptel-openai.el#L40-L54

emacs -q ignores your init file -- that's the point of -q, so you don't need to do this.

Yes, but I moved the .emacs.d to avoid referencing an existing gptel when I run emacs -q. I wanted to use a fresh .emacs.d.

Compat should be installed automatically as a dependency of transient, you don't need to install it.

Indeed, there was no need to explicitly mention "compat".

yorisilo commented 11 months ago

@karthink I tried again by initializing as follows:

Moved the existing ~/.emacs.d outside of ~/. (because avoid referencing my .emacs.d dir)
Ran emacs -q
Executed the following in the scratch buffer. I ran (package-refresh-contents) first because emacs complains if the package-archives variable is not set.

(package-refresh-contents)

(add-to-list 'package-archives '("melpa" . "http://melpa.org/packages/"))

(package-refresh-contents)

Ran M-x package-install gptel
Explicitly used emacs-lisp-native-compile-and-load on all files within gptel

M-x gptel

It worked with the default setting (setq gptel-stream t)!

Also, without running emacs -q, I brought my .emacs.d back to ~/, deleted the eln-cache, and then explicitly used emacs-lisp-native-compile-and-load on all files within gptel. It now works perfectly with (setq gptel-stream t)!

yorisilo commented 11 months ago

Thank you so much for your patience and assistance throughout this process!

To summarize:

By deleting the eln-cache and explicitly using emacs-lisp-native-compile-and-load on all .el files within gptel, the response gets rendered in the buffer even with gptel-stream set to t.

karthink commented 11 months ago

Glad you found a solution.

Unfortunately Native Comp makes debugging much harder when there are multiple versions of a file involved.

On Mon, Oct 30, 2023, 11:45 AM yorisilo @.***> wrote:

Closed #120 https://github.com/karthink/gptel/issues/120 as completed.

— Reply to this email directly, view it on GitHub https://github.com/karthink/gptel/issues/120#event-10811791913, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACBVOLACECY7PBLYFDOZQFTYB7YULAVCNFSM6AAAAAA6VQBHJGVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJQHAYTCNZZGE4TCMY . You are receiving this because you were mentioned.Message ID: @.***>

kofm commented 11 months ago

Had the same problem, solved in the same way. Thank you!

meditans commented 11 months ago

I have this problem in doom emacs, but I'm not sure how to apply this workaround, as I don't have an eln-cache folder.

karthink commented 11 months ago

@meditans Instead of trying to delete the eln cache, could you just try M-x emacs-lisp-native-compile-and-load on the files in the gptel repo?

This should hopefully replace the ones in the eln cache. Please let me know if this works.

Also I'm reopening this thread since it appears to be affecting many people, and it will be easier to find at the top of the issues list.

wunki commented 11 months ago

This worked for me. It only reverts to the old behavior as soon as I restart Emacs, this may be because I'm using use-package with straight.

sauvala commented 11 months ago

I also had the same problem and deleting the eln-cache solved the issue. Thank you!

karthink / gptel