Losing chunks of replies

[jwr]

I only noticed this with Anthropic, but I am not 100% certain this applies to Anthropic only.

I get replies where often several characters are missing. I investigated, turned on logging and here is an example. In my output gptel session I see "dostępnośćwencji", while the log shows:

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" dost"}    }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"ęp"}            }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"ność"}           }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" sek"}  }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"w"}            }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"enc"}               }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"ji"}    }

Note the missing " sek" chunk. It's there in the log, so this would indicate that we received it, but lost it afterwards.

This happens irregularly, sometimes 1-2 times per paragraph, sometimes once every several paragraphs.

This happens at least with output to a new buffer, or to a gptel session.

[karthink]

Thanks, I guess the fix in #233 isn't quite complete. I It's a timing based bug, so I'll need to set up a simulation of the streaming and try to find the right conditions to reproduce it.

[karthink]

@jwr Could you try this:

Advise the stream filter as follows to capture the output (clear the scratch buffer first):

(defun gptel-filter-debug (_ output)
  (with-current-buffer (get-buffer-create "*scratch*")
    (insert output "\n-----\n")))

(advice-add 'gptel-curl--stream-filter :before #'gptel-filter-debug)

Use the Anthropic backend and reproduce the error, then paste the contents of the scratch buffer here.

Remove the advice afterwards so you can use gptel normally.

(advice-remove 'gptel-curl--stream-filter #'gptel-filter-debug)

This should help me nail down the timing to test against.

[jwr]

This doesn't seem to happen as often as before, but here are some examples:

Output is "By combining anch n", should be "By combining anchored n":

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" By"}               }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" combining"}         }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" anch"}           }

event: cont
-----
ent_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"ored"}        }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" n"}     }

Output is "the relev and accuracy", should be "the relevance and accuracy":

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" the"}   }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" relev"}  }

event: conten
-----
t_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"ance"}  }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" and"}      }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" accuracy"}             }

I hope this helps.

[jwr]

I just realized you wanted the entire *scratch* contents:

scratch.txt

[CestDiego]

I'm having this same issue only with claude models as well.

[jwr]

Yes, this seems to happen with Haiku, Sonnet and Opus.

[karthink]

Pushed a commit (26326c30), please check if this is fixed now.

[jwr]

I am no longer able to reproduce the problem 🙂🙏 Thank you!

[benthamite]

Unfortunately, I am experiencing this same issue, despite having updated to the latest commit (#70889ad). Attached are the contents of the *scratch* buffer, after following the instructions in this comment above.

Not sure if it’s useful, but an example of missing text occurs in the sentence starting "Pero cuando encuentres algo que...".

scratch.txt

[karthink]

@benthamite thanks for the data. Do you have the text as it is inserted into your buffer? I'd like to know exactly which chunk was missed.

[benthamite]

@karthink, yes, I saved the gptel buffer to a file, which you can find below. (The file is zipped because GitHub didn't allow me to upload the org-mode file.)

In the example above, the truncated sentence is

Pero cuando encuentres algo que teté amargando, detente.

I think the full full sentence should have been

Pero cuando encuentres algo que te esté amargando, detente.

alegremente-claude-translate.org.zip

[benthamite]

(There are other examples. I can list them if that helps.)