Claude input processing issue

[agheieff]

There is an issue with claude, not sure if it affects the other models. It seems to stop taking the input long before the message_stop appears. It usually starts by working correctly, printing 2 words, sometimes somewhat more (up to around 20), and then just stops. I tried printing out the chunks into the nvim console, and the last one just ended in the middle of the JSON response. When I tried the curl request from my terminal, it worked fine, so the issue is not on anthropic's side. So the issue is probably with either the JSON input stopping before it should, or with the parsing of it.

P.S. if you need a temporary key for testing, please reach out directly at agheieff@pm.me

[agheieff]

Here are some improvised logs. As you can see, there is no message_end in there

`2024-06-22 02:50:03: Response processing complete 2024-06-22 02:50:03: API request completed 2024-06-22 02:50:03: Timeout reached after 28287.751483 ms 2024-06-22 02:50:03: Full API Response: event: message_start data: {"type":"message_start","message":{"id":"msg_01RaQ5vdvWtvg9CwHk7TTpSa","type":"message","role":"assistant","model":"claude-3-5-sonnet-20240620","content":[],"stop_reason":null,"stop_sequence":null,"usage":{"input_tokens":217,"output_tokens":2}} }

event: content_block_start data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""} }

event: ping data: {"type": "ping"}

event: content_block_delta data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Understoo"} }

2024-06-22 02:50:03: Parsed JSON object: { message = { content = {}, id = "msg_01RaQ5vdvWtvg9CwHk7TTpSa", model = "claude-3-5-sonnet-20240620", role = "assistant", stop_reason = vim.NIL, stop_sequence = vim.NIL, type = "message", usage = { input_tokens = 217, output_tokens = 2 } }, type = "message_start" } 2024-06-22 02:50:03: Parsed JSON object: { content_block = { text = "", type = "text" }, index = 0, type = "content_block_start" } 2024-06-22 02:50:03: Parsed JSON object: { type = "ping" } 2024-06-22 02:50:03: Parsed JSON object: { delta = { text = "Understoo", type = "text_delta" }, index = 0, type = "content_block_delta" } 2024-06-22 02:50:03: Response processing complete 2024-06-22 02:50:03: API request completed`

[melbaldove]

@agheieff Claude works fine for me.

• What machine are you using?
• Can you send the prompt?
• Does this issue always happen such that you never get full outputs?

[agheieff]

@melbaldove huh, I'll try to take some screenshots when I get to my PC in about an hour. It worked with any prompt really. For example, I would put in:

Tell me about python?

And it would respond (after Ctrl+, on a new line

Sure! Python is a high leve

And stop like this in the middle of the message.

I'm on arch, ThinkPad t480 (I don't know if that's relevant), and otherwise use an almost default kickstart setup for nvim.

[agheieff]

It works with any prompt really, here's an example

Sometimes it just times out without any output. I can see that the output is being generated in the Anthropic console. I just tested it on windows, and there it just times out without output every time.

[agheieff]

Sorry, for the last question. I sometimes get very short outputs, that seem complete (like two lines of code), or when I specifically ask it to reply with a single word, but never a full longer answer

[melbaldove]

Ya, looks like something is broken on your end. Im thinking might be something to do with curl. Btw I am on a mac, so there might be some minor differences there, unsure yet. I'll install linux to debug when I get to.

Does groq, or any openai api model work?

[jakobdylanc]

Not sure if related but this definitely looks like a typo: https://github.com/melbaldove/llm.nvim/blob/1ad97eb73ce480af7fa4ba1bb6b312623fa6cfed/lua/llm.lua#L71

[agheieff]

Not sure if related but this definitely looks like a typo: https://github.com/melbaldove/llm.nvim/blob/1ad97eb73ce480af7fa4ba1bb6b312623fa6cfed/lua/llm.lua#L71

Probably a typo, but not related. I fixed that and it still didn't work

[agheieff]

Could be something with nio tbh. Curl requests work fine from terminal, giving the entire output. The issue is somewhere between the response being sent by the API and the JSON processing. Either only a part of the response is received, or the whole response is received, but only a part is processed into chunks correctly

[melbaldove]

@jakobdylanc thanks, fixed it. @agheieff just for good measure, please check if it fixes your issue(doubt it)

[jakobdylanc]

Also FYI I'm experiencing similar issues with both Groq and OpenAI (responses cut short, frequent timeouts). Haven't tested with Claude yet.

And for some reason it's not working at all now (I see llm.nvim has timed out! with no respose ever generated, with both Groq and OpenAI).

I'm a nvim noob so still learning here...could be user error.

[melbaldove]

@agheieff does your earlier logs have the print statement after this line?

https://github.com/melbaldove/llm.nvim/blob/cd44a78c0d9fa102436f0710e2795ac4ab0c7392/lua/llm.lua#L107

[melbaldove]

@jakobdylanc are you on linux too?

[jakobdylanc]

Yes, Ubuntu 22.04.2

[agheieff]

Sorry, I don't remember the exact logging mechanism. But I tried printing the chunks as well, and from what I understood, done is never evaluated to true, but still the loop that you referenced is never exited (any code I put after that loop never run)

This is, for example, what I get when putting

local function process_data_lines(lines, service, process_data)
    for _, line in ipairs(lines) do
        print(line) --the rest is the same

around line 63 (sorry, I dont know how to reference the code directly)

P.S. still not working with the comparison fix

[agheieff]

And as @jakobdylanc said, around half of the time it just times out with no output.

Also when I tried it on windows, it would just time out every single time, but it is possible I missed something in the setup, I didn't really test it out properly then

[melbaldove]

Also when I tried it on windows, it would just time out every single time, but it is possible I missed something in the setup, I didn't really test it out properly then

its not gonna work on windows because the plugin uses curl, prolly need to reimplement the sse processing without curl EDIT: realized windows 10 has curl

around line 63 (sorry, I dont know how to reference the code directly)

will you be able to help me and show me the logs after adding a print after this line? https://github.com/melbaldove/llm.nvim/blob/cd44a78c0d9fa102436f0710e2795ac4ab0c7392/lua/llm.lua#L107

[agheieff]

Sure, here you go The code:

[agheieff]

Fixed by making my own plugin from scratch