continuedev / continue

⏩ Continue is the leading open-source AI code assistant. You can connect any models and any context to build custom autocomplete and chat experiences inside VS Code and JetBrains
https://docs.continue.dev/
Apache License 2.0
18.27k stars 1.48k forks source link

Ollama Unexpected end of JSON input #751

Closed matbee-eth closed 9 months ago

matbee-eth commented 9 months ago

Before submitting your bug report

Relevant environment info

- OS: Mac / WSL2 / Cloudflare tunnel
- Continue:

Description

I'm running a cloudflare tunnel for my server-based ollama service. It seems like Continue errors when receiving chunked encoded JSONL responses and is expecting a full JSON object in a response, during a streamed response. It should be waiting for a \n in a JSONL response before it attempts to JSON.parse.

[Extension Host] Error: Error parsing Ollama response: SyntaxError: Unexpected end of JSON input {"model":"phind-codellama","created_at":"2024-01-14T21:06:21.760129309Z","message":{"role":"assistant","content"
    at Ollama._streamChat (c:\Users\mail\.vscode\extensions\continue.continue-0.8.1-win32-x64\out\extension.js:35485:23)
    at _streamChat.next (<anonymous>)
    at Ollama.streamChat (c:\Users\mail\.vscode\extensions\continue.continue-0.8.1-win32-x64\out\extension.js:34521:30)
    at streamChat.next (<anonymous>)
    at f.value (c:\Users\mail\.vscode\extensions\continue.continue-0.8.1-win32-x64\out\extension.js:54527:28)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

To reproduce

No response

Log output

No response

matbee-eth commented 9 months ago

https://github.com/continuedev/continue/blob/5b56150a8dc8345422f75107c899d31b87417d65/core/llm/llms/Ollama.ts#L163C1-L163C41

Seems to parse the JSON even without validating the response type. ndjson responses need to be newline delimited.

Submitted a small quick and dirty PR to fix my issue. I haven't looked too deeply but I'm going to assume streamed ndjson/JSONL parsing in general will need to be fixed. https://github.com/continuedev/continue/pull/752