Getting the following exception 500: Expecting value: line 1 column 1443 (char 1442)

[gizbo]

Hello guys, Thank you for this awesome project. I'm getting this exception: 500: Expecting value: line 1 column 1443 (char 1442). I'm using a local Ollama (0.1.39) docker instance with the llama3 model. Here is the details log:

INFO: 172.25.0.1:52334 - "POST /chat HTTP/1.1" 200 OK Search provider: searxng Traceback (most recent call last): File "/workspace/src/backend/chat.py", line 111, in stream_qa_objects async for completion in response_gen: File "/workspace/.venv/lib/python3.11/site-packages/llama_index/core/llms/callbacks.py", line 280, in wrapped_gen async for x in f_return_val: File "/workspace/.venv/lib/python3.11/site-packages/llama_index/llms/ollama/base.py", line 408, in gen chunk = json.loads(line) ^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/init.py", line 346, in loads return _default_decoder.decode(s) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1443 (char 1442)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/workspace/src/backend/main.py", line 97, in generator async for obj in stream_qa_objects(chat_request): File "/workspace/src/backend/chat.py", line 140, in stream_qa_objects raise HTTPException(status_code=500, detail=detail) fastapi.exceptions.HTTPException: 500: Expecting value: line 1 column 144

Thanks for your help,

[ProjectMoon]

I am experiencing basically the same error, with searxng. Results stream in and then blow up with this: 500: Expecting value: line 1 column 4091 (char 4090). Wonder if this is a problem in the ollama library, given that it's happening in site-packages.

Edit: answers sometimes complete, but not often.

[Simsk8er]

Same thing except 500: Expecting value: line 1 column 2891 (char 2890)

[louisradtke]

I'm sure, it isn't the perfect solution, but might be a quick fix for you guys. In my setup using the docker-compose.dev.yaml, the error appears in chat.py, ln. 111. So surrounding it with a try-catch lets me at least continue my conversation with farfalle. I just had to rebuild and -start the compose stack (docker compose -f docker-compose.dev.yaml up --build). But without digging deeper, I can't tell if I'm breaking anything else. Here's my fixed code:

        try:
            async for completion in response_gen:
                full_response += completion.delta or ""
                yield ChatResponseEvent(
                    event=StreamEvent.TEXT_CHUNK,
                    data=TextChunkStream(text=completion.delta or ""),
                )
        except Exception as ex:
            print('ERROR:\terror completing the stream')

[ProjectMoon]

While that stops the page from crashing, it suppresses the entire answer from the LLM. A better way to do it would be to put the try-catch inside the loop, but I'm not sure that's possible, because I think it is deserializing the JSON when yielding. That is, the error happens before the completion variable is even assigned to the loop iteration.