Ollama JSON format bug

[doslindos]

Hello team,

There's apparently known issue where sometimes the LLM sends junk whitespaces when mode is set to JSON with Ollama.

How this is related to langchaingo ? I was trying the nice example to add function calls to Ollama models. The example itself works fine, but when I added an own function to be called, I started getting this:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x6569b4]

goroutine 1 [running]:
github.com/tmc/langchaingo/llms/ollama.(*LLM).GenerateContent(0xc0000f4280, {0x7505f0, 0x9695e0}, {0xc0000a05a0, 0x2, 0xc0000160f0?}, {0x0, 0x0, 0xc0000ae060?})
    /home/juuso/go/pkg/mod/github.com/tmc/langchaingo@v0.1.10/llms/ollama/ollamallm.go:152 +0x9d4
main.main()

I added some logs and saw that there was indeed correct JSON format and actually correct function call with valid input out from the LLM, but a lot of junk whitespaces. But there were so many whitespaces that the response done field was false.

2024/06/15 17:06:46 {codestral:22b-v0.1-q8_0 2024-06-15 15:06:46.237829539 +0000 UTC 0xc000094540 false {0s 0s 0 0s 0 0s}}

After I found that here there's an assumption that if stream is false then the response will be "done" everytime and the resp.Message is not populated which is later tried to reference in here.

I fixed this by changing the clause to:

if *req.Stream == false || response.Done {

Which will catch the unfortunate scenario.

There are some open PRs to fix this on Ollama side, but adding this small change wouldn't make a lot of harm IMO and would make it so that expanding from the nicely written example will not turn into an hour debug session and sanity checking for some JSON string format incompatabilities :D

[tmc]

Very supportive -- do you want to open a PR for this?

[doslindos]

Sure I can slap it in tomorrow 👍