Failed to deserialize response: trailing characters at line 2 column 1

[manglemix]

I am getting this error while trying to run dolphin-mixtral:latest. It seems to be related to how serde_json parses json. I am using the stream feature, and I get this error on unwrapping the elements returned from next, but I do get some Ok values before that happens.

[manglemix]

Also this error is directly printed to console instead of being returned, even though the returned error is (). It would be much better if you return this message as a String instead of just ().

[pepperoni21]

I am getting this error while trying to run dolphin-mixtral:latest. It seems to be related to how serde_json parses json. I am using the stream feature, and I get this error on unwrapping the elements returned from next, but I do get some Ok values before that happens.

Thanks for reporting this. There might have been a change in the response format, I'm gonna run some tests and I'll let you know

Also this error is directly printed to console instead of being returned, even though the returned error is (). It would be much better if you return this message as a String instead of just ().

Yes you're right, I'm gonna change that

[pepperoni21]

I am getting this error while trying to run dolphin-mixtral:latest. It seems to be related to how serde_json parses json. I am using the stream feature, and I get this error on unwrapping the elements returned from next, but I do get some Ok values before that happens.

I'm not able to reproduce this issue with this model, can you try using the issue-12 branch, I made so that the json is debugged when using generate_stream so I can see what is wrong with the json: ollama-rs = { git = "https://github.com/pepperoni21/ollama-rs", branch = "issue-embeddings"}

[manglemix]

Sorry it took me a while to reserve an A100 on google colab. Here is some output:

[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] "Trying to parse response: {:?}" = "Trying to parse response: {:?}"
[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] String::from_utf8_lossy(&bytes) = "{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:12:45.474281683Z\",\"response\":\" Hello\",\"done\":false}\n"
 Hello[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] "Trying to parse response: {:?}" = "Trying to parse response: {:?}"
[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] String::from_utf8_lossy(&bytes) = "{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:12:45.495563603Z\",\"response\":\"!\",\"done\":false}\n"
![C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] "Trying to parse response: {:?}" = "Trying to parse response: {:?}"
[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] String::from_utf8_lossy(&bytes) = "{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:12:45.516758534Z\",\"response\":\" How\",\"done\":false}\n"
 How[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] "Trying to parse response: {:?}" = "Trying to parse response: {:?}"
[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] String::from_utf8_lossy(&bytes) = "{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:12:45.535628614Z\",\"response\":\" can\",\"done\":false}\n"
 can[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] "Trying to parse response: {:?}" = "Trying to parse response: {:?}"
[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] String::from_utf8_lossy(&bytes) = "{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:12:45.553561916Z\",\"response\":\" I\",\"done\":false}\n"
 I[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] "Trying to parse response: {:?}" = "Trying to parse response: {:?}"
[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] String::from_utf8_lossy(&bytes) = "{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:12:45.571562091Z\",\"response\":\" assist\",\"done\":false}\n"
 assist[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] "Trying to parse response: {:?}" = "Trying to parse response: {:?}"
[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] String::from_utf8_lossy(&bytes) = "{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:12:45.58944567Z\",\"response\":\" you\",\"done\":false}\n{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:12:45.607384298Z\",\"response\":\" today\",\"done\":false}\n{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:12:45.625372937Z\",\"response\":\"?\",\"done\":false}\n{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:12:45.643531751Z\",\"response\":\"\",\"done\":true,\"context\":[32001,6574,13,24205,574,8570,6817,28723,32000,13,32001,1838,13,21558,28801,13,32000,13,32001,489,11143,13,22557,28808,1602,541,315,6031,368,3154,28804],\"total_duration\":376468647,\"load_duration\":758387,\"prompt_eval_count\":23,\"prompt_eval_duration\":226302000,\"eval_count\":9,\"eval_duration\":147877000}\n"
An error occurred with ollama-rs: trailing characters at line 2 column 1

[manglemix]

More output if you need it

[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] "Trying to parse response: {:?}" = "Trying to parse response: {:?}"
[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] String::from_utf8_lossy(&bytes) = "{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:14:24.149187697Z\",\"response\":\" Sure\",\"done\":false}\n"
 Sure[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] "Trying to parse response: {:?}" = "Trying to parse response: {:?}"
[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] String::from_utf8_lossy(&bytes) = "{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:14:24.168397594Z\",\"response\":\",\",\"done\":false}\n"
,[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] "Trying to parse response: {:?}" = "Trying to parse response: {:?}"
[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] String::from_utf8_lossy(&bytes) = "{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:14:24.186455338Z\",\"response\":\" I\",\"done\":false}\n"
 I[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] "Trying to parse response: {:?}" = "Trying to parse response: {:?}"
[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] String::from_utf8_lossy(&bytes) = "{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:14:24.204385487Z\",\"response\":\" can\",\"done\":false}\n"
 can[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] "Trying to parse response: {:?}" = "Trying to parse response: {:?}"
[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] String::from_utf8_lossy(&bytes) = "{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:14:24.222460852Z\",\"response\":\" help\",\"done\":false}\n"
 help[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] "Trying to parse response: {:?}" = "Trying to parse response: {:?}"
[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] String::from_utf8_lossy(&bytes) = "{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:14:24.240432697Z\",\"response\":\" with\",\"done\":false}\n"
 with[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] "Trying to parse response: {:?}" = "Trying to parse response: {:?}"
[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] String::from_utf8_lossy(&bytes) = "{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:14:24.258394124Z\",\"response\":\" that\",\"done\":false}\n{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:14:24.27637587Z\",\"response\":\"!\",\"done\":false}\n{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:14:24.294387144Z\",\"response\":\" Could\",\"done\":false}\n{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:14:24.312447488Z\",\"response\":\" you\",\"done\":false}\n{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:14:24.330516399Z\",\"response\":\" provide\",\"done\":false}\n{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:14:24.348585Z\",\"response\":\" me\",\"done\":false}\n"
An error occurred with ollama-rs: trailing characters at line 2 column 1
[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] "Trying to parse response: {:?}" = "Trying to parse response: {:?}"
[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] String::from_utf8_lossy(&bytes) = "{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:14:24.366630652Z\",\"response\":\" with\",\"done\":false}\n"
 with[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] "Trying to parse response: {:?}" = "Trying to parse response: {:?}"
[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] String::from_utf8_lossy(&bytes) = "{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:14:24.384812565Z\",\"response\":\" more\",\"done\":false}\n"
 more[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] "Trying to parse response: {:?}" = "Trying to parse response: {:?}"
[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] String::from_utf8_lossy(&bytes) = "{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:14:24.402817915Z\",\"response\":\" details\",\"done\":false}\n"
 details[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] "Trying to parse response: {:?}" = "Trying to parse response: {:?}"
[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] String::from_utf8_lossy(&bytes) = "{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:14:24.420805745Z\",\"response\":\" or\",\"done\":false}\n"
 or[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] "Trying to parse response: {:?}" = "Trying to parse response: {:?}"
[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] String::from_utf8_lossy(&bytes) = "{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:14:24.438855239Z\",\"response\":\" context\",\"done\":false}\n"
 context[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] "Trying to parse response: {:?}" = "Trying to parse response: {:?}"
[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] String::from_utf8_lossy(&bytes) = "{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:14:24.456813299Z\",\"response\":\" about\",\"done\":false}\n"
 about[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] "Trying to parse response: {:?}" = "Trying to parse response: {:?}"
[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] String::from_utf8_lossy(&bytes) = "{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:14:24.474910724Z\",\"response\":\" the\",\"done\":false}\n"
 the[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] "Trying to parse response: {:?}" = "Trying to parse response: {:?}"
[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] String::from_utf8_lossy(&bytes) = "{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:14:24.49292812Z\",\"response\":\" task\",\"done\":false}\n"
 task[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] "Trying to parse response: {:?}" = "Trying to parse response: {:?}"
[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] String::from_utf8_lossy(&bytes) = "{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:14:24.510961798Z\",\"response\":\" you\",\"done\":false}\n"
 you[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] "Trying to parse response: {:?}" = "Trying to parse response: {:?}"
[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] String::from_utf8_lossy(&bytes) = "{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:14:24.528932685Z\",\"response\":\" need\",\"done\":false}\n"
 need[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] "Trying to parse response: {:?}" = "Trying to parse response: {:?}"
[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] String::from_utf8_lossy(&bytes) = "{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:14:24.546956679Z\",\"response\":\" assistance\",\"done\":false}\n"
 assistance[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] "Trying to parse response: {:?}" = "Trying to parse response: {:?}"
[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] String::from_utf8_lossy(&bytes) = "{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:14:24.564929951Z\",\"response\":\" with\",\"done\":false}\n"
 with[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] "Trying to parse response: {:?}" = "Trying to parse response: {:?}"
[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] String::from_utf8_lossy(&bytes) = "{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:14:24.582929451Z\",\"response\":\"?\",\"done\":false}\n"
?[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] "Trying to parse response: {:?}" = "Trying to parse response: {:?}"
[C:\Users\shabo\.cargo\git\checkouts\ollama-rs-ce3b35b7a91af229\3d5c0f7\src\generation\completion\mod.rs:43] String::from_utf8_lossy(&bytes) = "{\"model\":\"dolphin-mixtral:latest\",\"created_at\":\"2023-12-25T01:14:24.601124568Z\",\"response\":\"\",\"done\":true,\"context\":[32001,6574,13,24205,574,8570,6817,28723,32000,13,32001,1838,13,1613,28801,13,32000,13,32001,489,11143,13,12875,28725,315,541,1316,395,369,28808,13955,368,3084,528,395,680,4162,442,2758,684,272,3638,368,927,11611,395,28804],\"total_duration\":656138919,\"load_duration\":870205,\"prompt_eval_count\":23,\"prompt_eval_duration\":221821000,\"eval_count\":25,\"eval_duration\":432633000}\n"

[manglemix]

These logs are interspersed with some valid output from the AI

[pepperoni21]

It seems to stream multiple json objects at the same time, which causes the error, I will make an issue on the ollama repo later in the day

[pepperoni21]

@manglemix It should be fixed in the issue-12 branch, let me know

[manglemix]

Hi it works perfectly!

[manglemix]

Actually I'm sorry to re-open this but I am getting more errors:

An error occurred with ollama-rs: EOF while parsing a value at line 1 column 4090
An error occurred with ollama-rs: invalid type: integer `707`, expected struct GenerationResponse at line 1 column 3

This error appears after a complete response has been sent (as far as I can tell). The new context has been provided though so the final data is being received.

[pepperoni21]

Actually I'm sorry to re-open this but I am getting more errors:

An error occurred with ollama-rs: EOF while parsing a value at line 1 column 4090
An error occurred with ollama-rs: invalid type: integer `707`, expected struct GenerationResponse at line 1 column 3

This error appears after a complete response has been sent (as far as I can tell). The new context has been provided though so the final data is being received.

Ok, that should be fixed too now