search

pepperoni21 / ollama-rs

A Rust library allowing to interact with the Ollama API.
MIT License
367 stars 47 forks source link

Question: What is the meaning ollama-rs 0.1.6 stream.next() returning a Vec<GenerationResponse> #20

Closed jeremychone closed 5 months ago

jeremychone commented 5 months ago

Hi, I'm certain it's logical, but I'd like to grasp the reason behind the change in ollama-rs 0.1.6, where stream.next() now returns a Vec<GenerationResponse> instead of a single GenerationResponse.

Context

I created a video about the ollama-rs full tutorial using ollama-rs 0.1.5. I've begun updating the code to accommodate ollama-rs 0.1.6, but I want to provide context in the readme and YouTube description.

pepperoni21 commented 5 months ago

Hey! Ollama can send multiple responses in the same payload (it uses the Newline-Delimited JSON format), I never had this issue myself but some people did when the payload contains multiple responses.

jeremychone commented 5 months ago

@pepperoni21 Ah, multiple responses to the same GenerationRequest. That's interesting. I'm not certain when that would occur, but if the underlying HTTP API allows for it, it's good to expose this at the library level.

I've implemented the fix, and it works, I've made a note of this in the video's comments/description. A little friction for the user, but I should have locked my code/video to ollama-rs =0.1.5. My bad.

Note: I'm not sure the currrent will display correctly if there are multiple GenerationResponses for the same request. However, I believe the code is unlikely to produce such a scenario.

Thank you for your response.