Mixing streaming and tool use

ollama / ollama-js

Ollama JavaScript library

https://ollama.com

MIT License

2.24k stars 175 forks source link

Mixing streaming and tool use #123

Open efritz opened 3 months ago

efritz commented 3 months ago

When I enable streaming it seems that the tool use JSON payload is sent as part of the content field of the ChatResponse value and tool_calls remains empty. Disabling streaming for the same model/prompt populates this array as I'd expect.

Is there a way to enable the same parsing with streaming enabled (or, alternatively a guide on how to consistently parse a tool use from the streaming model response)?

BruceMacD commented 3 months ago

Hey @efritz, this is a limitation on tool use with the latest version of Ollama. There is a fix open in the main repository now: https://github.com/ollama/ollama/pull/6066

I'll keep this issue open to track when this gets merged into the JavaScript SDK.