Streaming responses

[JonHolman]

Has anyone used this package to stream the responses back to the requestor as llama.cpp generates the response? If not, any tips to achieving that? Thank you.

[synw]

Yes: I use server sent events in my minimalist inference server project. Check the code here:

• https://github.com/synw/goinfer/blob/main/server/infer.go#L155
• https://github.com/synw/goinfer/blob/main/lm/infer.go#L16

[mudler]

the example in the go-llama.cpp repository shows how to stream responses: https://github.com/go-skynet/go-llama.cpp/blob/0f3da8c646b780d865264aa76403d48763ace289/examples/main.go#L49C65-L49C65

Closing this issue - feel free to open up a discussion as there is nothing to do