Closed JonHolman closed 11 months ago
Yes: I use server sent events in my minimalist inference server project. Check the code here:
the example in the go-llama.cpp repository shows how to stream responses: https://github.com/go-skynet/go-llama.cpp/blob/0f3da8c646b780d865264aa76403d48763ace289/examples/main.go#L49C65-L49C65
Closing this issue - feel free to open up a discussion as there is nothing to do
Has anyone used this package to stream the responses back to the requestor as llama.cpp generates the response? If not, any tips to achieving that? Thank you.