How to use streaming response?

NVIDIA / NeMo-Guardrails

NeMo Guardrails is an open-source toolkit for easily adding programmable guardrails to LLM-based conversational systems.

Other

3.78k stars 335 forks source link

How to use streaming response? #48

Open YoshitakaOyama opened 1 year ago

YoshitakaOyama commented 1 year ago

First of all, thank you for doing a great library development. It's very cool and easy to use and we would like to incorporate it into our chatbot projects!

But I would like to use the ability to generate answers in real time. Equivalent to openai stream option. How to use it ...?

drazvan commented 1 year ago

Hi @YoshitakaOyama ! Streaming is challenging (see https://github.com/NVIDIA/NeMo-Guardrails/discussions/15). It's on our roadmap but we don't yet have an ETA for when we'll ship updates related to this.

YoshitakaOyama commented 1 year ago

Hi @drazvan! ok, got it. I hope it will be implemented soon. Thank you for your answer!

Sassun commented 11 months ago

Any updates on this?

closedLoop commented 9 months ago

I think a minimal requirement would be to enable steaming (via SSE) of:

[ ] The last response in a flow
[ ] Optionally some data related to the actions being taken by the agent