search

microsoft / autogen

A programming framework for agentic AI 🤖
https://microsoft.github.io/autogen/
Creative Commons Attribution 4.0 International
32.85k stars 4.78k forks source link

[Info]: Autogen studio - streaming response to web client #3089

Open satnair opened 4 months ago

satnair commented 4 months ago

Describe the issue

Hi,

As part of autogen studio, is there a support for streaming response to the web client, than flushing the response towards the end. Any suggestions or guidelines on how to implement, if not already available.

Thanks.

Steps to reproduce

No response

Screenshots and logs

No response

Additional Information

No response

victordibia commented 4 months ago

Hi,

Do you mean streaming responses to a web client other than the autogenstudio web client or streaming responses to the AutoGen studio UI itself.

Currently, AutoGen studio supports streaming complete messages from agents (not tokens generated by an LLM) as then become available. It does this using a queue + threaded implementation where agents can put messages on a shared queue, and a background thread reads from that queue and sends the messages over a socket to the autogen studio UI. See this video as an example of messages being streamed as they as agents send them

image

I also wrote about some patterns for streaming autogen agent messages to your own app here in case that is useful.

satnair commented 4 months ago

@victordibia Thanks much for the details. Yes, for sure it is helpful.

I am looking to stream responses, to the autogen studio UI itself as the tokens gets generated by LLM, so that the responses get printed on the UI in an incremental fashion, than waiting for the whole response from an agent to flush it to the UI.

It also improves the user experience , that the user see output getting generated and just don't need to see the spinner on "Agents working on it". Any suggestions on how to implement this in this framework , would be helpful. Thank you