WebChat: Support for ‘stream’ output in ChatGPT style while using Azure OpenAI Completion

[dawwa]

Bot responses with Azure OpenAI Completion, but we need the response to be displayed in a live streaming format.

Describe the suggestion or request in detail

Allow Application that uses Bot framework webchat to stream out the response like ChatGPT web chat does.

[compulim]

This is good.

Internally, we are planning this feature, streaming texts/completions from Azure OpenAI and other capable services into Web Chat.

[HesselWellema]

Implementing this would be much appreciatedd. We still rely on your webchat widget for most of our implementation. The time it takes to get the full response from the azure openai-api is perceived as latency by users.

[pedrojlazevedo]

This feature, after MSTeams update, is a must! Really keen to see an update on this

[wsp174212]

Thanks for your help, Is there an ETA for this feature?

[HesselWellema]

Or maybe share some insight in what to expect

[biyukai88]

May I ask when you plan to go online？We are all looking forward to this feature！

[wsp174212]

@compulim I hope this message finds you well. Could you please let me know if there is an ETA for this feature. More and more popular tools are releasing new features like GPT. We really need this to expand our business.

[rliberoff]

Hello Team!

This feature is really important. Today, it is possible to update an activity in teams (await turnContext.UpdateActivityAsync(responseActivity, cancellationToken);) creating a stream (or typing) experience like ChatGPT.

However, it is currently impossible to do so with the Webchat 😟

When can we expect this feature?

Thank you.

[anshumankmr]

Hey folks, is there a workaround for this in Bot Framework WebChat?

[HesselWellema]

Update from @compulim : https://github.com/microsoft/BotFramework-WebChat/issues/5121#issuecomment-2042229329

[DukaJ]

Dear Mr. Wong, first of all thank you for your effort and commitment that Webchat is working stable so many years! Please check my question below:

Are there any updates on this topic that enable streaming responses through Webchat component through Bot framework SDK? This is extremely important feature since without it, in the era of LLM, usage of the component is having challenges.

If you may please share expected date for this to be enabled @compulim

Kind regards and thank you in advance!

[Joekraft]

@compulim could you clarify what #5141 is meant to resolve here? It appears to be implementing a typing indicator, and "livestreaming"? But there is no clear documentation or examples how this links back to the expected support of response streaming from AI generated answers?

[compulim]

I am adding a LIVESTREAMING.md in the PR #5265. The document outlines how to send livestreaming from the bot code and also bot/channel/client supportability.

@DukaJ thanks for your support and understanding, the document would explain how to add it to the bot.

@Joekraft we are extending the typing activity for livestreaming. So, yes, you will send a typing activity for livestreaming. Despite typing activity has been available for a very long time, its design is naturally extensible to cover livestreaming cases in additional to typing indicator. In some future, it could be used for bot-assisted auto-completion too. 😉

Thank you everyone for support. We are working very hard to add livestreaming support to a very broad ecosystem at Microsoft, including but not limited to Azure Bot Services, Copilot Studio, Teams.