.Net: New Feature: Add support for GPT-4o Real Time endpoint

microsoft / semantic-kernel

Integrate cutting-edge LLM technology quickly and easily into your apps

https://aka.ms/semantic-kernel

MIT License

21.53k stars 3.18k forks source link

.Net: New Feature: Add support for GPT-4o Real Time endpoint #9075

Open rboen opened 17 hours ago

rboen commented 17 hours ago

Low latency conversational interactions using speech is an impressive enhancement and game changer for audio chat bots. With the emergence of the gpt-40-realtime-preview in Azure OpenAI I'd love to see an integration with the Sematik Kernel in order to facilitate agents, skills / plugins in call agents scenarios.

Please have a look at https://github.com/azure-samples/aoai-realtime-audio-sdk

RogerBarreto commented 10 hours ago

@rboen Thanks for the ask.

We will keep track on this feature and investigate how to bring as a Speech-to-Speech streaming abstraction to SK, for now our suggestion is while we don't have this abstraction in place to use our current APIs with the breaking glass option (Providing either the OpenAIClient or AzureOpenAIClient, directly) and consuming the RealtimeConversationClient directly.