Integrate GPT 4o without TTS/STT

[clemlesne]

OpenAI GPT 4o model supports both in and out of text, image and audio. Understanding is finer than usual STT > model > TTS approach because the model has direct access to user behavior, emotions, etc.

Is there a way to use Communication Services and receive the raw audio flow, bypassing the STT step?

### Tasks
- [ ] https://github.com/microsoft/call-center-ai/issues/316
- [ ] https://github.com/microsoft/call-center-ai/issues/317

[Qwatro55]

I'm also interested in this question.

[agentverket]

What about response time? What about costs? Can you stream data?

[clemlesne]

I know I know :) OpenAI APIs are not yet available:

• https://community.openai.com/t/what-will-the-gpt-4o-audio-api-look-like/754242
• https://community.openai.com/t/gpt-4o-chat-completion-with-audio-response/752149/6

Plus, Communication Services APIs are not yet available to use with raw audio stream.

If you have ideas, don't hesitate!

[JunJD]

m

[clemlesne]

Audio streaming is now available with Communication Services!

https://learn.microsoft.com/en-us/azure/communication-services/how-tos/call-automation/audio-streaming-quickstart?pivots=programming-language-python

[ngoanpv]

Realtime API now support speech to speech from OpenAI. https://platform.openai.com/docs/guides/realtime/overview

I would like to explore more and add this feature to this project @clemlesne

[clemlesne]

We're working on it!