Add support for realtime API

[mudler]

Is your feature request related to a problem? Please describe.

OpenAI just extended their API with realtime support with web sockets https://openai.com/index/introducing-the-realtime-api/?s=09

Describe the solution you'd like

LocalAI should support backends with voice capabilities and introduce a compatible API endpoint with OpenAI clients.

Ideally it should support also function calling as OpenAI does:

Under the hood, the Realtime API lets you create a persistent WebSocket connection to exchange messages with GPT-4o. The API supports function calling(opens in a new window), which makes it possible for voice assistants to respond to user requests by triggering actions or pulling in new context. For example, a voice assistant could place an order on behalf of the user or retrieve relevant customer information to personalize its responses.

Seems that also Chat completion API is gonna have audio output/input too, but API specs are not available yet:

Audio in the Chat Completions API will be released in the coming weeks, as a new model gpt-4o-audio-preview. With gpt-4o-audio-preview, developers can input text or audio into GPT-4o and receive responses in text, audio, or both.

Describe alternatives you've considered

Additional context

https://github.com/mudler/LocalAI/issues/3602 https://github.com/mudler/LocalAI/pull/3722

API docs: https://platform.openai.com/docs/guides/realtime https://platform.openai.com/docs/api-reference/realtime-client-events/session-update

https://github.com/tmc/grpc-websocket-proxy https://github.com/openconfig/grpctunnel

https://github.com/mudler/LocalAI/tree/feat/realtime

open source models that can handle realtime speech:

• https://github.com/homebrewltd/ichigo
• https://github.com/ictnlp/LLaMA-Omni
• https://github.com/kyutai-labs/moshi
• https://speech.fish.audio/

[mudler]

A good candidate VAD library: https://github.com/snakers4/silero-vad/tree/master/examples/go

[mattkanwisher]

I started looking at stubbing out the api, it's mostly just json, curious why you are suggesting the grpc-websocket-proxy?

[mudler]

I started looking at stubbing out the api, it's mostly just json, curious why you are suggesting the grpc-websocket-proxy?

I was digging a bit into projects that are interfacing with grpc and websockets - was just adding some code/notes here to pick up brain with, very preliminar search, that might be useful as reference/getting some ideas from

[mudler]

I started looking at stubbing out the api

Are you going to open up a PR? I was about to start playing with it as well, but if you are already taking a stab at it I'd go with #3670 instead :)

Update: Opened #3722 with what I had laying around. Now gonna have a look at vLLM first :goggles:

[thiswillbeyourgithub]

A good candidate VAD library: https://github.com/snakers4/silero-vad/tree/master/examples/go

Heard about that one btw : https://github.com/wavey-ai/mel-spec?tab=readme-ov-file

Also copy pasting maybe useful ressources I linked in another repo, as 4o will not be the only one to support this if we want to bot rely too much on openai's code :

---

i saw on hackernews that agents by livekits used to make the openai realtime api as well as cerebras voice seems to be open source.

They have tons of demos and code on their github. I think there must be a llama-omni implementation somewhere that would be a killer feature for open-webui!

Here's a particularly interesting demo that connects stt + llm + tts: https://github.com/livekit/agents/blob/main/examples/voice-pipeline-agent/minimal_assistant.py

I made an issue to ask for a demo for Llama-Omni, also for kyutai's moshi model. There's also model's moshi implementation : https://github.com/modal-labs/quillman

[fofsinx]

I'm trying to build the server implementation based on openai spec for their Realtime API.

https://github.com/iamharshdev/OLlamaGate

[mudler]

There is a WIP branch over here : https://github.com/mudler/LocalAI/pull/3722

Contribution and feedbacks always welcome!