Request: Enhance audio-text synchronization for RESPONSE_AUDIO_TRANSCRIPT_DELTA and RESPONSE_AUDIO_DELTA events

[opchronatron]

Objective: Improve the ability to align text and audio deltas for smoother playback and interruption handling. Proposed solutions (in order of preference):

• Implement corresponding event_ids between text and audio delta events.
• Alternatively, provide approximate audio frame numbers for sentence pauses or completions.

Benefits: Enables graceful sentence completion before cutting off buffered audio from the previous turn. Improves overall user experience with more natural speech flow and interruptions.

This would make life much easier :)

[khorwood-openai]

I've forwarded this to the Realtime team! They can sync if there are any updates here.

[Stevenic]

Nice feature request... I suspect it'll be difficult for them todo as it doesn't look like there's a 1:1 mapping of audio to text deltas. They're close but i've seen different lengths...