Azure-Samples / aoai-realtime-audio-sdk

Azure OpenAI code resources for using gpt-4o-realtime capabilities.
MIT License
586 stars 89 forks source link

Stop playing the audio before the end of the text and do not generate the complete text. #26

Open AnderRedna opened 2 weeks ago

AnderRedna commented 2 weeks ago

Please provide us with the following information:

This issue is for a: (mark with an x)

- [X] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

Use javascript sample -> Click on record -> say "Tell me a long history" or another input to get a larger response -> The audio will stop in the middle of audio and the text will not generate complete text

Any log messages given by the failure

In the console will appear: { "type": "response.content_part.done", "event_id": "event_AEnf0tlKz1cUHJPbRnBCR", "response_id": "resp_AEnepctGRAxjngM5HsEZd", "item_id": "item_AEnepHVWZY51yDgcnjDt8", "output_index": 0, "content_index": 0, "content": { "type": "audio", "transcript": "Once upon a magical autumn evening, a lone leaf clung to a branch high in an ancient oak tree. The leaf, whose name was Rusty, was adorned in vibrant hues of red and orange. Rusty had watched the days grow shorter and the winds grow crisper, knowing it was nearly time for the grand adventure.\n\nEvery autumn, the leaves of the great oak would take flight on the wind, swirling and dancing through the air, to explore the wide world beyond their tree. Rusty had been dreaming of this journey since he was a bud in the spring.\n\n\"Hold on tight!\" the wind howled playfully, shaking the branches with a gust.\n\nRusty quivered with excitement and a hint of nervousness. \"I'm ready!\" he called back, feeling the wind's cool embrace.\n\nWith one final breath, the wind swept through, and Rusty let go, soaring into the sky. He twisted and" }, "part": { "type": "audio", "transcript": "Once upon a magical autumn evening, a lone leaf clung to a branch high in an ancient oak tree. The leaf, whose name was Rusty, was adorned in vibrant hues of red and orange. Rusty had watched the days grow shorter and the winds grow crisper, knowing it was nearly time for the grand adventure.\n\nEvery autumn, the leaves of the great oak would take flight on the wind, swirling and dancing through the air, to explore the wide world beyond their tree. Rusty had been dreaming of this journey since he was a bud in the spring.\n\n\"Hold on tight!\" the wind howled playfully, shaking the branches with a gust.\n\nRusty quivered with excitement and a hint of nervousness. \"I'm ready!\" he called back, feeling the wind's cool embrace.\n\nWith one final breath, the wind swept through, and Rusty let go, soaring into the sky. He twisted and" } } main.ts:102 { "type": "response.output_item.done", "event_id": "event_AEnf0RwRyJF4hQK4XWtuL", "response_id": "resp_AEnepctGRAxjngM5HsEZd", "output_index": 0, "item": { "id": "item_AEnepHVWZY51yDgcnjDt8", "object": "realtime.item", "type": "message", "status": "incomplete", "role": "assistant", "content": [ { "type": "audio", "transcript": "Once upon a magical autumn evening, a lone leaf clung to a branch high in an ancient oak tree. The leaf, whose name was Rusty, was adorned in vibrant hues of red and orange. Rusty had watched the days grow shorter and the winds grow crisper, knowing it was nearly time for the grand adventure.\n\nEvery autumn, the leaves of the great oak would take flight on the wind, swirling and dancing through the air, to explore the wide world beyond their tree. Rusty had been dreaming of this journey since he was a bud in the spring.\n\n\"Hold on tight!\" the wind howled playfully, shaking the branches with a gust.\n\nRusty quivered with excitement and a hint of nervousness. \"I'm ready!\" he called back, feeling the wind's cool embrace.\n\nWith one final breath, the wind swept through, and Rusty let go, soaring into the sky. He twisted and" } ] } } main.ts:102 response.done

Expected/desired behavior

Generate completed text and speak all text

OS and Version?

Windows 11

Versions

0.0.0

Mention any other details that might be useful


Thanks! We'll be in touch soon.

trrwilson commented 2 weeks ago

Thanks, @AnderRedna! I've confirmed with the AOAI service team that this is a known, tracked issue with the current model deployment -- output audio and transcripts can be "cut off" midway, often along with a supplementary error message containing the rather cryptic "Response parsing interrupted" detail in the payload. The problem reproduces readily independently of client (including the playground) and occurs right around the 1200-1400 token mark.

This problem will be addressed ASAP with a deployment fix. In the interim, limiting response output (resuming across responses is fine) is the best workaround.

AnderRedna commented 2 weeks ago

Thanks for the response! How can i limite response output? Im currently using the js sample.

trrwilson commented 2 weeks ago

As silly as it sounds, the model is surprisingly receptive to being asked (via user input, be it audio or text) to keep it short. E.g. here's some incremental output for the long story using the interactive .NET sample (applicable everywhere):

 <<< Start of speech detected
 <<< End of speech detected
 >>> USER: Give me the first two sentences of a long story.

Once upon a time, in a mystical forest shrouded in mist, there was an ancient oak tree that whispered secrets of the past to those who listened closely. Legends spoke of a hidden portal within its roots that could transport one to realms beyond imagination.
 <<< Start of speech detected
 <<< End of speech detected
 >>> USER: Give me the next two sentences.

Many had tried to find it, but only the pure of heart and those daring enough to seek the unknown could uncover its secrets. The forest was alive with the hum of magic, and the air buzzed with tales waiting to be told.
 <<< Start of speech detected
 <<< End of speech detected
 >>> USER: the next two.

One day, a curious young adventurer named Elara, driven by tales of this enchanted portal, set forth on a journey to uncover its mysteries. With a heart full of wonder and a spirit unyielding, she ventured into the depths of the forest, guided by the whispers of the ancient oak. Little did she know, her quest would lead her to discoveries beyond her wildest dreams.