search

pipecat-ai / pipecat

Open Source framework for voice and multimodal conversational AI
BSD 2-Clause "Simplified" License
3.38k stars 325 forks source link

Daily join timeout / Deepgram STT timeout #457

Closed bigahega closed 1 month ago

bigahega commented 2 months ago

Sometimes daily sdk raises a timeout warning while joining the room. In this case if daily transcriptions are enabled, nothing gets transcribed.

A similar issue happens when deepgram TTS is used explicitly without enabling the daily transcription. If joining the room times out, deepgram sdk raises timeout errors and does not transcribe anything further even if daily later on joins the room successfully.

I'll add logs in short.

bigahega commented 2 months ago

First daily sdk prints the following log:

ERROR | pipecat.transports.services.daily:join:316 - Time out joining https://******.daily.co/uuqM706DW6gGDtPRw72P

and right after this log, deepgram sdk prints the following:

ConnectionClosed in AsyncListenWebSocketClient._listening with code 1011: Deepgram did not receive audio data or a text message within the timeout window. See https://dpgr.am/net0001 async_client.py :403 2024-09-13 12:33:52,825 ConnectionClosed in AsyncListenWebSocketClient._listening with code 1011: Deepgram did not receive audio data or a text message within the timeout window. See https://dpgr.am/net0001

this is for the case daily.co + deepgram stt

For the daily.co and its own transcription case, first log is the same; daily errors out saying joining the room timed out and that's it. In a few seconds it connects but it does not emit any transcription.

golbin commented 2 months ago

Maybe, I have the same issus.

2024-09-14 15:46:20.742 | DEBUG    | pipecat.processors.frame_processor:link:169 - Linking Source#0 -> Pipeline#0
2024-09-14 15:46:20.742 | DEBUG    | pipecat.pipeline.runner:run:28 - Runner PipelineRunner#0 started running PipelineTask#0
2024-09-14 15:46:20.742 | INFO     | pipecat.transports.services.daily:join:281 - Joining https://---.daily.co/---
2024-09-14 15:46:22.078 | INFO     | pipecat.transports.services.daily:join:303 - Joined https://---.daily.co/---
WebSocketException in AsyncListenWebSocketClient.start:
2024-09-14 15:46:32.084 | ERROR    | pipecat.services.deepgram:start:149 - DeepgramSTTService#0: Unable to connect to Deepgram
bigahega commented 2 months ago

I have seen the same deepgram log this morning.

"message":"2024-09-16 07:45:32.345 | ERROR | pipecat.services.deepgram:_connect:171 - DeepgramSTTService#0: Unable to connect to Deepgram\n"

Is it maybe possible to add a retry mechanism? Daily.co retries joining automatically but if deepgram fails, it's a dead end.

bigahega commented 1 month ago

After much digging around, I found out this was due to some of my other code blocking the asyncio event loop whilst joining the room.

xpluscal commented 1 month ago

Hey @bigahega could you add some detail how you fixed the issue? Having the same issue.

xpluscal commented 1 month ago

Digging deeper, it seems to be that deepgram returns a 400 error "Bad Request" on the ones sent from pipecat.