search

deepgram / deepgram-python-sdk

Official Python SDK for Deepgram's automated speech recognition APIs.
https://developers.deepgram.com
MIT License
178 stars 48 forks source link

Fatal error on SSL transport #332

Closed motapinto closed 3 months ago

motapinto commented 3 months ago

What is the current behavior?

Deepgram seems to stop working after running live transcription script for some time, usually accompanied by an "Fatal error on SSL transport" error. This is delaying a wider release for live products.

Steps to reproduce

I leave transcription script that handles transcription running for some time and eventually that error appears. Can share entire code privately. The client is already initialized with config = DeepgramClientOptions(verbose=logging.WARN, options={"keepalive": "true"}). I tested this and keep alive message is sent even without starting microphone, so not sure why in "Error message type 2" I get 1011 websocket error. Is it necessary to send keep alive even with those options or can this be another problem?

Please tell us about your environment

Other information

Error message type 1

Exception in callback _ProactorBaseWritePipeTransport._loop_writing(<_OverlappedF...hed result=32>) handle: <Handle _ProactorBaseWritePipeTransport._loop_writing(<_OverlappedF...hed result=32>)> Traceback (most recent call last): File "C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\events.py", line 80, in _run self._context.run(self._callback, *self._args) File "C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\proactor_events.py", line 381, in _loop_writing assert f is self._write_fut AssertionError Fatal error on SSL transport protocol: <asyncio.sslproto.SSLProtocol object at 0x00000216DD3799F0> transport: <_ProactorSocketTransport fd=688 read=<_OverlappedFuture pending overlapped=<pending, 0x216dd36a680> cb=[_ProactorReadPipeTransport._loop_reading()]> write=<_OverlappedFuture finished result=32> write_bufsize=16440> Traceback (most recent call last): File "C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\sslproto.py", line 703, in _process_write_backlog del self._write_backlog[0] IndexError: deque index out of range data transfer failed Traceback (most recent call last): File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 963, in transfer_data message = await self.read_message() File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 1033, in read_message frame = await self.read_data_frame(max_size=self.max_size) File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 1108, in read_data_frame frame = await self.read_frame(max_size) File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 1165, in read_frame frame = await Frame.read( File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\framing.py", line 68, in read data = await reader(2) File "C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\streams.py", line 708, in readexactly await self._wait_for_data('readexactly') File "C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\streams.py", line 501, in _wait_for_data await self._waiter File "C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\sslproto.py", line 703, in _process_write_backlog del self._write_backlog[0] IndexError: deque index out of range WebSocket connection closed with code 1006: Error while sending: no close frame received or sent Traceback (most recent call last): File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 963, in transfer_data message = await self.read_message() File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 1033, in read_message frame = await self.read_data_frame(max_size=self.max_size) File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 1108, in read_data_frame frame = await self.read_frame(max_size) File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 1165, in read_frame frame = await Frame.read( File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\framing.py", line 68, in read data = await reader(2) File "C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\streams.py", line 708, in readexactly await self._wait_for_data('readexactly') File "C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\streams.py", line 501, in _wait_for_data await self._waiter File "C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\sslproto.py", line 703, in _process_write_backlog del self._write_backlog[0] IndexError: deque index out of range

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\deepgram\audio\microphone\microphone.py", line 135, in _callback self.push_callback(input_data) File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\deepgram\audio\microphone\microphone.py", line 54, in ).result() File "C:\Users\admin\anaconda3\envs\deepgram\lib\concurrent\futures_base.py", line 458, in result return self.get_result() File "C:\Users\admin\anaconda3\envs\deepgram\lib\concurrent\futures_base.py", line 403, in get_result raise self._exception File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\deepgram\clients\live\v1\async_client.py", line 265, in send cnt = await self._socket.send(data) File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 635, in send await self.ensure_open() File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 939, in ensure_open raise self.connection_closed_exc() websockets.exceptions.ConnectionClosedError: no close frame received or sent unhandled exception during asyncio.run() shutdown task: <Task finished name='Task-2' coro=<DeepgramTranscriptionWorker.transcribe() done, defined at C:\Users\admin\Documents\v4.2\src\transcriptions\deepgram.py:43> exception=ConnectionClosedError(None, None, None)> Traceback (most recent call last): File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 963, in transfer_data message = await self.read_message() File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 1033, in read_message frame = await self.read_data_frame(max_size=self.max_size) File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 1108, in read_data_frame frame = await self.read_frame(max_size) File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 1165, in read_frame frame = await Frame.read( File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\framing.py", line 68, in read data = await reader(2) File "C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\streams.py", line 708, in readexactly await self._wait_for_data('readexactly') File "C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\streams.py", line 501, in _wait_for_data await self._waiter File "C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\sslproto.py", line 703, in _process_write_backlog del self._write_backlog[0] IndexError: deque index out of range

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "C:\Users\admin\Documents\v4.2\src\main.py", line 37, in main await main_task File "C:\Users\admin\Documents\v4.2\src\transcriptions\deepgram.py", line 58, in transcribe await self.close() File "C:\Users\admin\Documents\v4.2\src\transcriptions\deepgram.py", line 84, in close await self.connection.finish() File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\deepgram\clients\live\v1\async_client.py", line 282, in finish await self._socket.send(json.dumps({"type": "CloseStream"})) File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 635, in send await self.ensure_open() File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 939, in ensure_open raise self.connection_closed_exc() websockets.exceptions.ConnectionClosedError: no close frame received or sent unhandled exception during asyncio.run() shutdown task: <Task finished name='Task-1' coro=<main() done, defined at C:\Users\admin\Documents\v4.2\src\main.py:13> exception=ConnectionClosedError(None, None, None)> Traceback (most recent call last): File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 963, in transfer_data message = await self.read_message() File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 1033, in read_message frame = await self.read_data_frame(max_size=self.max_size) File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 1108, in read_data_frame frame = await self.read_frame(max_size) File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 1165, in read_frame frame = await Frame.read( File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\framing.py", line 68, in read data = await reader(2) File "C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\streams.py", line 708, in readexactly await self._wait_for_data('readexactly') File "C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\streams.py", line 501, in _wait_for_data await self._waiter File "C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\sslproto.py", line 703, in _process_write_backlog del self._write_backlog[0] IndexError: deque index out of range

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "C:\Users\admin\Documents\v4.2\src\main.py", line 37, in main await main_task File "C:\Users\admin\Documents\v4.2\src\transcriptions\deepgram.py", line 58, in transcribe await self.close() File "C:\Users\admin\Documents\v4.2\src\transcriptions\deepgram.py", line 84, in close await self.connection.finish() File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\deepgram\clients\live\v1\async_client.py", line 282, in finish await self._socket.send(json.dumps({"type": "CloseStream"})) File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 635, in send await self.ensure_open() File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 939, in ensure_open raise self.connection_closed_exc() websockets.exceptions.ConnectionClosedError: no close frame received or sent Traceback (most recent call last): File "C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\runners.py", line 44, in run return loop.run_until_complete(main) File "C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\base_events.py", line 636, in run_until_complete self.run_forever() File "C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\windows_events.py", line 321, in run_forever super().run_forever() File "C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\base_events.py", line 603, in run_forever self._run_once() File "C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\base_events.py", line 1871, in _run_once event_list = self._selector.select(timeout) File "C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\windows_events.py", line 444, in select self._poll(timeout) File "C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\windows_events.py", line 797, in _poll status = _overlapped.GetQueuedCompletionStatus(self._iocp, ms) TypeError: ConnectionClosed.init() missing 2 required positional arguments: 'rcvd' and 'sent'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users\admin\Documents\v4.2\src\main.py", line 61, in asyncio.run(main(args.config, transcription_api_key, translation_api_key, translation_api_region, logger)) File "C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\runners.py", line 52, in run loop.close() File "C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\proactor_events.py", line 692, in close self._proactor.close() File "C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\windows_events.py", line 887, in close self._poll(msg_update) File "C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\windows_events.py", line 797, in _poll status = _overlapped.GetQueuedCompletionStatus(self._iocp, ms) KeyboardInterrupt Exception ignored in: <function BaseEventLoop.del at 0x00000216DB7F6710> Traceback (most recent call last): File "C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\base_events.py", line 690, in del File "C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\proactor_events.py", line 691, in close File "C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\proactor_events.py", line 759, in _close_self_pipe

Error message type 2

Error while sending: Task <Task pending name='Task-39' coro=<AsyncLiveClient.send() running at C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\deepgram\clients\live\v1\async_client.py:265> cb=[_chain_future.._call_set_state() at C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\futures.py:392]> got Future attached to a different loop

WebSocket connection closed with code 1011: Deepgram did not receive audio data or a text message within the timeout window. See https://dpgr.am/net0001 Traceback (most recent call last): File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\deepgram\audio\microphone\microphone.py", line 135, in _callback self.push_callback(input_data) File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\deepgram\audio\microphone\microphone.py", line 54, in ).result() File "C:\Users\admin\anaconda3\envs\deepgram\lib\concurrent\futures_base.py", line 458, in result return self.get_result() File "C:\Users\admin\anaconda3\envs\deepgram\lib\concurrent\futures_base.py", line 403, in get_result raise self._exception File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\deepgram\clients\live\v1\async_client.py", line 265, in send cnt = await self._socket.send(data) File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 647, in send await self.write_frame(True, opcode, data) File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 1209, in write_frame await self.drain() File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 1192, in drain await self._drain() File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 344, in _drain await self._drain_helper() File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 327, in _drain_helper await waiter RuntimeError: Task <Task pending name='Task-39' coro=<AsyncLiveClient.send() running at C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\deepgram\clients\live\v1\async_client.py:265> cb=[_chain_future.._call_set_state() at C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\futures.py:392]> got Future attached to a different loop Traceback (most recent call last): File "C:\Users\admin\Documents\v4.2\src\main.py", line 61, in asyncio.run(main(args.config, transcription_api_key, translation_api_key, translation_api_region, logger)) File "C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\runners.py", line 44, in run return loop.run_until_complete(main) File "C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\base_events.py", line 649, in run_until_complete return future.result() File "C:\Users\admin\Documents\v4.2\src\main.py", line 37, in main await main_task File "C:\Users\admin\Documents\v4.2\src\transcriptions\deepgram.py", line 58, in transcribe await self.close() File "C:\Users\admin\Documents\v4.2\src\transcriptions\deepgram.py", line 84, in close await self.connection.finish() File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\deepgram\clients\live\v1\async_client.py", line 282, in finish await self._socket.send(json.dumps({"type": "CloseStream"})) File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 635, in send await self.ensure_open() File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 939, in ensure_open raise self.connection_closed_exc() websockets.exceptions.ConnectionClosedError: received 1011 (internal error) Deepgram did not receive audio data or a text message within the timeout window. See https://dpgr.am/net0001; then sent 1011 (internal error) Deepgram did not receive audio data or a text message within the timeout window. See https://dpgr.am/net0001

Error message type 3

Error while sending: Task <Task pending name='Task-495' coro=<AsyncLiveClient.send() running at C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\deepgram\clients\live\v1\async_client.py:265> cb=[_chain_future.._call_set_state() at C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\futures.py:392]> got Future attached to a different loop

Traceback (most recent call last): File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\deepgram\audio\microphone\microphone.py", line 135, in _callback self.push_callback(input_data) File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\deepgram\audio\microphone\microphone.py", line 54, in ).result() File "C:\Users\admin\anaconda3\envs\deepgram\lib\concurrent\futures_base.py", line 458, in result return self.get_result() File "C:\Users\admin\anaconda3\envs\deepgram\lib\concurrent\futures_base.py", line 403, in get_result raise self._exception File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\deepgram\clients\live\v1\async_client.py", line 265, in send cnt = await self._socket.send(data) File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 647, in send await self.write_frame(True, opcode, data) File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 1209, in write_frame await self.drain() File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 1192, in drain await self._drain() File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 344, in _drain await self._drain_helper() File "C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\websockets\legacy\protocol.py", line 327, in _drain_helper await waiter RuntimeError: Task <Task pending name='Task-495' coro=<AsyncLiveClient.send() running at C:\Users\admin\anaconda3\envs\deepgram\lib\site-packages\deepgram\clients\live\v1\async_client.py:265> cb=[_chain_future.._call_set_state() at C:\Users\admin\anaconda3\envs\deepgram\lib\asyncio\futures.py:392]> got Future attached to a different loop

dvonthenen commented 3 months ago

Hi @motapinto

Thanks for providing all this info. Will need a second to sift through it. I can answer some of your initial comments though in the meantime.

I leave transcription script that handles transcription running for some time and eventually that error appears. Can share entire code privately.

If you can log into Discord and share, that would be very helpful.

I tested this and keep alive message is sent even without starting microphone, so not sure why in "Error message type 2" I get 1011 websocket error.

If you have KeepAlive=True, the Live Client will send keepalive messages continuously. The purpose of the message is when audio data from your source is not streaming through the client, the keepalive message will prevent DG from closing the socket. The 1011 close socket could happen though for other unforeseen reasons, however.

Is it necessary to send keep alive even with those options or can this be another problem?

If you aren't sending data and want to hold the DG connection open, then you need to enable the keepalive. The other option is that if you don't have the option set, you might want to control the "I'm not sending any data". In other words, you might prefer the disconnect, and if your application still needs the transcription because the process isn't done, you re-create the client for a reconnect. The advantage here is that this client has a fresh state.

Will take a look through the logs and drop more info here.

dvonthenen commented 3 months ago

For my notekeeping, this thread in Discord is the same problem: https://discord.com/channels/1108042150941294664/1214170879647289416

dvonthenen commented 3 months ago

Via discord, we ended up finding the problem for anyone else that is interested. the problem is:

in the meantime, these are just examples to demonstrate functionality and don't affect the core of the client code