Describe the bug
The Google STT connection fails with an error indicating that the RPC operation was cancelled, causing retries and eventually resulting in an InvalidStateError for an already finished RPC, somtime it intterupt the stt, it don't here
Environment: local dev
Versions:
livekit-agents: >=0.10.2,<0.11
livekit-api: >=0.7.1,<0.8
livekit-plugins-google: >=0.7.1,<0.8
livekit-plugins-silero: >=0.7.2,<0.8
livekit-plugins-openai: >=0.10.4,<0.11
python >=3.12
To Reproduce
Steps to reproduce the behavior:
Integrated TTS and STT Google plugins to LiveKit.
Run the setup and connect clients.
Observe the error during Google STT operation.
Expected behavior
Google STT should successfully establish a connection without the operation being cancelled, and should handle streaming correctly.
Log for the error
2024-11-02 11:54:50,604 - WARNING livekit.plugins.google - google stt connection failed, retrying in 0s Traceback (most recent call last): File "/home/sunsilk/livekit_agents/.pixi/envs/default/lib/python3.12/site-packages/google/api_core/grpc_helpers_async.py", line 106, in _wrapped_aiter async for response in self._call: # pragma: no branch File "/home/sunsilk/livekit_agents/.pixi/envs/default/lib/python3.12/site-packages/grpc/aio/_call.py", line 365, in _fetch_stream_responses await self._raise_for_status() File "/home/sunsilk/livekit_agents/.pixi/envs/default/lib/python3.12/site-packages/grpc/aio/_call.py", line 272, in _raise_for_status raise _create_rpc_error( grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with: status = StatusCode.CANCELLED details = "The operation was cancelled." debug_error_string = "UNKNOWN
received from peer ipv6:%5B2404:6800:4003:c03::5f%5D:443 {created_time:"2024-11-02T11:54:50.603802009+07:00", grpc_status:1, grpc_message:"The operation was cancelled."}"
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/home/sunsilk/livekit_agents/.pixi/envs/default/lib/python3.12/site-packages/livekit/plugins/google/stt.py", line 259, in _run await self._run_stream(stream) File "/home/sunsilk/livekit_agents/.pixi/envs/default/lib/python3.12/site-packages/livekit/plugins/google/stt.py", line 279, in _run_stream async for resp in stream: File "/home/sunsilk/livekit_agents/.pixi/envs/default/lib/python3.12/site-packages/google/api_core/grpc_helpers_async.py", line 109, in _wrapped_aiter raise exceptions.from_grpc_error(rpc_error) from rpc_error google.api_core.exceptions.InternalServerError: 500 Internal error from Core {"pid": 428442, "job_id": "AJ_TJ3xoiGxCZHf"}
davidzhao
commented
2 weeks ago
Describe the bug The Google STT connection fails with an error indicating that the RPC operation was cancelled, causing retries and eventually resulting in an InvalidStateError for an already finished RPC, somtime it intterupt the stt, it don't here
To Reproduce Steps to reproduce the behavior:
Expected behavior Google STT should successfully establish a connection without the operation being cancelled, and should handle streaming correctly.
Log for the error 2024-11-02 11:54:50,604 - WARNING livekit.plugins.google - google stt connection failed, retrying in 0s Traceback (most recent call last): File "/home/sunsilk/livekit_agents/.pixi/envs/default/lib/python3.12/site-packages/google/api_core/grpc_helpers_async.py", line 106, in _wrapped_aiter async for response in self._call: # pragma: no branch File "/home/sunsilk/livekit_agents/.pixi/envs/default/lib/python3.12/site-packages/grpc/aio/_call.py", line 365, in _fetch_stream_responses await self._raise_for_status() File "/home/sunsilk/livekit_agents/.pixi/envs/default/lib/python3.12/site-packages/grpc/aio/_call.py", line 272, in _raise_for_status raise _create_rpc_error( grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with: status = StatusCode.CANCELLED details = "The operation was cancelled." debug_error_string = "UNKNOWN received from peer ipv6:%5B2404:6800:4003:c03::5f%5D:443 {created_time:"2024-11-02T11:54:50.603802009+07:00", grpc_status:1, grpc_message:"The operation was cancelled."}"
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/home/sunsilk/livekit_agents/.pixi/envs/default/lib/python3.12/site-packages/livekit/plugins/google/stt.py", line 259, in _run await self._run_stream(stream) File "/home/sunsilk/livekit_agents/.pixi/envs/default/lib/python3.12/site-packages/livekit/plugins/google/stt.py", line 279, in _run_stream async for resp in stream: File "/home/sunsilk/livekit_agents/.pixi/envs/default/lib/python3.12/site-packages/google/api_core/grpc_helpers_async.py", line 109, in _wrapped_aiter raise exceptions.from_grpc_error(rpc_error) from rpc_error google.api_core.exceptions.InternalServerError: 500 Internal error from Core {"pid": 428442, "job_id": "AJ_TJ3xoiGxCZHf"}
...
asyncio.exceptions.InvalidStateError: RPC already finished.
Additional context This issue may be
relatedto improper handling of stream states in the Google STT plugin or maybe google issuses