search

Teekeks / pyTwitchAPI

A Python 3.7 compatible implementation of the Twitch API, EventSub, PubSub and Chat
https://pytwitchapi.dev
MIT License
256 stars 38 forks source link

When internet is down, library will eventually fail to connect without notifying client #318

Open adhawkins opened 4 months ago

adhawkins commented 4 months ago

I have a chat bot based on pytwitchapi. My internet was down for a few hours recently, and it was only when I happened to notice the bot wasn't working when I realised that it had failed to connect to the Twitch API (understandably) and gave up attempting to reconnect:

Jul 17 01:55:13 xcp-dev python[808]: Reached timeout for websocket receive, will attempt a reconnect
Jul 17 01:55:16 xcp-dev python[808]: connection attempt failed, retry in 0s...
Jul 17 01:55:20 xcp-dev python[808]: connection attempt failed, retry in 1s...
Jul 17 01:55:21 xcp-dev python[808]: connection attempt failed, retry in 2s...
Jul 17 01:55:23 xcp-dev python[808]: connection attempt failed, retry in 4s...
Jul 17 01:55:27 xcp-dev python[808]: connection attempt failed, retry in 8s...
Jul 17 01:55:39 xcp-dev python[808]: connection attempt failed, retry in 16s...
Jul 17 01:55:58 xcp-dev python[808]: connection attempt failed, retry in 32s...
Jul 17 01:56:33 xcp-dev python[808]: connection attempt failed, retry in 64s...
Jul 17 01:56:50 xcp-dev python[808]: Reached timeout for websocket receive, will attempt a reconnect
Jul 17 01:56:53 xcp-dev python[808]: connection attempt failed, retry in 0s...
Jul 17 01:56:57 xcp-dev python[808]: connection attempt failed, retry in 1s...
Jul 17 01:56:58 xcp-dev python[808]: connection attempt failed, retry in 2s...
Jul 17 01:57:00 xcp-dev python[808]: connection attempt failed, retry in 4s...
Jul 17 01:57:04 xcp-dev python[808]: connection attempt failed, retry in 8s...
Jul 17 01:57:15 xcp-dev python[808]: connection attempt failed, retry in 16s...
Jul 17 01:57:35 xcp-dev python[808]: connection attempt failed, retry in 32s...
Jul 17 01:57:40 xcp-dev python[808]: connection attempt failed, retry in 128s...
Jul 17 01:58:10 xcp-dev python[808]: connection attempt failed, retry in 64s...
Jul 17 01:59:17 xcp-dev python[808]: connection attempt failed, retry in 128s...
Jul 17 01:59:48 xcp-dev python[808]: Connection to chat websocket lost and unable to reestablish connection!
Jul 17 01:59:48 xcp-dev python[808]: Traceback (most recent call last):
Jul 17 01:59:48 xcp-dev python[808]:   File "/home/andy/software/pytwitchchatlogger/venv/lib/python3.11/site-packages/aiohttp/client_ws.py", line 244, in receive
Jul 17 01:59:48 xcp-dev python[808]:     msg = await self._reader.read()
Jul 17 01:59:48 xcp-dev python[808]:           ^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 17 01:59:48 xcp-dev python[808]:   File "/home/andy/software/pytwitchchatlogger/venv/lib/python3.11/site-packages/aiohttp/streams.py", line 681, in read
Jul 17 01:59:48 xcp-dev python[808]:     return await super().read()
Jul 17 01:59:48 xcp-dev python[808]:            ^^^^^^^^^^^^^^^^^^^^
Jul 17 01:59:48 xcp-dev python[808]:   File "/home/andy/software/pytwitchchatlogger/venv/lib/python3.11/site-packages/aiohttp/streams.py", line 640, in read
Jul 17 01:59:48 xcp-dev python[808]:     await self._waiter
Jul 17 01:59:48 xcp-dev python[808]: asyncio.exceptions.CancelledError
Jul 17 01:59:48 xcp-dev python[808]: During handling of the above exception, another exception occurred:
Jul 17 01:59:48 xcp-dev python[808]: Traceback (most recent call last):
Jul 17 01:59:48 xcp-dev python[808]:   File "/home/andy/software/pytwitchchatlogger/venv/lib/python3.11/site-packages/twitchAPI/chat/__init__.py", line 913, in __task_receive
Jul 17 01:59:48 xcp-dev python[808]:     message = await self.__connection.receive(timeout=receive_timeout)
Jul 17 01:59:48 xcp-dev python[808]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 17 01:59:48 xcp-dev python[808]:   File "/home/andy/software/pytwitchchatlogger/venv/lib/python3.11/site-packages/aiohttp/client_ws.py", line 243, in receive
Jul 17 01:59:48 xcp-dev python[808]:     async with async_timeout.timeout(timeout or self._receive_timeout):
Jul 17 01:59:48 xcp-dev python[808]:   File "/usr/lib/python3.11/asyncio/timeouts.py", line 98, in __aexit__
Jul 17 01:59:48 xcp-dev python[808]:     raise TimeoutError
Jul 17 01:59:48 xcp-dev python[808]: TimeoutError
Jul 17 01:59:48 xcp-dev python[808]: During handling of the above exception, another exception occurred:
Jul 17 01:59:48 xcp-dev python[808]: Traceback (most recent call last):
Jul 17 01:59:48 xcp-dev python[808]:   File "/home/andy/software/pytwitchchatlogger/venv/lib/python3.11/site-packages/twitchAPI/chat/__init__.py", line 918, in __task_receive
Jul 17 01:59:48 xcp-dev python[808]:     await self._handle_base_reconnect()
Jul 17 01:59:48 xcp-dev python[808]:   File "/home/andy/software/pytwitchchatlogger/venv/lib/python3.11/site-packages/twitchAPI/chat/__init__.py", line 956, in _handle_base_reconnect
Jul 17 01:59:48 xcp-dev python[808]:     await self.__connect(is_startup=False)
Jul 17 01:59:48 xcp-dev python[808]:   File "/home/andy/software/pytwitchchatlogger/venv/lib/python3.11/site-packages/twitchAPI/chat/__init__.py", line 866, in __connect
Jul 17 01:59:48 xcp-dev python[808]:     raise TwitchBackendException('can\'t connect')
Jul 17 01:59:48 xcp-dev python[808]: twitchAPI.type.TwitchBackendException: can't connect
Jul 17 02:01:25 xcp-dev python[808]: Connection to chat websocket lost and unable to reestablish connection!
Jul 17 02:01:25 xcp-dev python[808]: Traceback (most recent call last):
Jul 17 02:01:25 xcp-dev python[808]:   File "/home/andy/software/pytwitchchatlogger/venv/lib/python3.11/site-packages/aiohttp/client_ws.py", line 244, in receive
Jul 17 02:01:25 xcp-dev python[808]:     msg = await self._reader.read()
Jul 17 02:01:25 xcp-dev python[808]:           ^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 17 02:01:25 xcp-dev python[808]:   File "/home/andy/software/pytwitchchatlogger/venv/lib/python3.11/site-packages/aiohttp/streams.py", line 681, in read
Jul 17 02:01:25 xcp-dev python[808]:     return await super().read()
Jul 17 02:01:25 xcp-dev python[808]:            ^^^^^^^^^^^^^^^^^^^^
Jul 17 02:01:25 xcp-dev python[808]:   File "/home/andy/software/pytwitchchatlogger/venv/lib/python3.11/site-packages/aiohttp/streams.py", line 640, in read
Jul 17 02:01:25 xcp-dev python[808]:     await self._waiter
Jul 17 02:01:25 xcp-dev python[808]: asyncio.exceptions.CancelledError
Jul 17 02:01:25 xcp-dev python[808]: During handling of the above exception, another exception occurred:
Jul 17 02:01:25 xcp-dev python[808]: Traceback (most recent call last):
Jul 17 02:01:25 xcp-dev python[808]:   File "/home/andy/software/pytwitchchatlogger/venv/lib/python3.11/site-packages/twitchAPI/chat/__init__.py", line 913, in __task_receive
Jul 17 02:01:25 xcp-dev python[808]:     message = await self.__connection.receive(timeout=receive_timeout)
Jul 17 02:01:25 xcp-dev python[808]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 17 02:01:25 xcp-dev python[808]:   File "/home/andy/software/pytwitchchatlogger/venv/lib/python3.11/site-packages/aiohttp/client_ws.py", line 243, in receive
Jul 17 02:01:25 xcp-dev python[808]:     async with async_timeout.timeout(timeout or self._receive_timeout):
Jul 17 02:01:25 xcp-dev python[808]:   File "/usr/lib/python3.11/asyncio/timeouts.py", line 98, in __aexit__
Jul 17 02:01:25 xcp-dev python[808]:     raise TimeoutError
Jul 17 02:01:25 xcp-dev python[808]: TimeoutError
Jul 17 02:01:25 xcp-dev python[808]: During handling of the above exception, another exception occurred:
Jul 17 02:01:25 xcp-dev python[808]: Traceback (most recent call last):
Jul 17 02:01:25 xcp-dev python[808]:   File "/home/andy/software/pytwitchchatlogger/venv/lib/python3.11/site-packages/twitchAPI/chat/__init__.py", line 918, in __task_receive
Jul 17 02:01:25 xcp-dev python[808]:     await self._handle_base_reconnect()
Jul 17 02:01:25 xcp-dev python[808]:   File "/home/andy/software/pytwitchchatlogger/venv/lib/python3.11/site-packages/twitchAPI/chat/__init__.py", line 956, in _handle_base_reconnect
Jul 17 02:01:25 xcp-dev python[808]:     await self.__connect(is_startup=False)
Jul 17 02:01:25 xcp-dev python[808]:   File "/home/andy/software/pytwitchchatlogger/venv/lib/python3.11/site-packages/twitchAPI/chat/__init__.py", line 866, in __connect
Jul 17 02:01:25 xcp-dev python[808]:     raise TwitchBackendException('can\'t connect')
Jul 17 02:01:25 xcp-dev python[808]: twitchAPI.type.TwitchBackendException: can't connect
Jul 18 10:22:28 xcp-dev systemd[751]: Stopping pytwitchchatlogger.service...

There apparently isn't a way that this exception could be caught so that the client can tell the library to initiate a re-connect.

Posting here at Lena's request (from Discord)

iturdikulov commented 1 month ago

Workaround, we can increase reconnect_delay_steps, to some high value. This is will increase chance to reconnect automatically. https://github.com/Teekeks/pyTwitchAPI/blob/2b90657491108cec172578afa0967c88282eddd0/twitchAPI/chat/__init__.py#L589C9-L589C81

Modify Chat instance, some like this:

chat = await Chat(twitch)
reconnect_steps = [i*i for i in range(50)]  # 11 hours
print(f"Hours to reconnect: {sum(reconnect_steps) // 3600}")  # replace to logger
chat.reconnect_delay_steps = reconnect_steps

I not verified this, but it can work.

There apparently isn't a way that this exception could be caught

So except TwitchBackendException: ... not working? According to logs this is possible.

Teekeks commented 1 month ago

correct, thats what that var is for