paulhoule / tentacruel

MIT License
0 stars 0 forks source link

pinger: recover from hue partial crash #41

Open paulhoule opened 5 years ago

paulhoule commented 5 years ago

A while back there was a partial crash of the pinger:

Apr 29 19:23:56 tamamo pinger[24058]: Task exception was never retrieved
Apr 29 19:23:56 tamamo pinger[24058]: future: <Task finished coro=<hue_loop() done, defined at /home/paul/src/cron/tentacruel/tentacruel/pinger/hue_ping.py:137> exception=ServerDisconnectedError(None)>
Apr 29 19:23:56 tamamo pinger[24058]: Traceback (most recent call last):
Apr 29 19:23:56 tamamo pinger[24058]:   File "/home/paul/src/cron/tentacruel/tentacruel/pinger/hue_ping.py", line 145, in hue_loop
Apr 29 19:23:56 tamamo pinger[24058]:     await pinger.poll()
Apr 29 19:23:56 tamamo pinger[24058]:   File "/home/paul/src/cron/tentacruel/tentacruel/pinger/hue_ping.py", line 79, in poll
Apr 29 19:23:56 tamamo pinger[24058]:     lights = json.loads(await self._hue.get_lights())
Apr 29 19:23:56 tamamo pinger[24058]:   File "/home/paul/src/cron/tentacruel/tentacruel/pinger/hue_ping.py", line 45, in get_lights
Apr 29 19:23:56 tamamo pinger[24058]:     async with self._session.get(url) as response:
Apr 29 19:23:56 tamamo pinger[24058]:   File "/home/paul/.cache/pypoetry/virtualenvs/tentacruel-py3.7/lib/python3.7/site-packages/aiohttp/client.py", line 1005, in __aenter__
Apr 29 19:23:56 tamamo pinger[24058]:     self._resp = await self._coro
Apr 29 19:23:56 tamamo pinger[24058]:   File "/home/paul/.cache/pypoetry/virtualenvs/tentacruel-py3.7/lib/python3.7/site-packages/aiohttp/client.py", line 497, in _request
Apr 29 19:23:56 tamamo pinger[24058]:     await resp.start(conn)
Apr 29 19:23:56 tamamo pinger[24058]:   File "/home/paul/.cache/pypoetry/virtualenvs/tentacruel-py3.7/lib/python3.7/site-packages/aiohttp/client_reqrep.py", line 844, in start
Apr 29 19:23:56 tamamo pinger[24058]:     message, payload = await self._protocol.read()  # type: ignore  # noqa
Apr 29 19:23:56 tamamo pinger[24058]:   File "/home/paul/.cache/pypoetry/virtualenvs/tentacruel-py3.7/lib/python3.7/site-packages/aiohttp/streams.py", line 588, in read
Apr 29 19:23:56 tamamo pinger[24058]:     await self._waiter
Apr 29 19:23:56 tamamo pinger[24058]: aiohttp.client_exceptions.ServerDisconnectedError: None

Note that the pinger has multiple asyncio tasks... in a case like this when we can't get through to the hue hub we ought to restart the thread. I am going to kick the process tonight but tomorrow I need to put in a recovery proc