django / channels_redis

Redis channel layer backend for Django Channels
BSD 3-Clause "New" or "Revised" License
602 stars 197 forks source link

Timeout Error 110, Can reconnections be handled gracefully? #323

Closed paulBlackburn closed 2 years ago

paulBlackburn commented 2 years ago

I'm using version 2.3.2 of channels-redis and python 3.6.8. I have an instance of Azure Cache for Redis. A few times a year, failovers occur. From Azure "A failover occurs when a primary node in the cache is taken offline for routine maintenance, and a replica node is promoted to replace it". When this happens, I have to restart the Django server and restart daphne. Is there any way for channels reddis to gracefully handle these timeouts. When these timeouts occur no new connections can be made until I restart. The stack trace below is from await self.channel_layer.group_send.

File "/home/ubuntu/.virtualenvs/venv/lib/python3.6/site-packages/channels_redis/core.py", line 611, in group_send key, min=0, max=int(time.time()) - self.group_expiry File "/home/ubuntu/.virtualenvs/venv/lib/python3.6/site-packages/aioredis/connection.py", line 186, in _read_data obj = await self._reader.readobj() File "/home/ubuntu/.virtualenvs/venv/lib/python3.6/site-packages/aioredis/stream.py", line 102, in readobj await self._wait_for_data('readobj') File "/usr/lib/python3.6/asyncio/streams.py", line 464, in _wait_for_data yield from self._waiter File "/usr/lib/python3.6/asyncio/selector_events.py", line 714, in _read_ready data = self._sock.recv(self.max_size) TimeoutError: [Errno 110] Connection timed out

python3.6 daphne -b 127.0.0.1 -p 9000 asgi:application python3.6 gunicorn wsgi:application -c gunicorn.conf.py

Do any newer versions of channels-redis gracefully handle this issue? I can't recreate this issue because it is dependent on azure performing routine maintenance.

carltongibson commented 2 years ago

This would be dependent on the underlying client library.

Aioredis, and we're just in the process of moving to redis-py.

You'd need to enquire there. It's not a channels-redis per se.