search

django / channels_redis

Redis channel layer backend for Django Channels
BSD 3-Clause "New" or "Revised" License
602 stars 197 forks source link

aioredis.errors.ConnectionClosedError: Reader at end of file #303

Closed samul-1 closed 2 years ago

samul-1 commented 2 years ago

channels==3.0.4 channels-redis==3.4.0 redis==6.2.6 running inside a docker container

Settings:

CHANNEL_LAYERS = {
    "default": {
        "BACKEND": "channels_redis.core.RedisChannelLayer",
        "CONFIG": {"hosts": [("localhost", 5673)]},
    },
}

As soon as I try to connect to one of the consumers in my application, I get this:

WebSocket HANDSHAKING /ws/events/ [127.0.0.1:59323]
WebSocket CONNECT /ws/events/ [127.0.0.1:59323]
Exception inside application: Reader at end of file
Traceback (most recent call last):
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/asgiref/sync.py", line 266, in main_wrap
    raise exc_info[1]
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/asgiref/sync.py", line 451, in thread_handler
    raise exc_info[1]
  File "/Users/myuser/Desktop/platform/backend/courses/consumers.py", line 69, in subscribe_instance
    response = await super().subscribe_instance(request_id, **kwargs)
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/djangochannelsrestframework/observer/generics.py", line 98, in subscribe_instance
    groups = set(await self.handle_instance_change.subscribe(instance=instance))
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/djangochannelsrestframework/observer/base_observer.py", line 144, in subscribe
    await consumer.add_group(group_name)
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/djangochannelsrestframework/consumers.py", line 84, in add_group
    await self.channel_layer.group_add(name, self.channel_name)
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/channels_redis/core.py", line 646, in group_add
    async with self.connection(self.consistent_hash(group)) as connection:
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/channels_redis/core.py", line 902, in __aenter__
    self.conn = await self.pool.pop()
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/channels_redis/core.py", line 93, in pop
    conn = await self.create_conn(loop)
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/channels_redis/core.py", line 79, in create_conn
    return await aioredis.create_redis_pool(**kwargs)
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/aioredis/commands/__init__.py", line 188, in create_redis_pool
    pool = await create_pool(address, db=db,
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/aioredis/pool.py", line 58, in create_pool
    await pool._fill_free(override_min=False)
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/aioredis/pool.py", line 386, in _fill_free
    await conn.execute('ping')
aioredis.errors.ConnectionClosedError: Reader at end of file

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/channels/utils.py", line 51, in await_many_dispatch
    await dispatch(result)
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/channels/consumer.py", line 73, in dispatch
    await handler(message)
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/channels/generic/websocket.py", line 194, in websocket_receive
    await self.receive(text_data=message["text"])
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/channels/generic/websocket.py", line 257, in receive
    await self.receive_json(await self.decode_json(text_data), **kwargs)
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/djangochannelsrestframework/consumers.py", line 180, in receive_json
    await self.handle_action(action, request_id=request_id, **content)
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/djangochannelsrestframework/consumers.py", line 175, in handle_action
    await self.handle_exception(exc, action=action, request_id=request_id)
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/djangochannelsrestframework/consumers.py", line 135, in handle_exception
    raise exc
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/djangochannelsrestframework/consumers.py", line 168, in handle_action
    response = await method(request_id=request_id, action=action, **kwargs)
  File "/Users/myuser/Desktop/platform/backend/courses/consumers.py", line 72, in subscribe_instance
    await database_sync_to_async(self.unlock_instance_or_give_up)(pk)
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/asgiref/sync.py", line 414, in __call__
    ret = await asyncio.wait_for(future, timeout=None)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/asyncio/tasks.py", line 455, in wait_for
    return await fut
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/channels/db.py", line 13, in thread_handler
    return super().thread_handler(loop, *args, **kwargs)
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/asgiref/sync.py", line 453, in thread_handler
    return func(*args, **kwargs)
  File "/Users/myuser/Desktop/platform/backend/courses/consumers.py", line 53, in unlock_instance_or_give_up
    return self.queryset.get(pk=pk).unlock(self.scope["user"])
  File "/Users/myuser/Desktop/platform/backend/courses/abstract_models.py", line 254, in unlock
    self.save(update_fields=["locked_by", "last_lock_update"])
  File "/Users/myuser/Desktop/platform/backend/courses/models.py", line 525, in save
    return super().save(*args, **kwargs)
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/django/db/models/base.py", line 806, in save
    self.save_base(
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/django/db/models/base.py", line 872, in save_base
    post_save.send(
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 176, in send
    return [
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 177, in <listcomp>
    (receiver, receiver(signal=self, sender=sender, **named))
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/djangochannelsrestframework/observer/model_observer.py", line 100, in post_save_receiver
    self.database_event(instance, Action.UPDATE)
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/djangochannelsrestframework/observer/model_observer.py", line 117, in database_event
    connection.on_commit(partial(self.post_change_receiver, instance, action))
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/django/db/backends/base/base.py", line 684, in on_commit
    func()
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/djangochannelsrestframework/observer/model_observer.py", line 140, in post_change_receiver
    self.send_messages(
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/djangochannelsrestframework/observer/model_observer.py", line 163, in send_messages
    async_to_sync(channel_layer.group_send)(group_name, message_to_send)
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/asgiref/sync.py", line 204, in __call__
    return call_result.result()
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/asgiref/sync.py", line 268, in main_wrap
    result = await self.awaitable(*args, **kwargs)
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/channels_redis/core.py", line 671, in group_send
    async with self.connection(self.consistent_hash(group)) as connection:
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/channels_redis/core.py", line 902, in __aenter__
    self.conn = await self.pool.pop()
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/channels_redis/core.py", line 93, in pop
    conn = await self.create_conn(loop)
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/channels_redis/core.py", line 79, in create_conn
    return await aioredis.create_redis_pool(**kwargs)
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/aioredis/commands/__init__.py", line 188, in create_redis_pool
    pool = await create_pool(address, db=db,
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/aioredis/pool.py", line 58, in create_pool
    await pool._fill_free(override_min=False)
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/aioredis/pool.py", line 386, in _fill_free
    await conn.execute('ping')
aioredis.errors.ConnectionClosedError: Reader at end of file

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/channels/staticfiles.py", line 44, in __call__
    return await self.application(scope, receive, send)
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/channels/routing.py", line 71, in __call__
    return await application(scope, receive, send)
  File "/Users/myuser/Desktop/platform/backend/core/middleware.py", line 26, in __call__
    return await super().__call__(scope, receive, send)
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/channels/middleware.py", line 26, in __call__
    return await self.inner(scope, receive, send)
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/channels/routing.py", line 150, in __call__
    return await application(
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/channels/consumer.py", line 94, in app
    return await consumer(scope, receive, send)
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/channels/consumer.py", line 58, in __call__
    await await_many_dispatch(
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/channels/utils.py", line 58, in await_many_dispatch
    await task
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/channels_redis/core.py", line 492, in receive
    message_channel, message = await self.receive_single(
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/channels_redis/core.py", line 547, in receive_single
    content = await self._brpop_with_clean(
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/channels_redis/core.py", line 380, in _brpop_with_clean
    async with self.connection(index) as connection:
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/channels_redis/core.py", line 902, in __aenter__
    self.conn = await self.pool.pop()
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/channels_redis/core.py", line 93, in pop
    conn = await self.create_conn(loop)
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/channels_redis/core.py", line 79, in create_conn
    return await aioredis.create_redis_pool(**kwargs)
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/aioredis/commands/__init__.py", line 188, in create_redis_pool
    pool = await create_pool(address, db=db,
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/aioredis/pool.py", line 58, in create_pool
    await pool._fill_free(override_min=False)
  File "/Users/myuser/.local/share/virtualenvs/backend-tRSrcCVT/lib/python3.8/site-packages/aioredis/pool.py", line 386, in _fill_free
    await conn.execute('ping')
aioredis.errors.ConnectionClosedError: Reader at end of file
WebSocket DISCONNECT /ws/events/ [127.0.0.1:59323]

As you can see, the connection to the consumer does happen, and the error happens a few milliseconds afterwards.

All of the consumers I have tried this with work fine if I use the InMemoryChannelLayer instead of redis channels layer.

Any input is highly appreciated.

carltongibson commented 2 years ago

This looks like a redis connection issue. Not something I can say anything about I'm afraid.