search

nickjj / docker-django-example

A production ready example Django app that's using Docker and Docker Compose.
MIT License
1.2k stars 259 forks source link

Celery worker crashes #12

Closed vic-cieslak closed 2 years ago

vic-cieslak commented 2 years ago

Hi, Fresh build, didn't modified anything.

Ubuntu 21.10 Docker version 20.10.12, build e91ed57 docker-compose version 1.29.2, build 5becea4c

worker_1    | [2022-03-01 10:38:31,399: CRITICAL/MainProcess] Unrecoverable error: TypeError('Channel._connparams.<locals>.Connection.disconnect() takes 1 positional argument but 2 were given')
worker_1    | Traceback (most recent call last):
worker_1    |   File "/home/python/.local/lib/python3.10/site-packages/kombu/transport/virtual/base.py", line 923, in create_channel
worker_1    |     return self._avail_channels.pop()
worker_1    | IndexError: pop from empty list
worker_1    | 
worker_1    | During handling of the above exception, another exception occurred:
worker_1    | 
worker_1    | Traceback (most recent call last):
worker_1    |   File "/home/python/.local/lib/python3.10/site-packages/redis/retry.py", line 45, in call_with_retry
worker_1    |     return do()
worker_1    |   File "/home/python/.local/lib/python3.10/site-packages/redis/connection.py", line 608, in <lambda>
worker_1    |     lambda: self._connect(), lambda error: self.disconnect(error)
worker_1    |   File "/home/python/.local/lib/python3.10/site-packages/redis/connection.py", line 673, in _connect
worker_1    |     raise err
worker_1    |   File "/home/python/.local/lib/python3.10/site-packages/redis/connection.py", line 661, in _connect
worker_1    |     sock.connect(socket_address)
worker_1    | TimeoutError: [Errno 110] Connection timed out
worker_1    | 
worker_1    | During handling of the above exception, another exception occurred:
worker_1    | 
worker_1    | Traceback (most recent call last):
worker_1    |   File "/home/python/.local/lib/python3.10/site-packages/celery/worker/worker.py", line 203, in start
worker_1    |     self.blueprint.start(self)
worker_1    |   File "/home/python/.local/lib/python3.10/site-packages/celery/bootsteps.py", line 116, in start
worker_1    |     step.start(parent)
worker_1    |   File "/home/python/.local/lib/python3.10/site-packages/celery/bootsteps.py", line 365, in start
worker_1    |     return self.obj.start()
worker_1    |   File "/home/python/.local/lib/python3.10/site-packages/celery/worker/consumer/consumer.py", line 326, in start
worker_1    |     blueprint.start(self)
worker_1    |   File "/home/python/.local/lib/python3.10/site-packages/celery/bootsteps.py", line 116, in start
worker_1    |     step.start(parent)
worker_1    |   File "/home/python/.local/lib/python3.10/site-packages/celery/worker/consumer/connection.py", line 21, in start
worker_1    |     c.connection = c.connect()
worker_1    |   File "/home/python/.local/lib/python3.10/site-packages/celery/worker/consumer/consumer.py", line 422, in connect
worker_1    |     conn = self.connection_for_read(heartbeat=self.amqheartbeat)
worker_1    |   File "/home/python/.local/lib/python3.10/site-packages/celery/worker/consumer/consumer.py", line 428, in connection_for_read
worker_1    |     return self.ensure_connected(
worker_1    |   File "/home/python/.local/lib/python3.10/site-packages/celery/worker/consumer/consumer.py", line 454, in ensure_connected
worker_1    |     conn = conn.ensure_connection(
worker_1    |   File "/home/python/.local/lib/python3.10/site-packages/kombu/connection.py", line 382, in ensure_connection
worker_1    |     self._ensure_connection(*args, **kwargs)
worker_1    |   File "/home/python/.local/lib/python3.10/site-packages/kombu/connection.py", line 434, in _ensure_connection
worker_1    |     return retry_over_time(
worker_1    |   File "/home/python/.local/lib/python3.10/site-packages/kombu/utils/functional.py", line 312, in retry_over_time
worker_1    |     return fun(*args, **kwargs)
worker_1    |   File "/home/python/.local/lib/python3.10/site-packages/kombu/connection.py", line 878, in _connection_factory
worker_1    |     self._connection = self._establish_connection()
worker_1    |   File "/home/python/.local/lib/python3.10/site-packages/kombu/connection.py", line 813, in _establish_connection
worker_1    |     conn = self.transport.establish_connection()
worker_1    |   File "/home/python/.local/lib/python3.10/site-packages/kombu/transport/virtual/base.py", line 947, in establish_connection
worker_1    |     self._avail_channels.append(self.create_channel(self))
worker_1    |   File "/home/python/.local/lib/python3.10/site-packages/kombu/transport/virtual/base.py", line 925, in create_channel
worker_1    |     channel = self.Channel(connection)
worker_1    |   File "/home/python/.local/lib/python3.10/site-packages/kombu/transport/redis.py", line 679, in __init__
worker_1    |     self.client.ping()
worker_1    |   File "/home/python/.local/lib/python3.10/site-packages/redis/commands/core.py", line 954, in ping
worker_1    |     return self.execute_command("PING", **kwargs)
worker_1    |   File "/home/python/.local/lib/python3.10/site-packages/redis/client.py", line 1173, in execute_command
worker_1    |     conn = self.connection or pool.get_connection(command_name, **options)
worker_1    |   File "/home/python/.local/lib/python3.10/site-packages/redis/connection.py", line 1370, in get_connection
worker_1    |     connection.connect()
worker_1    |   File "/home/python/.local/lib/python3.10/site-packages/redis/connection.py", line 607, in connect
worker_1    |     sock = self.retry.call_with_retry(
worker_1    |   File "/home/python/.local/lib/python3.10/site-packages/redis/retry.py", line 48, in call_with_retry
worker_1    |     fail(error)
worker_1    |   File "/home/python/.local/lib/python3.10/site-packages/redis/connection.py", line 608, in <lambda>
worker_1    |     lambda: self._connect(), lambda error: self.disconnect(error)
worker_1    | TypeError: Channel._connparams.<locals>.Connection.disconnect() takes 1 positional argument but 2 were given
hellodjango_worker_1 exited with code 0

any clues?

nickjj commented 2 years ago

Hi,

Based on:

worker_1    |     sock.connect(socket_address)
worker_1    | TimeoutError: [Errno 110] Connection timed out

It looks like maybe it can't connect to Redis.

After you docker-compose up does Redis start up successfully?

vic-cieslak commented 2 years ago

yep, it was availability issue. It's working fine now, thanks! Great stuff :+1: