search

Bogdanp / django_dramatiq

A Django app that integrates with Dramatiq.
https://dramatiq.io
Other
331 stars 77 forks source link

Error running manage.py collectstatic #43

Closed th0th closed 5 years ago

th0th commented 5 years ago

First of all, thanks for the awesome project.

I feel like title to this issue should be: "Actor somehow tries to run when manage.py collectstatic is run" but I didn't go with it since I am not sure that's the issue.

I am trying replace celery with dramatiq on an existing project. I started with installing dramatiq, django_dramatiq, updating settings.py and replacing celery decorators with @dramatiq.actor. But when I add @dramatiq.actor decorator to any function and try to run collectstatic I get something like this:

Step 14/20 : RUN python3 ./manage.py collectstatic --no-input
 ---> Running in fd28d8437e31
[ERROR] pika.adapters.utils.selector_ioloop_adapter: Address resolution failed: gaierror(-2, 'Name does not resolve')
[ERROR] pika.adapters.utils.connection_workflow: getaddrinfo failed: gaierror(-2, 'Name does not resolve').
[ERROR] pika.adapters.utils.connection_workflow: AMQP connection workflow failed: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - gaierror(-2, 'Name does not resolve'); first exception - None.
[ERROR] pika.adapters.utils.connection_workflow: AMQPConnectionWorkflow - reporting failure: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - gaierror(-2, 'Name does not resolve'); first exception - None
[ERROR] pika.adapters.blocking_connection: Connection workflow failed: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - gaierror(-2, 'Name does not resolve'); first exception - None
[ERROR] pika.adapters.blocking_connection: Error in _create_connection().
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 451, in _create_connection
    raise self._reap_last_connection_workflow_error(error)
  File "/usr/lib/python3.6/site-packages/pika/adapters/utils/selector_ioloop_adapter.py", line 564, in _resolve
    self._flags)
  File "/usr/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name does not resolve
Traceback (most recent call last):
  File "./manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python3.6/site-packages/django/core/management/__init__.py", line 357, in execute
    django.setup()
  File "/usr/lib/python3.6/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/usr/lib/python3.6/site-packages/django/apps/registry.py", line 114, in populate
    app_config.import_models()
  File "/usr/lib/python3.6/site-packages/django/apps/config.py", line 211, in import_models
    self.models_module = import_module(models_module_name)
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/project/apps/app/models.py", line 20, in <module>
    from .tasks.task import run_task
  File "/project/apps/app/tasks/task.py", line 10, in <module>
    def run_task(serialized_data, kind, down_validations=0):
  File "/usr/lib/python3.6/site-packages/dramatiq/actor.py", line 225, in actor
    return decorator(fn)
  File "/usr/lib/python3.6/site-packages/dramatiq/actor.py", line 220, in decorator
    priority=priority, broker=broker, options=options,
  File "/usr/lib/python3.6/site-packages/dramatiq/actor.py", line 52, in __init__
    self.broker.declare_actor(self)
  File "/usr/lib/python3.6/site-packages/dramatiq/broker.py", line 174, in declare_actor
    self.declare_queue(actor.queue_name)
  File "/usr/lib/python3.6/site-packages/dramatiq/brokers/rabbitmq.py", line 199, in declare_queue
    self._declare_queue(queue_name)
  File "/usr/lib/python3.6/site-packages/dramatiq/brokers/rabbitmq.py", line 238, in _declare_queue
    return self.channel.queue_declare(queue=queue_name, durable=True, arguments=arguments)
  File "/usr/lib/python3.6/site-packages/dramatiq/brokers/rabbitmq.py", line 131, in channel
    channel = self.state.channel = self.connection.channel()
  File "/usr/lib/python3.6/site-packages/dramatiq/brokers/rabbitmq.py", line 111, in connection
    parameters=self.parameters)
  File "/usr/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 360, in __init__
    self._impl = self._create_connection(parameters, _impl_class)
  File "/usr/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 451, in _create_connection
    raise self._reap_last_connection_workflow_error(error)
  File "/usr/lib/python3.6/site-packages/pika/adapters/utils/selector_ioloop_adapter.py", line 564, in _resolve
    self._flags)
  File "/usr/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name does not resolve

The rabbitmq server is not accessible during the build. If I delete the @dramatiq.actor decorator it runs without any errors.

Bogdanp commented 5 years ago

Thank you for the kind words!

This is somewhat expected. Dramatiq actors try to eagerly declare themselves as soon as the code runs. You can work around it by setting the broker to StubBroker during the build, perhaps by using an environment variable.

th0th commented 5 years ago

That feels strange but I got your point, thanks for the explanation @Bogdanp đź‘Ť

rpkilby commented 5 years ago

Does this also fail with redis, or just with rmq? I don't recall running into this in the last project I used, but it's been a while.

Bogdanp commented 5 years ago

@rpkilby Redis queues don't need to be pre-declared so this doesn't happen with Redis (see https://github.com/Bogdanp/dramatiq/blob/master/dramatiq/brokers/redis.py#L127-L141).