Problem with celery

[martingaldeca]

Hi, I'm using this slack integration in my project. I have all dockerized, and the main application container works fine, but, I'm also using celery in the project, and with celery version 5.0.0 I see this error in logs in the worker containers:

Traceback (most recent call last):
  File "/usr/local/bin/celery", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/celery/__main__.py", line 15, in main
    sys.exit(_main())
  File "/usr/local/lib/python3.9/site-packages/celery/bin/celery.py", line 150, in main
    return celery(auto_envvar_prefix="CELERY")
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/celery/bin/worker.py", line 323, in worker
    worker = app.Worker(
  File "/usr/local/lib/python3.9/site-packages/celery/worker/worker.py", line 94, in __init__
    self.app.loader.init_worker()
  File "/usr/local/lib/python3.9/site-packages/celery/loaders/base.py", line 111, in init_worker
    self.import_default_modules()
  File "/usr/local/lib/python3.9/site-packages/celery/loaders/base.py", line 105, in import_default_modules
    raise response
  File "/usr/local/lib/python3.9/site-packages/celery/utils/dispatch/signal.py", line 276, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/usr/local/lib/python3.9/site-packages/vine/promises.py", line 160, in __call__
    return self.throw()
  File "/usr/local/lib/python3.9/site-packages/vine/promises.py", line 157, in __call__
    retval = fun(*final_args, **final_kwargs)
  File "/usr/local/lib/python3.9/site-packages/celery/app/base.py", line 640, in _autodiscover_tasks
    return self._autodiscover_tasks_from_fixups(related_name)
  File "/usr/local/lib/python3.9/site-packages/celery/app/base.py", line 649, in _autodiscover_tasks_from_fixups
    return self._autodiscover_tasks_from_names([
  File "/usr/local/lib/python3.9/site-packages/celery/app/base.py", line 644, in _autodiscover_tasks_from_names
    return self.loader.autodiscover_tasks(
  File "/usr/local/lib/python3.9/site-packages/celery/loaders/base.py", line 219, in autodiscover_tasks
    mod.__name__ for mod in autodiscover_tasks(packages or (),
  File "/usr/local/lib/python3.9/site-packages/celery/loaders/base.py", line 245, in autodiscover_tasks
    return [find_related_module(pkg, related_name) for pkg in packages]
  File "/usr/local/lib/python3.9/site-packages/celery/loaders/base.py", line 245, in <listcomp>
    return [find_related_module(pkg, related_name) for pkg in packages]
  File "/usr/local/lib/python3.9/site-packages/celery/loaders/base.py", line 270, in find_related_module
    raise e
  File "/usr/local/lib/python3.9/site-packages/celery/loaders/base.py", line 266, in find_related_module
    return importlib.import_module(module_name)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/site-packages/django_slack/tasks.py", line 1, in <module>
    from celery import task
  File "/usr/local/lib/python3.9/site-packages/celery/local.py", line 497, in __getattr__
    module = __import__(self._direct[name], None, None, [name])
ModuleNotFoundError: No module named 'celery.task'

[lamby]

It seems like Celery from 5.x upwards uses a global-ish variable, app, to setup tasks:

https://docs.celeryproject.org/en/stable/getting-started/first-steps-with-celery.html#application

Do you have app elsewhere in your Django application...? Wondering what the best way that third-party applications can get hold of this app instance, or if there is another integration method altogether.

[martingaldeca]

Hi!, I changed the variable name app to celery_app in my code, and checked that I'm not using it anywhere, but still failling.

For me, the solution was fix celery version to 4.4.7 and kombu to 4.6.11, and with this version it works fine.

[clokep]

I think you can use shared_task if you don't have access to the application object?

[martingaldeca]

I think you can use shared_task if you don't have access to the application object?

It is still failing, even with the shared_tasks decorator

[vindemasi]

@lamby @martingaldeca IMHO it could be related with this: https://github.com/celery/celery/pull/6315

https://github.com/lamby/django-slack/blob/409521f427a68d12ad7e7ac3e19d517d2ba131ba/django_slack/tasks.py#L1

[lociii]

See also https://github.com/celery/celery/issues/6406

Just use shared_task as a replacement for task https://docs.celeryproject.org/en/stable/django/first-steps-with-django.html#using-the-shared-task-decorator

[lamby]

Fixed in https://github.com/lamby/django-slack/commit/3996517db688a390bccb159f17acebe791dc608c - thanks all.