lamby / django-slack

Slack integration for Django, using the templating engine to generate messages
https://django-slack.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
241 stars 57 forks source link

Problem with celery #97

Closed martingaldeca closed 3 years ago

martingaldeca commented 4 years ago

Hi, I'm using this slack integration in my project. I have all dockerized, and the main application container works fine, but, I'm also using celery in the project, and with celery version 5.0.0 I see this error in logs in the worker containers:

Traceback (most recent call last):
  File "/usr/local/bin/celery", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/celery/__main__.py", line 15, in main
    sys.exit(_main())
  File "/usr/local/lib/python3.9/site-packages/celery/bin/celery.py", line 150, in main
    return celery(auto_envvar_prefix="CELERY")
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/celery/bin/worker.py", line 323, in worker
    worker = app.Worker(
  File "/usr/local/lib/python3.9/site-packages/celery/worker/worker.py", line 94, in __init__
    self.app.loader.init_worker()
  File "/usr/local/lib/python3.9/site-packages/celery/loaders/base.py", line 111, in init_worker
    self.import_default_modules()
  File "/usr/local/lib/python3.9/site-packages/celery/loaders/base.py", line 105, in import_default_modules
    raise response
  File "/usr/local/lib/python3.9/site-packages/celery/utils/dispatch/signal.py", line 276, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/usr/local/lib/python3.9/site-packages/vine/promises.py", line 160, in __call__
    return self.throw()
  File "/usr/local/lib/python3.9/site-packages/vine/promises.py", line 157, in __call__
    retval = fun(*final_args, **final_kwargs)
  File "/usr/local/lib/python3.9/site-packages/celery/app/base.py", line 640, in _autodiscover_tasks
    return self._autodiscover_tasks_from_fixups(related_name)
  File "/usr/local/lib/python3.9/site-packages/celery/app/base.py", line 649, in _autodiscover_tasks_from_fixups
    return self._autodiscover_tasks_from_names([
  File "/usr/local/lib/python3.9/site-packages/celery/app/base.py", line 644, in _autodiscover_tasks_from_names
    return self.loader.autodiscover_tasks(
  File "/usr/local/lib/python3.9/site-packages/celery/loaders/base.py", line 219, in autodiscover_tasks
    mod.__name__ for mod in autodiscover_tasks(packages or (),
  File "/usr/local/lib/python3.9/site-packages/celery/loaders/base.py", line 245, in autodiscover_tasks
    return [find_related_module(pkg, related_name) for pkg in packages]
  File "/usr/local/lib/python3.9/site-packages/celery/loaders/base.py", line 245, in <listcomp>
    return [find_related_module(pkg, related_name) for pkg in packages]
  File "/usr/local/lib/python3.9/site-packages/celery/loaders/base.py", line 270, in find_related_module
    raise e
  File "/usr/local/lib/python3.9/site-packages/celery/loaders/base.py", line 266, in find_related_module
    return importlib.import_module(module_name)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/site-packages/django_slack/tasks.py", line 1, in <module>
    from celery import task
  File "/usr/local/lib/python3.9/site-packages/celery/local.py", line 497, in __getattr__
    module = __import__(self._direct[name], None, None, [name])
ModuleNotFoundError: No module named 'celery.task'
lamby commented 4 years ago

It seems like Celery from 5.x upwards uses a global-ish variable, app, to setup tasks:

https://docs.celeryproject.org/en/stable/getting-started/first-steps-with-celery.html#application

Do you have app elsewhere in your Django application...? Wondering what the best way that third-party applications can get hold of this app instance, or if there is another integration method altogether.

martingaldeca commented 4 years ago

Hi!, I changed the variable name app to celery_app in my code, and checked that I'm not using it anywhere, but still failling.

For me, the solution was fix celery version to 4.4.7 and kombu to 4.6.11, and with this version it works fine.

clokep commented 4 years ago

I think you can use shared_task if you don't have access to the application object?

martingaldeca commented 4 years ago

I think you can use shared_task if you don't have access to the application object?

It is still failing, even with the shared_tasks decorator

vindemasi commented 4 years ago

@lamby @martingaldeca IMHO it could be related with this: https://github.com/celery/celery/pull/6315

https://github.com/lamby/django-slack/blob/409521f427a68d12ad7e7ac3e19d517d2ba131ba/django_slack/tasks.py#L1

lociii commented 3 years ago

See also https://github.com/celery/celery/issues/6406

Just use shared_task as a replacement for task https://docs.celeryproject.org/en/stable/django/first-steps-with-django.html#using-the-shared-task-decorator

lamby commented 3 years ago

Fixed in https://github.com/lamby/django-slack/commit/3996517db688a390bccb159f17acebe791dc608c - thanks all.