search

celery / django-celery

Old Celery integration project for Django
http://celery.github.com/django-celery
BSD 3-Clause "New" or "Revised" License
1.53k stars 453 forks source link

django-celery >= 3.0 and django-configurations don't work #186

Closed jeanlucmongrain closed 10 years ago

jeanlucmongrain commented 12 years ago

With celery 2.5 it used to work. now that I upgraded to latest version it fail, because django configure itself using the traditional way:

 importlib.import_module(self.SETTINGS_MODULE)

which django-configurations don't allow.

I was able to fix my problem by adding the line 14 and 15 to loaders.py:

 10 from celery import signals
 11 from celery.loaders.base import BaseLoader
 12 from celery.datastructures import DictAttribute
 13 
 14 from configurations import importer
 15 importer.install()
 16 
 17 import django
 18 from django import db

is there a way to hook something in django-celery before loaders import django.db?

complete stacktrace:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/patate/sandbox/sg/boilerplate/env/lib/python2.7/site-packages/billiard/forking.py", line 525, in main
    self = load(from_parent)
  File "/sw/lib/python2.7/pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "/sw/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/sw/lib/python2.7/pickle.py", line 1133, in load_reduce
    value = func(*args)
  File "/Users/patate/sandbox/sg/boilerplate/env/lib/python2.7/site-packages/celery/app/utils.py", line 129, in _unpickle_app
    return pickler()(cls, *args)
  File "/Users/patate/sandbox/sg/boilerplate/env/lib/python2.7/site-packages/celery/app/utils.py", line 108, in __call__
    self.prepare(app, **kwargs)
  File "/Users/patate/sandbox/sg/boilerplate/env/lib/python2.7/site-packages/celery/app/utils.py", line 112, in prepare
    app.conf.update(kwargs['changes'])
  File "/Users/patate/sandbox/sg/boilerplate/env/lib/python2.7/site-packages/kombu/utils/__init__.py", line 294, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/Users/patate/sandbox/sg/boilerplate/env/lib/python2.7/site-packages/celery/app/base.py", line 449, in conf
    return self._get_config()
  File "/Users/patate/sandbox/sg/boilerplate/env/lib/python2.7/site-packages/celery/app/base.py", line 320, in _get_config
    s = Settings({}, [self.prepare_config(self.loader.conf),
  File "/Users/patate/sandbox/sg/boilerplate/env/lib/python2.7/site-packages/kombu/utils/__init__.py", line 294, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/Users/patate/sandbox/sg/boilerplate/env/lib/python2.7/site-packages/celery/app/base.py", line 461, in loader
    return get_loader_cls(self.loader_cls)(app=self)
  File "/Users/patate/sandbox/sg/boilerplate/env/lib/python2.7/site-packages/celery/loaders/__init__.py", line 23, in get_loader_cls
    return symbol_by_name(loader, LOADER_ALIASES)
  File "/Users/patate/sandbox/sg/boilerplate/env/lib/python2.7/site-packages/kombu/utils/__init__.py", line 81, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/sw/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/Users/patate/sandbox/sg/boilerplate/env/lib/python2.7/site-packages/djcelery/loaders.py", line 15, in <module>
    from django import db
  File "/Users/patate/sandbox/sg/boilerplate/env/lib/python2.7/site-packages/django/db/__init__.py", line 14, in <module>
    if not settings.DATABASES:
  File "/Users/patate/sandbox/sg/boilerplate/env/lib/python2.7/site-packages/django/utils/functional.py", line 276, in __getattr__
    self._setup()
  File "/Users/patate/sandbox/sg/boilerplate/env/lib/python2.7/site-packages/django/conf/__init__.py", line 42, in _setup
    self._wrapped = Settings(settings_module)
  File "/Users/patate/sandbox/sg/boilerplate/env/lib/python2.7/site-packages/django/conf/__init__.py", line 90, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/Users/patate/sandbox/sg/boilerplate/env/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/Users/patate/sandbox/sg/boilerplate/boilerplate/settings/local.py", line 13, in <module>
    from boilerplate.settings.dev import DevProjectSettings
  File "/Users/patate/sandbox/sg/boilerplate/boilerplate/settings/dev.py", line 7, in <module>
    from boilerplate.settings.base import BoilerBaseSettings
  File "/Users/patate/sandbox/sg/boilerplate/boilerplate/settings/base.py", line 16, in <module>
    class BoilerBaseSettings(Settings):
  File "/Users/patate/sandbox/sg/boilerplate/env/lib/python2.7/site-packages/configurations/base.py", line 23, in __new__
    raise ImproperlyConfigured(install_failure)
django.core.exceptions.ImproperlyConfigured: django-configurations settings importer wasn't correctly installed. Please use one of the starter functions to install it as mentioned in the docs: http://django-configurations.readthedocs.org/
jeanlucmongrain commented 12 years ago

just to make it easier to troubleshoot this, I created this project that reproduce the problem:

https://github.com/bclermont/django-celery-issue186

ask commented 12 years ago

Django has no defined entrypoint, but you could add it to settings.py.

If settings.py is loaded after djcelery/loaders.py then you should find a way to avoid that

jeanlucmongrain commented 12 years ago

I just spent the few last hours reading and trying working with entrypoint.

this is something new for me.

I noticed that celery.load_extension_commands try to load entry points that start with celery.commands, is that related?

If not please, can you point me where to look to do this? an example or something?

ask commented 12 years ago

That's exactly it, Django doesn't have a specific entry-point. The only place you can add stuff like this is in settings.py.

In your settings you should have import djcelery; djcelery.setup_loader(), and it's important that djcelery/loaders is not imported before settings.py.

ask commented 10 years ago

Since Celery 3.1 django-celery is no longer necessary and Django users can now use the normal Celery API!

See the new tutorial at: http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html

Adding the stuff you want here is now possible, using the example from the tutorial you can add it to the proj/celery/celery.py module:

UPDATED: There is now an example in the django-configurations documentation available here: http://django-configurations.readthedocs.org/en/latest/cookbook/#celery

jezdez commented 10 years ago

See http://django-configurations.readthedocs.org/en/latest/cookbook/#celery for an updated example (which is not how @ask explains it here).