Task fail with ImportError: No module named 'django-db'

mounirmesselmeni commented 7 years ago

Current configuration

Celery report:

software -> celery:4.0.2 (latentcall) kombu:4.0.2 py:3.5.2 billiard:3.5.0.2 py-amqp:2.1.4 platform -> system:Linux arch:64bit, ELF imp:CPython loader -> celery.loaders.app.AppLoader settings -> transport:amqp results:django-db

CELERYD_HIJACK_ROOT_LOGGER: False CELERY_RESULT_BACKEND: 'django-db' CELERY_RESULT_PERSISTENT: True CELERY_IGNORE_RESULT: True CELERY_TASK_RESULT_EXPIRES: datetime.timedelta(7) CELERY_QUEUE_HA_POLICY: 'all' CELERY_BROKER_URL: 'amqp://project:****@***:5672/project' CELERY_WORKER_DIRECT: False CELERY_TIMEZONE: 'Europe/Berlin' CELERY_SEND_EVENTS: False CELERY_TRACK_STARTED: True

Content of celery.py:

import os
from celery import Celery
from celery.schedules import crontab

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings')

app = Celery('project')

app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

tasks.py:

@celery_app.task(autoretry_for=(Exception,), retry_kwargs={'max_retries': 5})
def revoke_x(name, id2, id2):
    return

Steps to reproduce

I'm not able to find out how to reproduce it

Expected behavior

Task running

Actual behavior

It's a weird behavior, as it's not failing always, I have many others tasks running with no problems, this problem happen sometimes but only with this task, and I'm not able to understand what's the source of the issue. Error happening before executing the tasks (When trying to send it to the queue) The call: revoke_x.apply_async(['param1', 1, 1])

traceback: ImportError: No module named 'django-db' KeyError 'backend' kombu/utils/objects.py in get at line 42

    def __get__(self, obj, type=None):
    if obj is None:
    return self
    try:
    return obj.__dict__[self.__name__]
    except KeyError:
    value = obj.__dict__[self.__name__] = self.__get(obj)
    return value
    def __set__(self, obj, value):

ImportError No module named 'django-db'

celery/app/backends.py in byurl at line 65 scheme, , _ = url.partition('://') if '+' in scheme: backend, url = url.split('+', 1) else: backend = scheme return by_name(backend, loader), url

Values in context (From Sentry) backend: 'django-db' loader: <celery.loaders.app.AppLoader object at 0x7f13258a2f28> url: None

georgepsarakis commented 7 years ago

Have you installed the Python package django-celery-results, as stated in the documentation?

mounirmesselmeni commented 7 years ago

Yes it's installed and it's in the INSTALLED_APPS

pip freeze | grep celery

celery==4.0.2 django-celery-beat==1.0.1 django-celery-results==1.0.1

atkawa7 commented 7 years ago

@MounirMesselmeni @georgepsarakis I had the same problem. Here is what I ended up adding

app.loader.override_backends['django-db'] = 'django_celery_results.backends.database:DatabaseBackend'

normic commented 6 years ago

I stumbled about this issue today on our production system, but interestingly not on my dev machine. I was able to fix it by setting the option suggested by @atkawa7 .

To sum things up, I'm using: celery==4.0.0
django-celery-results==1.0.1

The only difference I could find within the virtualenvs is the used python version. Python 2.7.3 (prod - not working) Python 2.7.12 (dev - working)

And of course, I'm using Linux Mint on Dev and Debian Wheezy on the Production machin, but due to the use of virtualenvs this will problably not matter.

Maybe this helps to sort things out.

auvipy commented 6 years ago

@normic did you tried @atkawa7 solution?

normic commented 6 years ago

@auvipy Of course I did and it fixed my problem, I've written this in my comment (probably you overlooked the second sentence). I simply wrote it to help sort out things.

peterfarrell commented 5 years ago

Just wanted to confirm with celery 4.2.1 and django-celery-results (installed as master from github) we encountered this issue on one of boxes but not during local development.

Using the fix from @atkawa7 from above in our celery.py configuration file fixed the issue:

app.loader.override_backends['django-db'] = 'django_celery_results.backends.database:DatabaseBackend'

DZPM commented 5 years ago

Hi, just got bit by this bug:

happened in production, didn't notice in local
used the fix from @atkawa7, successfully (Thanks!)

If the bug is still happening, I guess it shouldn't be closed yet?

retosteffen commented 5 years ago

I had the same issue today. It worked on a local dev machine but not on a cloud test one. @atkawa7 fix solves it. Thanks!

washeck commented 5 years ago

@auvipy Why was this issue closed? I understand the issue can be solved by add the above live (override_backends) but the documentation does not mention it.

So is it required to always the override and the documentation should be fixed? Or is it a bug in celery and the override is just a workaround?

thedrow commented 5 years ago

I honestly don't know if this issue was resolved. I'm not all that familiar with how the Django integration works.

Kirade commented 4 years ago

Hi, This also happend in my project. I upgrade celery version from 4.2.1 to 4.4.2

In my case, This happens while running by_name function in celery/app/backends.py.

# celery/app/backends.py 

BACKEND_ALIASES = {
    'amqp': 'celery.backends.amqp:AMQPBackend',
    'rpc': 'celery.backends.rpc.RPCBackend',
    'cache': 'celery.backends.cache:CacheBackend',
    'redis': 'celery.backends.redis:RedisBackend',
    'rediss': 'celery.backends.redis:RedisBackend',
    'sentinel': 'celery.backends.redis:SentinelBackend',
    'mongodb': 'celery.backends.mongodb:MongoBackend',
    'db': 'celery.backends.database:DatabaseBackend',
    'database': 'celery.backends.database:DatabaseBackend',
    'elasticsearch': 'celery.backends.elasticsearch:ElasticsearchBackend',
    'cassandra': 'celery.backends.cassandra:CassandraBackend',
    'couchbase': 'celery.backends.couchbase:CouchbaseBackend',
    'couchdb': 'celery.backends.couchdb:CouchBackend',
    'cosmosdbsql': 'celery.backends.cosmosdbsql:CosmosDBSQLBackend',
    'riak': 'celery.backends.riak:RiakBackend',
    'file': 'celery.backends.filesystem:FilesystemBackend',
    'disabled': 'celery.backends.base:DisabledBackend',
    'consul': 'celery.backends.consul:ConsulBackend',
    'dynamodb': 'celery.backends.dynamodb:DynamoDBBackend',
    'azureblockblob': 'celery.backends.azureblockblob:AzureBlockBlobBackend',
    'arangodb': 'celery.backends.arangodb:ArangoDbBackend',
    's3': 'celery.backends.s3:S3Backend',
}

aliases = dict(BACKEND_ALIASES, **loader.override_backends)
aliases.update(
        load_extension_class_names(extension_namespace) or {})
try:
    cls = symbol_by_name(backend, aliases)
except ValueError as exc:
    ....

Inteneded value of aliases should be BACKEND_ALIASES. As BACKEND_ALIASES defined just above the aliases assignment

But when exception raised, value of aliases was

{
    'amqp': 'celery.backends.amqp:AMQPBackend', 
    'cassandra': 'celery.backends.cassandra:CassandraBackend', 
    'consul': 'celery.backends.consul:ConsulBackend', 
    'cosmosdbsql': 'celery.backends.cosmosdbsql:CosmosDBSQLBackend', 
    'couchbase': 'celery.backends.couchbase:CouchbaseBackend', 
    'couchdb': 'celery.backends.couchdb:CouchBackend', 
    'elasticsearch': 'celery.backends.elasticsearch:ElasticsearchBackend', 
    'file': 'celery.backends.filesystem:FilesystemBackend', 
    'mongodb': 'celery.backends.mongodb:MongoBackend', 
    'sentinel': 'celery.backends.redis:SentinelBackend'
}

which is outdated. This might be existed in version 3.1 ( My Project migration history: 3.1.x -> 4.2.1 -> 4.4.2 )

For some reason, constant BACKEND_ALIASES value pointing outdated value.

Though @atkawa7 's solution works well, I'm trying to find exact cause of this problem. (Maybe Some kind of cache or import stuffs in my project are problem)

Any idea on this issue?