Celery fails with "cannot find module 'django-db'"

[chlarsen]

Environmental Information

• Onadata version: v2.4.5(2020-10-23), FreeBSD jail with Python 3.7.9

Problem description

Onadata starts fine using the prescribed uWSGI method, but bother celeryd as well as celerybeat fail with:

  File "/usr/local/onadata/lib/python3.7/site-packages/celery/bootsteps.py", line 112, in start
    self.on_start()
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/apps/worker.py", line 136, in on_start
    self.emit_banner()
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/apps/worker.py", line 170, in emit_banner
    ' \n', self.startup_info(artlines=not use_image))),
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/apps/worker.py", line 232, in startup_info
    results=self.app.backend.as_uri(),
  File "/usr/local/onadata/lib/python3.7/site-packages/kombu/utils/objects.py", line 43, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/app/base.py", line 1170, in backend
    return self._get_backend()
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/app/base.py", line 887, in _get_backend
    self.loader)
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/app/backends.py", line 70, in by_url
    return by_name(backend, loader), url
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/app/backends.py", line 50, in by_name
    cls = symbol_by_name(backend, aliases)
  File "/usr/local/onadata/lib/python3.7/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/usr/local/onadata/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'django-db'
[2020-10-31 00:35:31,723: CRITICAL/MainProcess] Unrecoverable error: ModuleNotFoundError("No module named 'django-db'")
Traceback (most recent call last):
  File "/usr/local/onadata/lib/python3.7/site-packages/kombu/utils/objects.py", line 41, in __get__
    return obj.__dict__[self.__name__]
KeyError: 'backend'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/usr/local/onadata/lib/python3.7/site-packages/celery/worker/worker.py", line 203, in start
    self.blueprint.start(self)
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/bootsteps.py", line 112, in start
    self.on_start()
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/apps/worker.py", line 136, in on_start
    self.emit_banner()
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/apps/worker.py", line 170, in emit_banner
    ' \n', self.startup_info(artlines=not use_image))),
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/apps/worker.py", line 232, in startup_info
    results=self.app.backend.as_uri(),
  File "/usr/local/onadata/lib/python3.7/site-packages/kombu/utils/objects.py", line 43, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/app/base.py", line 1170, in backend
    return self._get_backend()
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/app/base.py", line 887, in _get_backend
    self.loader)
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/app/backends.py", line 70, in by_url
    return by_name(backend, loader), url
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/app/backends.py", line 50, in by_name
    cls = symbol_by_name(backend, aliases)
  File "/usr/local/onadata/lib/python3.7/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/usr/local/onadata/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'django-db'
[2020-10-31 00:42:18,803: CRITICAL/MainProcess] Unrecoverable error: ModuleNotFoundError("No module named 'django-db'")
Traceback (most recent call last):
  File "/usr/local/onadata/lib/python3.7/site-packages/kombu/utils/objects.py", line 41, in __get__
    return obj.__dict__[self.__name__]
KeyError: 'backend'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/worker/worker.py", line 203, in start
    self.blueprint.start(self)
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/bootsteps.py", line 112, in start
    self.on_start()
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/apps/worker.py", line 136, in on_start
    self.emit_banner()
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/apps/worker.py", line 170, in emit_banner
    ' \n', self.startup_info(artlines=not use_image))),
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/apps/worker.py", line 232, in startup_info
    results=self.app.backend.as_uri(),
  File "/usr/local/onadata/lib/python3.7/site-packages/kombu/utils/objects.py", line 43, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/app/base.py", line 1170, in backend
    return self._get_backend()
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/app/base.py", line 887, in _get_backend
    self.loader)
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/app/backends.py", line 70, in by_url
    return by_name(backend, loader), url
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/app/backends.py", line 50, in by_name
    cls = symbol_by_name(backend, aliases)
  File "/usr/local/onadata/lib/python3.7/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/usr/local/onadata/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'django-db'
[2020-10-31 00:42:27,250: CRITICAL/MainProcess] Unrecoverable error: ModuleNotFoundError("No module named 'django-db'")
Traceback (most recent call last):
  File "/usr/local/onadata/lib/python3.7/site-packages/kombu/utils/objects.py", line 41, in __get__
    return obj.__dict__[self.__name__]
KeyError: 'backend'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/worker/worker.py", line 203, in start
    self.blueprint.start(self)
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/bootsteps.py", line 112, in start
    self.on_start()
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/apps/worker.py", line 136, in on_start
    self.emit_banner()
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/apps/worker.py", line 170, in emit_banner
    ' \n', self.startup_info(artlines=not use_image))),
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/apps/worker.py", line 232, in startup_info
    results=self.app.backend.as_uri(),
  File "/usr/local/onadata/lib/python3.7/site-packages/kombu/utils/objects.py", line 43, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/app/base.py", line 1170, in backend
    return self._get_backend()
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/app/base.py", line 887, in _get_backend
    self.loader)
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/app/backends.py", line 70, in by_url
    return by_name(backend, loader), url
  File "/usr/local/onadata/lib/python3.7/site-packages/celery/app/backends.py", line 50, in by_name
    cls = symbol_by_name(backend, aliases)
  File "/usr/local/onadata/lib/python3.7/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/usr/local/onadata/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'django-db'

Expected behavior

Both celery instances should start up smoothly when using the scripts provided.

Steps to reproduce the behavior

Also, adjust the onadata/script/init.d/celerybeat-generic script (tailored for use with celery < 5) as follows:

#DEFAULT_CELERYBEAT="$CELERY_BIN beat"
DEFAULT_CELERYBEAT="$CELERY_BIN"

#CELERYBEAT_OPTS="$CELERYBEAT_OPTS -f $CELERYBEAT_LOG_FILE -l $CELERYBEAT_LOG_LEVEL"
CELERYBEAT_OPTS="$CELERYBEAT_OPTS beat -f $CELERYBEAT_LOG_FILE -l $CELERYBEAT_LOG_LEVEL"

Additional Information

First, new scripts are required in onadata/script/ to accommodate the new celery 5 syntax.

Logs, related issues, weird / out of place occurrences, local settings, possible approach to solving this... In older celery versions, this had to be added to local_settings.py:

celery_app.loader.override_backends['django-db'] = 'django_celery_results.backends.database:DatabaseBackend'

However, this won't work any more and creates server crashes. What is the updated fix?

Thank you so much! Chris

[chlarsen]

The only workaround is to use OnaData version v2.4.4, which uses celery version <5. There are other issues, however.

[DavisRayM]

Hey @chlarsen, we've since downgraded back to celery <5 due to the instability in the latest versions of celery. This issue should have been solved by #1942... Kindly try testing out whether this is still an issue with the latest changes in the master branch