search

Koed00 / django-q

A multiprocessing distributed task queue for Django
https://django-q.readthedocs.org
MIT License
1.83k stars 289 forks source link

django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet. #313

Open razens opened 6 years ago

razens commented 6 years ago

Traceback (most recent call last): File "<string>", line 1, in <module> File "C:\Python27\Lib\multiprocessing\forking.py", line 381, in main self = load(from_parent) File "C:\Python27\Lib\pickle.py", line 1384, in load return Unpickler(file).load() File "C:\Python27\Lib\pickle.py", line 864, in load dispatch[key](self) File "C:\Python27\Lib\pickle.py", line 1096, in load_global klass = self.find_class(module, name) File "C:\Python27\Lib\pickle.py", line 1130, in find_class __import__(module) File "C:\Source\GitHub\octopus\server\venv\lib\site-packages\django_q\cluster.py", line 24, in <module> from django_q import tasks File "C:\Source\GitHub\octopus\server\venv\lib\site-packages\django_q\tasks.py", line 13, in <module> from django_q.models import Schedule, Task File "C:\Source\GitHub\octopus\server\venv\lib\site-packages\django_q\models.py", line 16, in <module> class Task(models.Model): File "C:\Source\GitHub\octopus\server\venv\lib\site-packages\django\db\models\base.py", line 110, in __new__ app_config = apps.get_containing_app_config(module) File "C:\Source\GitHub\octopus\server\venv\lib\site-packages\django\apps\registry.py", line 247, in get_containing_app_config self.check_apps_ready() File "C:\Source\GitHub\octopus\server\venv\lib\site-packages\django\apps\registry.py", line 125, in check_apps_ready raise AppRegistryNotReady("Apps aren't loaded yet.") django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.

pip list

Package Version

amqp 2.2.2 arrow 0.12.1 asgiref 1.1.2 attrs 17.2.0 autobahn 17.9.3 Automat 0.6.0 backports.functools-lru-cache 1.5 BeautifulSoup 3.2.1 beautifulsoup4 4.6.0 billiard 3.5.0.3 blessed 1.15.0 celery 4.1.0 Cerberus 1.2 certifi 2017.7.27.1 channels 1.1.6 chardet 3.0.4 constantly 15.1.0 coreapi 2.3.3 coreschema 0.0.4 daphne 1.3.0 Django 1.11.13 django-celery-results 1.0.1 django-crispy-forms 1.6.1 django-debug-toolbar 1.9.1 django-enumchoicefield 0.9.0 django-extensions 1.9.8 django-filter 1.0.4 django-guardian 1.4.9 django-picklefield 1.0.0 django-q 0.9.4 django-reset-migrations 0.3.1 django-rest-swagger 2.1.2 django-split-settings 0.3.0 djangorestframework 3.8.2 djangorestframework-jwt 1.11.0 drf-flex-fields 0.3.4 encryptor64 1.2.0 enum34 1.1.6 future 0.16.0 gunicorn 19.7.1 hyperlink 17.3.1 idna 2.6 incremental 17.5.0 itypes 1.1.0 Jinja2 2.9.6 kombu 4.1.0 Markdown 2.6.8 MarkupSafe 1.0 mongo-py-logger 1.8.0 mysqlclient 1.3.9 newrelic 2.96.0.80 ntlm-auth 1.2.0 openapi-codec 1.3.2 ordereddict 1.1 packaging 16.8 pip 18.0 psycopg2-binary 2.7.4 pyCLI 2.0.3 Pygments 2.2.0 PyJWT 1.5.3 pymongo 3.7.1 pyparsing 2.2.0 Pyro4 4.63 pyteamcity 0.1.1 python-dateutil 2.6.1 python-ldap 2.4.41 python-ntlm 1.0 pytz 2017.2 requests 2.18.4 requests-ntlm 1.0.0 selectors34 1.2 serpent 1.27 setuptools 40.0.0 simplejson 3.11.1 six 1.11.0 sqlparse 0.2.4 Twisted 17.9.0 txaio 2.8.2 typing 3.6.4 uritemplate 3.0.0 urllib3 1.22 vine 1.1.4 wcwidth 0.1.7 wheel 0.31.1 xmltodict 0.11.0 zope.interface 4.4.3

It works for me just if I add import django django.setup() to django_q/cluster.py

Can you help me with this issue.

am2222 commented 6 years ago

I had the same problem and it is fixed now by adding # Django import django django.setup()

into django_q/cluster.py my django-q version is 0.9.6 If you upgrade it this issue will be fixed

shacker commented 5 years ago

I have just upgraded python from 3.6 to 3.8 and am suddenly getting this same error (it ran perfectly under python 3.6 for the past couple of years). It stops at that error, and Ctrl-C will not work - I have to completely quit the terminal app.

Running django-q version 1.0.2 on Mac OS 10.14.6. I've been experimenting with cpu_affinity and other options but am getting nowhere fast. Anyone else?

shacker commented 5 years ago

FWIW I tried the suggestion from @am2222 above, and added:

import django
django.setup()

before the #Local section in cluster.py. That allowed it to get further, but now it crashes on startup with:

Process Process-1:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/path/to/venv/lib/python3.8/site-packages/django_q/cluster.py", line 124, in __init__
    self.start()
  File "/path/to/venv/lib/python3.8/site-packages/django_q/cluster.py", line 128, in start
    self.spawn_cluster()
  File "/path/to/venv/lib/python3.8/site-packages/django_q/cluster.py", line 199, in spawn_cluster
    self.monitor = self.spawn_monitor()
  File "/path/to/venv/lib/python3.8/site-packages/django_q/cluster.py", line 163, in spawn_monitor
    return self.spawn_process(monitor, self.result_queue, self.broker)
  File "/path/to/venv/lib/python3.8/site-packages/django_q/cluster.py", line 153, in spawn_process
    p.start()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_thread.lock' object
shacker commented 5 years ago

Interesting - It seems to be working fine on Linux instances with the same version of python and django, but not on MacOS.

maerteijn commented 5 years ago

See https://github.com/raonyguimaraes/django-q/commit/668cf95fd6540de66a0a55c863f5bacef99058d0

Where is django_q in your INSTALLED_APPS setting?

shacker commented 5 years ago

I just tried making it last:

INSTALLED_APPS = DJANGO_APPS + THIRD_PARTY_APPS + LOCAL_APPS
INSTALLED_APPS.append("django_q")

but it crashes on the same error:

TypeError: cannot pickle '_thread.lock' object

shacker commented 5 years ago

At this point I'm pretty certain this is a new Python 3.8 issue, unrelated to order of loading. OK if I open a new ticket for this?

maerteijn commented 5 years ago

The Readme states Tested with: Python 3.6. 3.7 Django 1.11.x and 2.2.x

So 3.8 is not officially supported (yet).

Opening a new ticket requesting python 3.8 support would be fine I guess, but note that Django itself has no released version yet which is 3.8 compatible.

shacker commented 5 years ago

Got it thanks. For now, we've decided to stick with py3.7 until Django support is official, but I'll open a ticket here for future reference.