Unable to run project under Gunicorn

[jacobh]

Hi,

I've been racking my brain all day with this issue and I've been unable to get it working.

We're in the process of switching over to using django-configurations for our project and for some reason it appears gunicorn isn't able to pick up the settings. Running the project with python manage.py runserver works perfectly.

My code is as follows:

gizmag/settings.py

# I can provide my entire settings file if required
from configurations import Configuration

class Base(Configuration):
    ...

class Dev(Base):
    DEBUG = True
    ...

class Test(Dev):
    ...

class Prod(Base):
    pass

gizmag/wsgi.py

import os

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'gizmag.settings')
os.environ.setdefault('DJANGO_CONFIGURATION', 'Prod')

from configurations.wsgi import get_wsgi_application

application = get_wsgi_application()

manage.py

#!/usr/bin/env python

import os
import sys

if 'test' in sys.argv:
    default_config = 'Test'
else:
    default_config = 'Dev'

if __name__ == "__main__":
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'gizmag.settings')
    os.environ.setdefault('DJANGO_CONFIGURATION', default_config)

    from configurations.management import execute_from_command_line

    execute_from_command_line(sys.argv)

And the error I receive if I don't explicitly export my settings module / configuration

(gizmag)jacob  [master] ~/p/gizmag $ gunicorn gizmag.wsgi:application
2013-09-24 16:38:23 [73705] [INFO] Starting gunicorn 18.0
2013-09-24 16:38:23 [73705] [INFO] Listening at: http://127.0.0.1:8000 (73705)
2013-09-24 16:38:23 [73705] [INFO] Using worker: sync
2013-09-24 16:38:23 [73708] [INFO] Booting worker with pid: 73708
2013-09-24 16:38:23 [73708] [ERROR] Exception in worker process:
Traceback (most recent call last):
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/gunicorn/arbiter.py", line 495, in spawn_worker
    worker.init_process()
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/gunicorn/workers/base.py", line 106, in init_process
    self.wsgi = self.app.wsgi()
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/gunicorn/app/base.py", line 114, in wsgi
    self.callable = self.load()
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 62, in load
    return self.load_wsgiapp()
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 49, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/gunicorn/util.py", line 354, in import_app
    __import__(module)
  File "/Users/jacob/p/gizmag/gizmag/__init__.py", line 16, in <module>
    post_migrate.connect(update_permissions_after_migration)
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 75, in connect
    if settings.DEBUG:
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/django/conf/__init__.py", line 53, in __getattr__
    self._setup(name)
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/django/conf/__init__.py", line 46, in _setup
    % (desc, ENVIRONMENT_VARIABLE))
ImproperlyConfigured: Requested setting DEBUG, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.
Traceback (most recent call last):
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/gunicorn/arbiter.py", line 495, in spawn_worker
    worker.init_process()
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/gunicorn/workers/base.py", line 106, in init_process
    self.wsgi = self.app.wsgi()
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/gunicorn/app/base.py", line 114, in wsgi
    self.callable = self.load()
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 62, in load
    return self.load_wsgiapp()
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 49, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/gunicorn/util.py", line 354, in import_app
    __import__(module)
  File "/Users/jacob/p/gizmag/gizmag/__init__.py", line 16, in <module>
    post_migrate.connect(update_permissions_after_migration)
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 75, in connect
    if settings.DEBUG:
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/django/conf/__init__.py", line 53, in __getattr__
    self._setup(name)
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/django/conf/__init__.py", line 46, in _setup
    % (desc, ENVIRONMENT_VARIABLE))
ImproperlyConfigured: Requested setting DEBUG, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.
2013-09-24 16:38:23 [73708] [INFO] Worker exiting (pid: 73708)
2013-09-24 16:38:23 [73705] [INFO] Shutting down: Master
2013-09-24 16:38:23 [73705] [INFO] Reason: Worker failed to boot.

And when I do explicitly define them (I also don't know why my default environment variables aren't working)

(gizmag)jacob  [master] ~/p/gizmag $ export DJANGO_CONFIGURATION=Prod
(gizmag)jacob  [master] ~/p/gizmag $ export DJANGO_SETTINGS_MODULE=gizmag.settings
(gizmag)jacob  [master] ~/p/gizmag $ gunicorn gizmag.wsgi:application
2013-09-24 16:39:09 [73732] [INFO] Starting gunicorn 18.0
2013-09-24 16:39:09 [73732] [INFO] Listening at: http://127.0.0.1:8000 (73732)
2013-09-24 16:39:09 [73732] [INFO] Using worker: sync
2013-09-24 16:39:09 [73735] [INFO] Booting worker with pid: 73735
2013-09-24 16:39:09 [73735] [ERROR] Exception in worker process:
Traceback (most recent call last):
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/gunicorn/arbiter.py", line 495, in spawn_worker
    worker.init_process()
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/gunicorn/workers/base.py", line 106, in init_process
    self.wsgi = self.app.wsgi()
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/gunicorn/app/base.py", line 114, in wsgi
    self.callable = self.load()
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 62, in load
    return self.load_wsgiapp()
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 49, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/gunicorn/util.py", line 354, in import_app
    __import__(module)
  File "/Users/jacob/p/gizmag/gizmag/__init__.py", line 16, in <module>
    post_migrate.connect(update_permissions_after_migration)
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 75, in connect
    if settings.DEBUG:
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/django/conf/__init__.py", line 53, in __getattr__
    self._setup(name)
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/django/conf/__init__.py", line 48, in _setup
    self._wrapped = Settings(settings_module)
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/django/conf/__init__.py", line 132, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/Users/jacob/p/gizmag/gizmag/settings.py", line 425, in <module>
    class Dev(Base):
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/configurations/base.py", line 30, in __new__
    raise ImproperlyConfigured(install_failure)
ImproperlyConfigured: django-configurations settings importer wasn't correctly installed. Please use one of the starter functions to install it as mentioned in the docs: http://django-configurations.readthedocs.org/
Traceback (most recent call last):
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/gunicorn/arbiter.py", line 495, in spawn_worker
    worker.init_process()
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/gunicorn/workers/base.py", line 106, in init_process
    self.wsgi = self.app.wsgi()
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/gunicorn/app/base.py", line 114, in wsgi
    self.callable = self.load()
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 62, in load
    return self.load_wsgiapp()
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 49, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/gunicorn/util.py", line 354, in import_app
    __import__(module)
  File "/Users/jacob/p/gizmag/gizmag/__init__.py", line 16, in <module>
    post_migrate.connect(update_permissions_after_migration)
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 75, in connect
    if settings.DEBUG:
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/django/conf/__init__.py", line 53, in __getattr__
    self._setup(name)
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/django/conf/__init__.py", line 48, in _setup
    self._wrapped = Settings(settings_module)
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/django/conf/__init__.py", line 132, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/Users/jacob/p/gizmag/gizmag/settings.py", line 425, in <module>
    class Dev(Base):
  File "/Users/jacob/.virtualenvs/gizmag/lib/python2.7/site-packages/configurations/base.py", line 30, in __new__
    raise ImproperlyConfigured(install_failure)
ImproperlyConfigured: django-configurations settings importer wasn't correctly installed. Please use one of the starter functions to install it as mentioned in the docs: http://django-configurations.readthedocs.org/
2013-09-24 16:39:09 [73735] [INFO] Worker exiting (pid: 73735)
2013-09-24 16:39:09 [73732] [INFO] Shutting down: Master
2013-09-24 16:39:09 [73732] [INFO] Reason: Worker failed to boot.

Thanks in advance for any help! and I'm so sorry if it turns out I've skipped over part of the documentation.

[jacobh]

I added

from configurations import importer
importer.install()

to the top of my settings file and I now get the error

ImproperlyConfigured: The SECRET_KEY setting must not be empty.

I'm not sure if my issues are related to https://github.com/jezdez/django-configurations/issues/27 . I'm inclined to think not as my settings file doesn't really do anything funky (as far as I know) https://gist.github.com/hzy/4c870d217d80bb398007

[jezdez]

@hzy I wasn't able to reproduce the problem you describe. What version of django-configurations do you run?

You definitely don't need the installer to be imported in your settings and called.

What happens if you comment out the import and call of djcelery.setup_loader()?

[jezdez]

regarding gizmag_secrets.settings.Secrets, does its module have an import for django.conf.settings?

[jacobh]

django-configurations==0.6 and Django==1.5.4.

I tried the different combinations of having djcelery.setup_loader() commented out and it made no difference.

gizmag_secrets.settings entire contents is

class Secrets(object):
    EMAIL_HOST_USER = ...
    EMAIL_HOST_PASSWORD = ...
    ...

[jacobh]

I think I found the issue. Just reading the traceback I noticed mention of gizmag/init.py. that file contains this little permission trick. When I commented it out gunicorn fired up instantly.

Maybe this is being called by gunicorn before configurations has a chance to kick in?

# register a signal do update permissions every migration.
# This is based on app django_extensions update_permissions command
from south.signals import post_migrate

def update_permissions_after_migration(app, **kwargs):
    """
    Update app permission just after every migration.
    This is based on app django_extensions update_permissions management command.
    """
    from django.conf import settings
    from django.db.models import get_app, get_models
    from django.contrib.auth.management import create_permissions

    create_permissions(get_app(app), get_models(), 2 if settings.DEBUG else 0)

post_migrate.connect(update_permissions_after_migration)

[jezdez]

Aha! That must be it. When you try to import a module in a package like gizmag.wsgi (as does gunicorn during startup) it'll also import the gismag/__init__.py first, triggering the settings loading cycle. It's generally a good idea to not do that and instead put signal connection inside of models.py modules, which are guarenteed to be loaded by Django after the settings system has initialized.

I'm going to close this as a WONTFIX, but I'll keep this in mind in case further reports are coming in with similar situations.

[jezdez]

BTW, this is the culprit: https://bitbucket.org/andrewgodwin/south/src/6612130a3287a9e74bfa157d0a03af0a5df467d3/south/signals.py?at=default#cl-6

[jacobh]

Thanks for the help :)

[jezdez]

Gladly, @hzy!