Django Settings

[willbarton]

We have many Django projects, pretty much all of them handling Django settings file(s) very differently. We need to come up with some form of standardization that we can apply across existing projects and can create skeleton for in Django project/app templates.

The issues, as I see them:

• Handling environment-specific settings (local_settings.py files, with sample local_settings.py files for deployment and local development? Using 12 Factor-style environment variables for all local settings?)
• Handling the distinction between our apps running in their own standalone Django project or as one of more apps within a larger Django project. How do we handling settings that could work in both places?

What other settings-related issues have people encountered that we should consider?

[willbarton]

My own thoughts are that we should stick to Django convention as much as possible. This would mean, in my understanding:

• Having a settings.py file within the Django project that can run the app (or apps) standalone.

• Providing example_dev_local_settings.py or something to that effect for general categories of environments ("dev", "production", etc) that provide templates for settings specific to those environments.

  These could include "standalone" versions as well as ones that are intended to integrate with larger Django projects and sit beside other applications (in which case they'd provide the template for configuration specific to that app or apps).

This would mean common Django patterns, like customization to INSTALLED_APPS would handled in the local_settings.py files, something like this:

INSTALLED_APPS += ()
    'my',
    'apps',
    'here',
)

I know we have a few projects that take some form of this approach. What are the limitations of this approach? What are the conflicts that might occur between apps in production? Can we ennuerate the cases in which this absolutely won't work?

• I am forcefully against 12 Factor style environment variables for configuration.
  • It obfuscates settings:
  • By moving them out of the familiar location that, say, a Django developer will look for them
  • Debugging is more complex, and depends on more moving parts (now the shell as well as, say, virtualenv)
  • You can't simply HUP a process to reload configuration
  • It defers the problem of having the configuraiton stored somewhere — it still has to be stored somewhere, except now it's in a different sort of place. It's inevitably going to end up being a file somewhere.

[rosskarchner]

Are you forcefully against them, outside of development (ie, deployment)?

All but your last point can be dealt with by having settings default to using the environment variable, but fall back to something hardcoded, like:

IMPORTANT_SETTING = environ.get('IMPORTANT_SETTING', 'some_default')

(recognizing that it can get more complex than that, like for things that aren't natively strings, like INSTALLED_APPS)

Without presuming that Docker is the future, I think it highlights the strength of environment variables: it's how you can make your software configurable at launch time, in a standard way. Also If we ever get sophisticated enough with configuration management tools, app settings could be configured via Ansible or Puppet, and conveyed to the software via environment variables.

(attacking my own points: In Docker, you could have an entrypoint script that translates environment variables into a settings.py file, and with ansible and puppet configuration could also be conveyed via templates)

[willbarton]

I can see quite a bit of merit in the "standard way" argument; if all our applications used environment variables, no matter what they're written in, no matter what framework(s) they're using, it would be a single source of configuration. But just how divergent is our software that this is how we should standardize?

As far as settings that aren't just strings, do we express them as JSON, and then for those settings in Django load them that way? That just seems like it could get ugly quick.

IMPORTANT_SETTING = json.loads(environ.get('IMPORTANT_SETTING', "{'some': 'default, 'as':'a', 'json':'string'}"))

And that's just one problem. What about different Django apps that append to these sorts of settings?

The other question I would have, particularly in the Docker case, is what's the advantage of using individual environment variables for all settings over the DJANGO_SETTINGS_MODULE environment variable?

I think what I object to most about environment variables for configuration is their imposition of deployment concerns upon a particular app for everyone who might come to it and try to look at it, or work on it, who's not trying to deploy it to a heterogeneous environment where envvars are the standard.

If we could define app-specific settings like a normal, sane Django way, and then have our top-level project settings check them against the environment, that might be one avenue to explore. Something like the above shifts the magic to the deployment side, where, if it needs to exist, it should exist in isolation.

[rosskarchner]

I think you want to mix the two-- some things (most, even) can definitely be in DJANGO_SETTINGS_MODULE, but allow things like the database connection details, API keys, etc to be set via the environment.

I saw something recently that handled non-string settings better, I would never advocate JSON in environment variables ;)

[willbarton]

I saw something recently that handled non-string settings better, I would never advocate JSON in environment variables ;)

:smiling_imp:

[rosskarchner]

found it: https://github.com/joke2k/django-environ

and specifically: https://github.com/rconradharris/envparse

[willbarton]

Now that I think I've done my arguing forcefully and can stand down a little bit :grin:

I think we could separate out our concerns here. Anything that needs to environment-specific would be stored in separate settings files (local_settings.py, or dev_settings.py and prod_settings.py or something like that) anyway — I think that's a given. If we're using environment variables in production, that doesn't mean they have to be imposed on the dev side (or even necessarily in the public project space at all). And then prod_settings.py can use either of those env approaches (though I think I like the look of django-environ better, at a glance) or it can environ.get() as needed.

As long as that's the case, I think that allays my obfuscation concerns.

[rosskarchner]

One thing I never think about (when it's useful). Is that the module invoked via DJANGO_SETTINGS_MODULE doesn't need to live in the same codebase. It just needs to be a valid module, somewhere in your python path.

[willbarton]

True.

Thinking about it, we want our projects to be developed and usable independent of our global Django project, but we also want them to be able to live side-by side there. In other words, we want to develop independent Django apps that behave well with others, right? So, I think this leaves us with three basic concerns:

1. Django project settings

   Each of our projects will probably have its own skeleton Django "project" that needs some bare-bones settings. This should not include things like DEBUG, SECRET_KEY, etc. There really should be no customization done here except to remove things that will be set locally. This will be completely thrown away when an app is run in any kind of production project.

2. Local Django project settings

   What we have now in most projects is "local"/environmental settings that include things like DEBUG, SECRET_KEY, INSTALLED_APPS, CACHES, and anything else that is either specific to the app or apps being developed. In other words it's both app and environment-specific. Except that sometimes those app-specific settings are also environment-specific. This is a problem area for me; I think I'd like to see some logical separation here, but I don't know if that's realistic.

   However, generally local settings should provide a basic template for the settings a particular app should be deployed, and possible provide separate 'dev' and 'production' examples.

3. New configuration that a particular app introduces

   I think we have a few apps that introduce their own configuration variables, and expect them to exist in the settings files somewhere. I've seen, and written, apps that provide their own settings.py file (myapp.settings) that does a getattr on django.conf.settings while providing a default, where possible.

   from django.conf import settings
   
   MY_SETTING_DEFAULT = 'foo'
   MY_SETTING = getattr(settings, 
      'MY_SETTING',
      MY_SETTING_DEFAULT)

   Then all settings are accessed in the app via myapp.settings rather than django.conf.settings. I like this model because it doesn't presume that settings exist if someone is trying to use an app for the first time, and provides the opportunity to error gracefully if no sane default value exists (rather than just spewing a traceback of a KeyError.

So, that's the problem space as I see it. I think we can incorporate environment variables into the local, environment-specific configuration as needed.

[rosskarchner]

Another interesting angle is templates and static assets: should apps include enough templates and static assets to be usable and attractive on their own? Must/should they be unbranded? Or have any relationship at all to our visual style, color palette, etc.

[willbarton]

I think that's also an interesting question.... although I think that veers more into

I know that eRegs is a bad citizen of cf.gov in this area, but it is completely usable independent of anything else, and all the branding is separate from the application. I think there's probably a happy medium — wherein an app provides enough that it could be integrated. But that depends on how templates and assets are structured, I think, which is up to the the front-ends.

[Ooblioob]

This is a great discussion, I'm actually trying to solve this exact problem with the Intranet right now!

I agree there are 3 general types of settings files:

• Settings that are applicable to the django project as a whole, regardless of environment
• Settings that are environment specific
• Settings that are app specific

In my mind we solve this by letting each app contain their own settings.py file. From an automation perspective, I'd like to be able to generate each "app settings" file if it doesn't exist with something simple, like

INSTALLED_APPS += ('some_app',)

This would let teams choose to overwrite the default, but give backwards compatibility while we work on adopting the new model.

Regarding environment specific, I feel like this could be handled through templating in a deployment script (i.e. Ansible) rather than storing settings_dev.py and such. That way we keep environment specific variables set in our deployment code rather than through magical environment variables. This might offer the nice middleground that you want. If you want to deploy the "production" version on your local machine, simply change the vagrant box to be in the "production" group and it applies the same environment variables.

It also simplifies our main settings file because we'd only have one target. So a settings file would look like this:

DATABASES = {
        'default': {
           ...
}

from project.local_settings import *
from project.app_settings import *

app_settings would be a folder with a file for each application containing their app-specific settings.

[willbarton]

@Ooblioob what does an Ansible template look like from a user perspective?

My concern is that I'd like to ensure that our apps are also usable in isolation for someone coming to them from anywhere. That might be a bit idealistic, but adding Ansible and Vagrant as requirements for taking a look at a quick checkout don't really work for me unless we're talking about using that Ansible template for a production deployment only... such that it replaces prod_settings.py in a hypothetical scenario.

But if an Ansible template would look more-or-less like an example settings file that someone could copy and edit manually, I think that would probably be the best of both worlds — I just don't want to impose any more requirements than we have to. Options? Yes. Let's just ensure users have optionality.

[Ooblioob]

Templates in Ansible should feel right at home for Django developers, it's all Jinja2 templates. An example:

DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.mysql',
            'NAME': {{ db_name }},
            'USER': {{ db_user }},
            'PASSWORD': {{ db_password }},
            'HOST': {{ db_host }},  
            'PORT': {{ db_port }},  
        },
}

Long term our goal is to have Ansible be our main deployment mechanism, and these it seems well suited to handle these types of scenarios. It enables us to provide a single mechanism for setting up a local dev environment as well as the production environment, with the differences being transparent. If we have different ways of setting up our dev environment than our production environment, we run the risk of configuration bugs when the environments don't match up.

[willbarton]

:+1: That looks great then. And that shifts the environment variable decision-making off further away from a particular project — it's the a matter for Ansible.

I like that, overall.

[Ooblioob]

One detail to this plan now that I've tried to implement it. It's not quite as easy to import a folder called app_settings as I'd thought. I had to go with this solution instead (as suggested by @rosskarchner):

# Include any settings files from our apps
from glob import glob
for app_settings_path in glob(os.path.join(PROJECT_PATH, 'collab/app_settings/*.py')):
    execfile(app_settings_path)

This lets us keep everything in a folder and we can still write simple files like this:

INSTALLED_APPS += ('some_app',)

The only disadvantage is that it's possible to overwrite other apps' settings if we add things carelessly (assign instead of append), so we just need to give guidance to teams and set good examples.

[willbarton]

Would it be helpful to see how we've handled per-regulation configs for the eRegs parser? I don't know that it's necessarily portable to Django (the parser is NOT Django, but uses Python settings files like Django), but this is what we have:

When you run the parser it looks for a settings.py in the same folder as the build_from.py script. Our standard settings.py looks like this:

try:
    from regconfig import *
except ImportError:
    from regparser.default_settings import *

# Try to import a local_settings module to override parser settings. If
# it doesn't exist, we must not have needed it anyway.
try:
    from local_settings import *
except ImportError:
    pass

The parser defines some defaults in regparser.default_settings, BUT if regconfig exists, we import that instead.

So, that separate package called regconfig is part of a separate repo, since it's CFPB-specific). Within that package we have individual python files for each CFPB regulation, then import them in the package's __init__.py:

It imports the default settings first, then each reg file:

from regparser.default_settings import *

## All Regulations
from .all import *
IGNORE_DEFINITIONS_IN['ALL'] = IGNORE_DEFINITIONS_IN_ALL

## Reg B
from .reg_b import *
IGNORE_DEFINITIONS_IN['1002'] = IGNORE_DEFINITIONS_IN_PART_1002
INCLUDE_DEFINITIONS_IN['1002'] = INCLUDE_DEFINITIONS_IN_PART_1002
APPENDIX_IGNORE_SUBHEADER_LABEL['1002'] = APPENDIX_IGNORE_SUBHEADER_LABEL_1002

Then after all the individual reg configs are in, the settings.py loads a local_settings.py file, which is presumed to contain local paths, API endpoints, etc, that are environment-specific.

So... I don't know if that's a reusable pattern here, but it seems like we could have a Python module (like regconfig/__ini__.py) that could load individual app configs, like regulations.app_settings (presuming those settings don't override INSTALLED_APPS, etc, and merely append to them).

What didn't work about that? I'm curious.

The only disadvantage is that it's possible to overwrite other apps' settings if we add things carelessly (assign instead of append), so we just need to give guidance to teams and set good examples.

Well, we can't do everything for teams. They still have to have some responsibility for their settings :stuck_out_tongue_winking_eye:

[willbarton]

I've tried to distill this discussion in PR #7, a guide for Django settings.

@cfpb/back-end-team-admin