How to handle different environments

[pySilver]

Hi!

I like this project! However I'm curious what's your approach on environments differences. For instance we might want to add some app or middleware only to development env or specifically define some settings in production while keeping django defaults for test.

[pySilver]

Spamer - go away. Potentially danger contents. (reported and banned by GH)

[erhosen]

Hi @pySilver,

Thanks for your interest in the project!

You can pass relevant env vars, for example for dev:

DEBUG=True
...
INSTALLED_APPS=["django.contrib.admin", "django.contrib.auth", "my.app"]
...

And for prod:

DEBUG=False
...
INSTALLED_APPS=["django.contrib.admin", "django.contrib.auth"]
...

Pydantic should be able to parse these configurations correctly.

I understand that handling lists like this might seem a bit overhead, but on the other hand it’s explicit :)

[pySilver]

@erhosen thats was my initial thought. However I'm in the middle of experimenting with a little bit different approach:

• Using yaml provider as it is much easier to describe complex structures than env (which is btw has no standard anyway)
• Using os.environment['PROJNAME_SETTINGS'] that points to correct yml file with .settings.yml by default
• Creating sample .settings.dev.example.yml, settings.production.example.yml etc so newcomers will see what we use in production typically, without secrets.

It is not ideal (well we moved environment settings from python to yml) and I'm not sure if PyCharm will not complain too much about installed middlewares / apps. BUT as you stated in project description - it has lots of positive outcomes.

[erhosen]

@pySilver cool stuff - it indeed might work!

Anyway be careful with taking pydjantic to some serious projects (it was mostly a fun concept for my own pet-project), - I see how It can be broken with a newer releases of Django, and I can't guarantee supporting it further 🤷🏻‍♂️

[pySilver]

@erhosen Would you please share how it can be broken?

[erhosen]

Here I'm patching locals of the settings module: https://github.com/erhosen-libs/pydjantic/blob/0a981396572909b40b1e278b4020a7560c741c93/pydjantic/pydjantic.py#L60-L61

It's still funny to me that Python allows to do so

I'm mostly worrying about scenario, when Django decide to rethink their settings-managment approach

[pySilver]

Oh django is so-backward-compatible that I bet 20$ it will never happen :D Sometimes I hate Django to be so slow with adoption modern web but there are positive aspects of it too.

I haven't tested the following myself, but apparently one can use something like this:

# settings.py
from typing import Any, List
from pydantic import BaseSettings

class DjangoSettings(BaseSettings):
    """Manage all the project settings"""
    DEBUG: bool = True
    TIME_ZONE: str = "Europe/Paris"
    ALLOWED_HOSTS: List[str] = []

_settings = DjangoSettings() # settings 
_settings_dict = _settings.dict() # dict representation

def __dir__() -> List[str]:
    """The list of available options are retrieved from 
    the dict view of our DjangoSettings object.
    """
    return list(_settings_dict.keys())

def __getattr__(name: str) -> Any:
    """Turn the module access into a DjangoSettings access"""
    return _settings_dict[name]

[pySilver]

After reviewing complex configurations in a few projects, I came to the conclusion that Pydantic alone isn’t a silver bullet here. Django settings include several complex structures like TEMPLATES, CACHES, and STATICFILES_FINDERS, along with third-party settings. While these can certainly be translated into environment variables or YAML files combined with a settings class, is it really worth it? Most of the settings aren't supposed to change (template loaders, auth backends) but may differ from environment to environment (test, production, development) and Pydantic itself does not solve it.

[pySilver]

@erhosen I slept over it and actually found a good way to manage different environments that may need a little bit different setups.

Basically, there is a settings class that defines different order of sources, so one can force initialization variables to be the most important:

class CustomisedSourceSettings(BaseSettings):
    @classmethod
    def settings_customise_sources(
        cls,
        settings_cls: type[BaseSettings],
        init_settings: PydanticBaseSettingsSource,
        env_settings: PydanticBaseSettingsSource,
        dotenv_settings: PydanticBaseSettingsSource,
        file_secret_settings: PydanticBaseSettingsSource,
    ) -> tuple[PydanticBaseSettingsSource, ...]:
        return (
            init_settings,
            YamlConfigSettingsSource(settings_cls),
        )

We are not using env variables for a reason: Yaml is way more easier to manage and modern build systems usually allows runtime creation. However nothing stops one to add env source back with a proper priority.

also, there are 3 envs tracked in git:

.envs/.local.yaml
.envs/.test.yaml
.envs/.production.example.yaml 

Code also expects .envs/.override.yaml may exist which will be handled by pydantic to override defaults

Next part is 3 different modules:

settings/local.py
settings/test.py
settings/production.py

where they are almost identical, but may add an app or middleware that is required in current environment. For instance there is local environment that adds debug and utils apps and middlewares to default list.

While it can be done via yaml file it feels important to enforce some of the settings and don't rely on yaml config which should not carry to much logic, imho.

from pydantic_settings import SettingsConfigDict

from config.settings.base import (
    _DEFAULT_APPS,
    _DEFAULT_MIDDLEWARE,
    BASE_DIR,
    AdminSettings,
    AuthSettings,
    CachesSettings,
    CelerySettings,
    CustomisedSourceSettings,
    DatabasesSettings,
    DjangoAllAuthSettings,
    DjangoDebugToolbarSettings,
    EmailSettings,
    FixturesSettings,
    GeneralSettings,
    InstalledAppsSettings,
    LoggingSettings,
    MediaSettings,
    MiddlewareSettings,
    MigrationSettings,
    PasswordSettings,
    SecuritySettings,
    StaticFilesSettings,
    StoragesSettings,
    TemplatesSettings,
    URLSettings,
    to_django,
)

class DjangoSettings(
    GeneralSettings,
    DatabasesSettings,
    CachesSettings,
    URLSettings,
    InstalledAppsSettings,
    MigrationSettings,
    AuthSettings,
    PasswordSettings,
    MiddlewareSettings,
    StoragesSettings,
    StaticFilesSettings,
    MediaSettings,
    TemplatesSettings,
    FixturesSettings,
    SecuritySettings,
    EmailSettings,
    AdminSettings,
    LoggingSettings,
    CelerySettings,
    DjangoAllAuthSettings,
    DjangoDebugToolbarSettings,
    CustomisedSourceSettings,
):
    model_config = SettingsConfigDict(
        yaml_file=[
            BASE_DIR / ".envs/.local.yaml",
            BASE_DIR / ".envs/.override.yaml",
        ],
        extra="ignore",
    )

# Setup django-extensions
# ------------------------------------------------------------------------------
# https://django-extensions.readthedocs.io/en/latest/installation_instructions.html#configuration
_DEFAULT_APPS = ["django_extensions", *_DEFAULT_APPS]

# Setup django-debug-toolbar
# ------------------------------------------------------------------------------
# https://django-debug-toolbar.readthedocs.io/en/latest/installation.html#installation
_DEFAULT_APPS = ["debug_toolbar", *_DEFAULT_APPS]
_DEFAULT_MIDDLEWARE = [
    "debug_toolbar.middleware.DebugToolbarMiddleware",
    *_DEFAULT_MIDDLEWARE,
]

django_settings = DjangoSettings(
    INSTALLED_APPS=_DEFAULT_APPS,
    MIDDLEWARE=_DEFAULT_MIDDLEWARE,
)

# Inject Pydantic settings into Django settings
to_django(django_settings)

I find using init vars that takes precedence sometimes more appropriate than having defaults that can be messed up with yaml config. More importantly - end user might not know "how we intend to run app in X env", this knowledge will have to be documented somewhere.

Hopefully this will save someones day one day :D

[erhosen]

Kudos, and thanks for sharing your approach 👍🏻