Pydantic settings not reloading env vars when .env file is updated

pongpatapee commented 5 months ago

Initial Checks

[X] I confirm that I'm using Pydantic V2

Description

Bug

Context: I was using Pydantic's BaseSettings to configure some Database settings.

The issue I faced was no matter what values I updated in the .env, the output of the settings always remained the same. This led me to:

Commenting out all values in the .env
Deleting the .env file
Changing .env_file field in model_config to random paths/strings

All of which resulted in the same values being outputted (The cached values that I have set before which were not updating)

My initial bug report was going to be that Pydantic is reading off a non-existent .env file. However, after more debugging, I realized that these environment variables were persistently set for my current directory (i.e., creating a new shell session did not fix the issue).

This means that Pydantic settings somehow previously set persistent environment variables that are not being overridden with the .env ones. I am unsure if this is expected behavior, but this feels like a bug to me, please correct me if I'm wrong.

Additionally, Pydantic is not validating the .env file path. For example, I can set the .env_file in model_config to any random string, and as long as the environment variables the model is expecting exist, no errors will be thrown.

E.g., this throws no errors

    model_config = SettingsConfigDict(
        env_file="random string",
        env_file_encoding="utf-8",
        env_prefix="DB_",
        case_sensitive=False,
        extra="ignore",
    )

How to reproduce

Set environment variables of what your model is expecting in your current shell
Define model_config in your setting class to expect an env_file
Put any path/string, valid or invalid, to the env_file
Print out the values in your setting class. No errors will be thrown given an invalid .env path, updated values will not be populated given a valid .env path.

Example Code

# .env
DB_USERNAME=test_user
DB_PASSWORD=test_password
DB_HOST=test_server

# Before running config.py
export DB_USERNAME=user_to_be_overridden
export DB_PASSWORD=password_to_be_overridden
export DB_HOST=server_to_be_overrriden

# config.py
import sqlalchemy
from pydantic import SecretStr, ValidationInfo, field_validator
from pydantic_settings import BaseSettings, SettingsConfigDict

class DatabaseSettings(BaseSettings):
    model_config = SettingsConfigDict(
        env_file=".env", # Can be replaced with an invalid path/string
        env_file_encoding="utf-8",
        env_prefix="DB_",
        case_sensitive=False,
        extra="ignore",
    )

    USERNAME: str
    PASSWORD: SecretStr
    HOST: str
    NAME: str
    PORT: str | None = None
    URL: str | None = None

    @field_validator("URL")
    @classmethod
    def assemble_db_url(cls, v: str, info: ValidationInfo) -> str:
        if isinstance(v, str):
            return v

        url = sqlalchemy.URL.create(
            drivername="mssql+pyodbc",
            username=info.data.get("USERNAME"),
            password=info.data.get("PASSWORD").get_secret_value(),
            host=info.data.get("HOST"),
            port=info.data.get("PORT"),
            database=info.data.get("NAME"),
            query=dict(driver="ODBC Driver 17 for SQL Server"),
        )

        return url.render_as_string(hide_password=False)

if __name__ == "__main__":
    from pprint import pprint

    database_settings = DatabaseSettings(NAME="TEST")

    print("Database settings model dump:")
    pprint(database_settings.model_dump())
    print("Database settings model config:")
    pprint(DatabaseSettings.model_config)

    """
    OUTPUT:

    Database settings model dump:
    {'HOST': 'server_to_be_overrriden', # Not value from .env
    'NAME': 'TEST',
    'PASSWORD': SecretStr('**********'),
    'PORT': None,
    'URL': 'mssql+pyodbc://user_to_be_overridden:password_to_be_overridden@server_to_be_overrriden/TEST?driver=ODBC+Driver+17+for+SQL+Server', # Not value from .env
    'USERNAME': 'user_to_be_overridden'} # Not value from .env
    Database settings model config:
    {'arbitrary_types_allowed': True,
    'case_sensitive': False,
    'env_file': '.env',
    'env_file_encoding': 'utf-8',
    'env_ignore_empty': False,
    'env_nested_delimiter': None,
    'env_parse_none_str': None,
    'env_prefix': 'DB_',
    'extra': 'ignore',
    'json_file': None,
    'json_file_encoding': None,
    'protected_namespaces': ('model_', 'settings_'),
    'secrets_dir': None,
    'toml_file': None,
    'validate_default': True,
    'yaml_file': None,
    'yaml_file_encoding': None}

    """

Python, Pydantic & OS Version

pydantic version: 2.6.4
        pydantic-core version: 2.16.3
          pydantic-core build: profile=release pgo=true
                 install path: C:\Users\dan\AppData\Local\anaconda3\envs\traceability\Lib\site-packages\pydantic
               python version: 3.11.8 | packaged by Anaconda, Inc. | (main, Feb 26 2024, 21:34:05) [MSC v.1916 64 bit (AMD64)]
                     platform: Windows-10-10.0.19045-SP0
             related packages: pydantic-settings-2.2.1 typing_extensions-4.10.0
                       commit: unknown

hramezani commented 5 months ago

This means that Pydantic settings somehow previously set persistent environment variables that are not being overridden with the .env ones. I am unsure if this is expected behavior, but this feels like a bug to me, please correct me if I'm wrong.

pydantic-settings never set the env variables. It only reads environment variables and builds the settings model.

Additionally, Pydantic is not validating the .env file path. For example, I can set the .env_file in model_config to any random string, and as long as the environment variables the model is expecting exist, no errors will be thrown.

Yes, if the env file does not exist, pydantic-settings ignore the file and doesn't load the values from the file. no error will be raised for this.

env variables that load by env settings source have priority over dotenv file values. it is mentioned in the doc Even when using a dotenv file, pydantic will still read environment variables as well as the dotenv file, environment variables will always take priority over values loaded from a dotenv file.

you can change the priority if you want.

pongpatapee commented 5 months ago

Ah I see, thanks for the response, that explains why the .env file was ignored.

pydantic-settings never set the env variables. It only reads environment variables and builds the settings model.

My reproducibility steps aside, I never set those environment variables myself, they were all originally from the .env file. I thought pydantic was responsible for this, but I can't seem to reproduce the issue in isolation.

It is possible that vscode and/or conda were responsible for setting those environment variables. Doing env | grep DB_ shows my environment variables as part of VSCODE_ENV_REPLACE=CONDA_DEFAULT_ENV in addition to the isolated variables themselves.

I'll close the issue for now, if I can identify the root cause coming from pydantic I can re-open the issue then.

Thanks!

pydantic / pydantic-settings