case_sensitive doesn't work in optional nested Settings models

[slingshotvfx]

Initial Checks

• [X] I confirm that I'm using Pydantic V2

Description

When using nested pydantic-settings models, I am seeing unexpected behavior with the case_sensitive flag when reading from a .env file.

When case_sensitive=True, things work as expected and are case sensitive, even though the docs say that on windows things should always be case insensitive.

However, when case_sensitive=False, I get ValidationErrors for Optional nested models regardless of what case I use.

Example Code

# .env
NOT_NESTED=works
NESTED__A=fails
nested__B=2

from pydantic import BaseModel
from pydantic_settings import BaseSettings, SettingsConfigDict

class NestedSettings(BaseModel):
    A: str
    B: int

class Settings(BaseSettings):
    not_nested: str
    nested: NestedSettings | None = None

    model_config = SettingsConfigDict(
        case_sensitive=False,
        env_file=".env",
        env_nested_delimiter="__",
        extra="forbid",
    )

print(Settings())

pydantic_core._pydantic_core.ValidationError: 2 validation errors for Settings
nested.A
  Field required [type=missing, input_value={'a': 'fails', 'b': '2'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.7/v/missing
nested.B
  Field required [type=missing, input_value={'a': 'fails', 'b': '2'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.7/v/missing

Changing case_sensitive=True and re-casing the .env file (e.g. nested__A=works) solves the issue. Removing the | None = None solves the issue too, although then of course the nested setting are required.

Python, Pydantic & OS Version

pydantic version: 2.7.1
        pydantic-core version: 2.18.2
          pydantic-core build: profile=release pgo=true
               python version: 3.12.1 (tags/v3.12.1:2305ca5, Dec  7 2023, 22:03:25) [MSC v.1937 64 bit (AMD64)]
                     platform: Windows-11-10.0.22631-SP0
             related packages: fastapi-0.110.3 pydantic-settings-2.2.1 typing_extensions-4.11.0

[sydney-runkle]

@hramezani,

Could you please take a look at this? Thanks!

[hramezani]

Thanks @slingshotvfx for reporting this bug. I will investigate and try to prepare a fix

[hramezani]

@slingshotvfx I created https://github.com/pydantic/pydantic-settings/pull/294 to fix the problem. Could you please confirm it?

[slingshotvfx]

Hey @hramezani, confirmed that #294 fixes the issues I was having with .env files, thanks!

This might be a separate issue, but I'm still a bit confused by the docs :

On Windows, Python's os module always treats environment variables as case-insensitive, so the case_sensitive config setting will have no effect - settings will always be updated ignoring case.

In my tests on windows 11:

• Case Sensitive: True, .env file works
• Case Sensitive: False, .env file works
• Case Sensitive: True, os.environ does NOT work
• Case Sensitive: False, os.environ works

The example given in the docs fails for me:

class RedisSettings(BaseModel):
    host: str
    port: int

class Settings(BaseSettings, case_sensitive=True):
    redis: RedisSettings

os.environ["redis"] = '{"host": "localhost", "port": 6379}'
print(Settings().model_dump())

pydantic_core._pydantic_core.ValidationError: 1 validation error for Settings
redis
  Field required [type=missing, input_value={}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.7/v/missing

But it works if I set case_sensitive=False.

[hramezani]

@slingshotvfx Thanks for checking. I am going to merge the PR and it will close the issue. It would be great to create a new issue for the things that you mentioned in your last comment.