Order dependent dict overwriting environment-variable issue

[merren-fx]

Hey all, I am experiencing a (to me) unexpected be behavior when parsing properties of type dict from env-vars.

When trying to set a dictionary value directly using after setting is using a json-encoded object in a env variable an error is thrown:

Traceback (most recent call last):
  File "a.py", line 62, in <module>
    print(Parent())
          ^^^^^^^^
  File ".venv/lib/python3.11/site-packages/pydantic_settings/main.py", line 85, in __init__
    **__pydantic_self__._settings_build_values(
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/pydantic_settings/main.py", line 187, in _settings_build_values
    return deep_update(*reversed([source() for source in sources]))
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/pydantic_settings/main.py", line 187, in <listcomp>
    return deep_update(*reversed([source() for source in sources]))
                                  ^^^^^^^^
  File ".venv/lib/python3.11/site-packages/pydantic_settings/sources.py", line 323, in __call__
    field_value = self.prepare_field_value(field_name, field, field_value, value_is_complex)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/pydantic_settings/sources.py", line 504, in prepare_field_value
    env_val_built = self.explode_env_vars(field_name, field, self.env_vars)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/pydantic_settings/sources.py", line 622, in explode_env_vars
    env_var[last_key] = env_val
    ~~~~~~~^^^^^^^^^^
TypeError: 'str' object does not support item assignment

My pydantic versions:

Package                           Version
--------------------------------- -----------
pydantic                          2.6.4
pydantic_core                     2.16.3
pydantic-settings                 2.2.1

Minimal example:

import os
import json 

from typing import Any, Dict 
from pydantic import field_validator, model_validator
from pydantic_settings import SettingsConfigDict
from pydantic_settings import BaseSettings as PydanticBaseSettings

old_environ = os.environ.copy()

class _SettingsBase(PydanticBaseSettings):
    # pydantic-internal
    model_config = SettingsConfigDict(
        env_nested_delimiter='__',
        extra='forbid',
        case_sensitive=True,
        frozen=True,
        revalidate_instances='always')
    @model_validator(mode='before')
    @classmethod
    def validator_model_before(cls, data: Any):
        if isinstance(data, str):
            data = json.loads(data)
        return data

class SubType(_SettingsBase):
    SUBFIELD: dict = {'a': 1, 'b': 2}
    @field_validator("SUBFIELD", mode='before')
    @classmethod
    def validate_subfield_from_str(cls, v) -> Any:
        if isinstance(v, str):
            v = json.loads(v)
        return v
class Parent(_SettingsBase):
    SUB: SubType = SubType()
    SUB_DICT: Dict[str, SubType] = {}
def setenv(env: dict):
    os.environ.clear()
    os.environ.update(old_environ)
    os.environ.update(env)
setenv({
    'SUB__SUBFIELD' : json.dumps({'other': 'stuff'})
})
print(Parent())
#SUB=SubType(SUBFIELD={'other': 'stuff'}) SUB_DICT={}

setenv({
    'SUB': SubType(
        SUBFIELD={'other': 'stuff'}).model_dump_json(),
    'SUB_DICT': json.dumps({"FOO": SubType(
        SUBFIELD={'other': 'indict'}).model_dump()})
})
print(Parent())
#SUB=SubType(SUBFIELD={'other': 'stuff'}) SUB_DICT={'FOO': SubType(SUBFIELD={'other': 'indict'})}

setenv({
    'SUB': SubType(
        SUBFIELD={'other': 'stuff'}).model_dump_json(),
    'SUB_DICT__FOO': SubType(
        SUBFIELD={'other': 'indict'}).model_dump_json(),
    'SUB_DICT__FOO__SUBFIELD': json.dumps({"subsub": "foobar"})
})
print(Parent())
# Error

When SUB_DICT__FOO__SUBFIELD and SUB_DICT__FOO are swapped the example code works. Is this really the expected behavior ?

[hramezani]

Thanks @merren-fx for reporting this.

Extracting data from env variables for nested model is not completed and probably is not working for all the cases. I've added a fix for your use case in https://github.com/pydantic/pydantic-settings/pull/261

It would be great if you can test it.

[merren-fx]

Hey @hramezani #261 seems to fix the issue I mentioned.

But while we are on the topic: is this the expected behavior:

• env.set('sub_dict', '"bar": { }')
  • fails
• env.set('sub_dict', '{"bar": { }}')
  • works

my tests

If not I am happy to contribute.

[hramezani]

Thanks @merren-fx for checking.

The first one does not work because '"bar": { }' is not a valid json and for the same reason the second one is working

[hramezani]

Fixed in https://github.com/pydantic/pydantic-settings/commit/a853a132cfeeb69b72c8c9f0378f860e9becfd3f