Allow recursive env variables loading based on models jsonschema

[jooola]

Initial Checks

• [X] I have searched Google & GitHub for similar requests and couldn't find anything
• [X] I have read and followed the docs and still think this feature is missing

Description

I've implemented a recursive env variables parser based on the models json schema (only used for the schema structure and field names). It isn't complete and only support the cases I needed, but I propose to upstream it or maybe package it as plugin if the pydantic maintainers have some interest.

I wasn't happy with the json based env values, so I created this.

The code is here:

• https://github.com/libretime/libretime/blob/e6945a0e451a53979e294d1d9801194e71f58b4b/shared/libretime_shared/config/_env.py
• https://github.com/libretime/libretime/blob/e6945a0e451a53979e294d1d9801194e71f58b4b/shared/libretime_shared/config/_base.py#L73-L108
• https://github.com/libretime/libretime/blob/e6945a0e451a53979e294d1d9801194e71f58b4b/shared/tests/config/env_test.py

It has some specific (non exhaustive) features:

• loading union types
• loading primitive types arrays using either a csv string or an indexed env name (MY_STR_LIST=one,two or MY_STR_LIST_0=one)
• loading complex types arrays (obj/unions)
• merging list onto another one should update items based on there index in the array. For example MY_OBJ_LIST_2_FIELD=changed should update an element at the index 2 in the array of objects.

And limitations:

• The one_two.three and one.two_three fields have overlapping env names if the delimiter is _, work around is to use a __ (double _) delimiter.
• Doesn't yet allow env_name override from config definition.

I've seem some work towards a similar feature in pydantic/pydantic#4216, and maybe they both overlap in features, but both implementation could live next to each other (probably as plugin ?) and let users decide what is best for there use case.

I am happy to provide a Pull request, but I prefer to talk a bit about this addition before I try anything. Maybe this has a better place in a community package (if one is planned ?) that people can install if they need tooling around pydantic ?

Affected Components

• [ ] Compatibility between releases
• [ ] Data validation/parsing
• [ ] Data serialization - .dict() and .json()
• [ ] JSON Schema
• [ ] Dataclasses
• [ ] Model Config
• [ ] Field Types - adding or changing a particular data type
• [ ] Function validation decorator
• [ ] Generic Models
• [ ] Other Model behaviour - construct(), pickling, private attributes, ORM mode
• [X] Settings Management
• [ ] Plugins and integration with other tools - mypy, FastAPI, python-devtools, Hypothesis, VS Code, PyCharm, etc.

[samuelcolvin]

Interesting idea, to avoid confusion, I think we should have one correct implementation, then the existing route of custom sources for everything else.

I don't think it's worth submitting a PR not since I'm pushing to get v1.10 released so probably won't accept new PRs for this kind of thing.

Also, I think settings will move to a separate package in V2, see #4240.

There won't be a "community" package I don't think, but I can add a section to the docs with links to community developed packages.

[jooola]

I don't think it's worth submitting a PR not since I'm pushing to get v1.10 released so probably won't accept new PRs for this kind of thing.

I didn't expect this to be merged in 1.10 that's for sure. I was more thinking for the long term.

Interesting idea, to avoid confusion, I think we should have one correct implementation, then the existing route of custom sources for everything else.

On a backward compatibility perspective, I know this will never be accepted as the default implementation. But if this implementation turns out to be easier to maintain and more powerful, would pydantic-settings consider a transition ?

If not I should probably create a plugin package.

Also, I think settings will move to a separate package in V2, see https://github.com/pydantic/pydantic/discussions/4240.

Maybe this is too soon, but I am wondering if pydantic-settings is the final name of this package ? If I name my package I would like it to extend the pydantic settings name.

[samuelcolvin]

I'm pretty sure that if we create a separate package, it'll be called pydantic-settings.

One of the changes I want to make in settings in V2 is to make all sources except arguments and field defaults "sources" following that interface, when if the env_source is installed by default.

So perhaps adding your source to pydantic-settings isn't such a bad idea.

[jooola]

I've come to realize that the schema definition passed to the new pydantic-core validator could be really easier to use than relying on the generated jsonschema. Is this validation schema stable ? And is there a way to add extra data to this schema ? For example the previous env_name ?

In addition, could this ticket be transfered to pydantic/pydantic-settings ?

[samuelcolvin]

I'm not sure on your question, but will transfer.