Support `keyring` encrypted credential storage

[lmmx]

Outline

• Setting env. variables for auth config is awkward, so python-dotenv can be used to load them from a .env text file instead

• Keyrings can be preferable to plain text secret storage for security, and Python has a keyring module [source]

  These recommended keyring backends are supported:

  • macOS Keychain
  • Freedesktop Secret Service supports many DE including GNOME (requires secretstorage)
  • KDE4 & KDE5 KWallet (requires dbus)
  • Windows Credential Locker

• This practice is used by other well known tools such as the GitHub CLI tool gh and twine [source]

  Where Twine gets configuration and credentials A user can set the repository URL, username, and/or password via command line, .pypirc files, environment variables, and keyring.

• This could be made an optional dependency if desired (perhaps to keep package size minimal/consistent).

Impact

When I pip install keyring on Linux after first installing pydantic and pydantic-settings the additional dependencies are:

Installing collected packages: zipp, pycparser, more-itertools, jeepney, jaraco.classes, importlib-metadata, cffi, cryptography, SecretStorage, keyring

Usage

Once installed, secret access is achieved like so:

import keyring
my_secret = keyring.get_password("MY_SECRET", "secret_username")

The gh tool sets the username to an empty string, indicating that it's used as a simple key-value secret store.

You can also access specific keyrings, also known as 'collections' (for instance if you wanted to have different applications using different keys with the same name, say a different API key for different services). For reference

Proposed implementation

Essentially we are replacing os.environ.get(validation_alias) for keyring.get_password(validation_alias)

In this library, both environment variables and .env configured variables are loaded into the env_vars attribute.

EnvSettingsSource calls _load_env_vars() at initialisation:

https://github.com/pydantic/pydantic-settings/blob/5933ea6ca51d58342378eed5db12d3f6f6dee8df/pydantic_settings/sources.py#L376-L381

DotEnvSettingsSource subclasses EnvSettingsSource and overrides the _load_env_vars() method

https://github.com/pydantic/pydantic-settings/blob/5933ea6ca51d58342378eed5db12d3f6f6dee8df/pydantic_settings/sources.py#L571-L590

I would have this work similarly to .env handling with a subclass exposing a custom way to load env vars.

We can enumerate all keys (as bytes) via:

all_items = keyring.core.get_keyring().get_preferred_collection().get_all_items()
keyring_vars: dict[str, str] = {
    item.get_attributes()["service"]: item.get_secret().decode()
    for item in all_items
}

(In real code you'd have to have some error handling in case the 3 chained methods error!)

I think default conversion of bytes to str type would be reasonable here?

Proof of concept

The attached PR supplies a working implementation of this feature on Linux, using the SecretStorage backend.

(keyringsettings) louis 🚶 ~/dev/testing/pydantic-settings $ keyring set MY_SECRET_KEY ''
Password for '' in 'MY_SECRET_KEY': 
(keyringsettings) louis 🚶 ~/dev/testing/pydantic-settings $ keyring get MY_SECRET_KEY ''
abc

from pydantic_settings import BaseSettings, SettingsConfigDict

class Settings(BaseSettings):
    MY_SECRET_KEY: str
    model_config = SettingsConfigDict(extra="ignore")

s = Settings()
print(s)

(keyringsettings) louis 🚶 ~/dev/testing/pydantic-settings $ python keyring_demo.py 
MY_SECRET_KEY='abc'

This is overridden by setting an environment variable.

(keyringsettings) louis 🚶 ~/dev/testing/pydantic-settings $ export MY_SECRET_KEY="foo"
(keyringsettings) louis 🚶 ~/dev/testing/pydantic-settings $ python keyring_demo.py 
MY_SECRET_KEY='foo'
(keyringsettings) louis 🚶 ~/dev/testing/pydantic-settings $ unset MY_SECRET_KEY
(keyringsettings) louis 🚶 ~/dev/testing/pydantic-settings $ python keyring_demo.py 
MY_SECRET_KEY='abc'

Selected Assignee: @dmontagu

[dmontagu]

Since @hramezani already replied on the PR I'll assign him here, but in general I think it makes sense for us to support/maintain this if you are willing to provide an initial implementation. (And it seems like you are — thank you!)

[ELC]

Are there any updates on this feature?

[lmmx]

Are there any updates on this feature?

Happy annual kanban refresh to all who celebrate: I've put it back onto the TODO list :face_with_peeking_eye: :hourglass_flowing_sand: