Pattern advice: How to use config file with required field without setting it?

[banool]

Hey hey,

I just read a few issues in this repo but so far I haven't been able to figure out if what I want to do is possible.

To me, one of the main advantages of reading configuration from a file is you can define exactly which fields are required and which are optional. This lets you shift a great amount of validation into the parsing layer.

Assume we have this function:

pub fn build_config(fname: PathBuf): Result<Config> {
    let file = File::open(fname)?;
    let reader = BufReader::new(file);
    let config: Self = serde_yaml::from_reader(reader)?;
    Ok(config)
}

This first code snippet here is much better than the second:

#[derive(Deserialize, Serialize)]
pub struct Config {
    postgres_connection_string: String,
}

fn main() -> Result<()> {
    let config = build_config("config.yaml")?;
    use_database(config.postgres_connection_string)?;
    Ok(())
}

#[derive(Deserialize, Serialize)]
pub struct Config {
    postgres_connection_string: Option<String>,
}

fn main() -> Result<()> {
    let config = build_config("config.yaml")?;
    use_database(config.postgres_connection_string.expect("postgres_connection_string was not set!"))?;
    Ok(())
}

Problem is, sometimes I want to declare a field in my config as not an Option, but I also don't want to read it from the config file sometimes (e.g. in some environments it's easier to read such fields as env vars). My first thought was Figment would be that magic solution. I have this code:

let config = Figment::new()
    .merge(Yaml::file(args.config_path))
    .merge(Env::prefixed("APP_"))
    .extract()
    .context("Failed to load config")?;

My assumption was this would happen:

1. Read config file.
2. Read env vars.
3. If I set a value in the env vars it would override what's in the config file, even if that value was missing.

Unfortunately this last part doesn't seem to be the case:

even if that value was missing

This code doesn't work:

APP_POSTGRES_CONNECTION_STRING='testing' cargo run --config-path config.yaml

I get this error:

Error: Failed to load config

Caused by:
    missing field `postgres_connection_string`

If the value is missing from the file then it doesn't matter that I set an env var for it. I thought perhaps there would be some kind of config "staging area" where Figment builds the final config, but it seems like maybe it just uses the underlying tool at each stage, so each config stage needs to be complete independently.

I have tried using figment_file_provider_adapter but that doesn't seem to help either (e.g. by using the _FILE env var suffix).

Is what I want to do possible with Figment?

[banool]

Oh well well I actually just made a minimal example and it works how I want!

src/main.rs

use figment::providers::Env;
use figment::providers::Format;
use figment::providers::Yaml;
use figment::Figment;
use serde::Deserialize;
use serde::Serialize;

#[derive(Debug, Deserialize, Serialize)]
pub struct Config {
    postgres_connection_string: String,
}

fn main() {
    let config: Config = Figment::new()
        .merge(Yaml::file("config.yaml"))
        .merge(Env::prefixed("APP_"))
        .extract()
        .unwrap();
    dbg!(config);
}

config.yaml

$ APP_POSTGRES_CONNECTION_STRING=yo cargo run
[src/main.rs:19] config = Config {
    postgres_connection_string: "yo",
}

In reality my config is much more complicated and nested. I'll see if I can find which part of this causes it not to work.

[banool]

Ha! I see, the issue was using an env var for a nested value.

Code:

use figment::providers::Env;
use figment::providers::Format;
use figment::providers::Yaml;
use figment::Figment;
use serde::Deserialize;
use serde::Serialize;

#[derive(Debug, Deserialize, Serialize)]
#[serde(tag = "type", rename_all = "snake_case")]
#[serde(deny_unknown_fields)]
pub enum RootConfig {
    This(ThisConfig),
    That(ThatConfig),
}

#[derive(Debug, Deserialize, Serialize)]
#[serde(deny_unknown_fields)]
pub struct ThisConfig {
    this_value: String,
    storage_config: StorageConfig,
}

#[derive(Debug, Deserialize, Serialize)]
#[serde(deny_unknown_fields)]
pub struct ThatConfig {
    that_value: String,
}

#[derive(Debug, Deserialize, Serialize)]
#[serde(deny_unknown_fields)]
pub struct StorageConfig {
    postgres_connection_string: String,
}

fn main() {
    let config: RootConfig = Figment::new()
        .merge(Yaml::file("config.yaml"))
        .merge(Env::prefixed("APP_").split("zzz"))
        .extract()
        .unwrap();
    dbg!(config);
}

Successful invocation:

APP_STORAGE_CONFIGzzzPOSTGRES_CONNECTION_STRING=yo cargo run

Certainly a little awkward, it's unfortunate that you can generally only use [a-zA-Z0-9_] as env var key characters. I found that using _ didn't work as I'd hope because it doesn't really know where to split stuff if the fields themselves have underscores in them.

[banool]

I see some kind of curly brace syntax is supported but I can't quite seem to get it to work. Nonetheless, just using .split("___") is probably sufficient for me. Thanks!

[SergioBenitez]

I see some kind of curly brace syntax is supported but I can't quite seem to get it to work. Nonetheless, just using .split("___") is probably sufficient for me. Thanks!

The syntax is here: https://docs.rs/figment/latest/figment/providers/struct.Env.html.

APP_STORAGE_CONFIG={postgres_connection_string="value"}

Also, using __ (two instead of three) would suffice.