ENH: Recursive filepath expansion

[cpelley]

Make Input plugin recursively expand the string. This enables having a formatting string embedded within a formatting string. We want to avoid at all cost referencing any environment variables directly within science configurations. Note that filepath patterns themselves can be constructed from environment variables and that's fine - these are then expanded by the plugin. We simply don't want to hardcode environment variable names anywhere within the configuration itself).

What does this mean?

Defining filepaths within the orchestration layer, rather than hardcoding them within the configuration itself.

input_settings = {
    "call": (
        "graph_config.plugins.Input",
        {
            "current_cycle": None,
            "data_input_path": None,
            "filepath": "{data_input_path}",
        },

    )
}

Illustrative json template variables within the orchestration:

{%- set DATA_INPUT_PATH = {
    "config1": "/path/to/share/cycle/{cyclepoint:%Y%m%dT%H%MZ}/decoupler/bla/teacup*/level_1/{validitytime:%Y%m%dT%H%MZ}-{isoleadtime}-{diagnostic}.nc",
    "config2": "/path/to/share/cycle/{cyclepoint:%Y%m%dT%H%MZ}/decoupler/bling/bleh*/level_1/{validitytime:%Y%m%dT%H%MZ}-{isoleadtime}-{diagnostic}.nc",
    }
%}

Issues

• https://github.com/MetOffice/improver_suite/issues/2253
• https://github.com/MetOffice/improver_suite/issues/2079

[cpelley]

of likely interest to you @JoshuaWiggs