[Feature Request] Minimal Version of Hydra With Bare-Bones Config Support

[playertr]

🚀 Feature Request

I am requesting a stripped-down version of Hydra with only support for YAML+StructuredConfig configuration and CLI support.

Motivation

Hydra's features of hierarchical configuration, seamless CLI support, plug-and-play architecture, and organizing runs with an output directory are incredible assets for an ML training workflow. Thank you for this terrific contribution to open source ML! However, I often find myself wanting only Hydra's basic configuration support -- including the CLI interface and YAML+StructuredConfig specification -- without creating a new working directory with logs, and without the advance launching, sweeping, and logging functionality.

This feature is especially important for small command line utilities, such a script to download models from a remote database, which really only need to execute with transparent configuration and no logging side effects. When I create these utilities as standalone scripts, I am reluctant to include Hydra as a dependency because these applications do not require Hydra's full set of rich features. I don't want to confuse potential users by introducing concepts such as workdir-changing and parameter sweeps for simple command-line utilities. However, the alternatives to Hydra, including Google Fire and manual pyyaml or ConfigParser, do not natively support a hierarchical configuration folder like Hydra does (as far as I know).

Pitch

I am not sure if this is a valid feature request because it is requesting a major difference, but I would love to see a minimal version of Hydra with the sole purpose of configuring and running scripts without changing directories. This version could be used for non-ML projects where scripts should have minimal side effects. Perhaps the user could install a reduced executable with pip install hydra-core[minimal], and it might have a different python invocation, such as:

import hydra_scripts as hydra

@hydra.main(config_path=None, config_name="main_cfg")
def app(cfg: DictConfig) -> None:
    # ...

Several feature requests have expressed interest in running Hydra without changing working directories, but my suggestion is different in that it involves releasing a minimal version that would be suitable for inclusion in a non-ML app (as an alternative to argparse+pyYAML, Click, or Fire requiring less boilerplate for static configuration). Relevant issues:

#1937: Conditionally disable the creation of folders in output/multirun via the CLI #990: Disable changing working directory

If Hydra developers or other members of the community have guidance or opinions on whether this feature fits within the scope of Hydra (or whether it would be well suited to its own project), I would love to hear your ideas! I may be overlooking another library which offers these configuration and execution features, too -- please let me know if so!

[pixelb]

Yes Hydra is generally very useful outside of ML, and you're right it's currently a bit awkward to use in the general case. Note hydra only depends on OmegaConf and antlr, which would be needed for your use cases anyway. The dev version of hydra already supports config for not changing directory. This will be the default or at least easily configurable, so we'd be leaning towards making hydra generally useful with the default config, rather than splitting it

[jieru-hu]

@playertr thanks for the issue. have you checked out compose_api? It is a leaner way to use Hydra for config composition.

[bradenkinard]

I second this feature request. Even within ML contexts, I find I often wish to use Hydra but without changing the current working directory, outputting logs of configurations, etc. While this behavior is sometimes very useful, they are ultimately side effects of what I, and I presume many, users are utilizing hydra for, which is the hierarchical configuration and CLI features. Side effects like those in question are often, at best, not relevant but ignorable, but at worst can be a headache to workaround and can make working with other frameworks and libraries more difficult.

In regards to the response by @pixelb, there is a gap between simply using OmegaConf and Hydra as they currently stand. In either case, it requires quite a bit of finagling to get to the streamlined level of functionality that @playertr is talking about. Disabling this functionality through Hydra configuration isn’t completely straightforward and the need to do so certainly isn’t something many users expect when they first start utilizing Hydra.

Perhaps rather than a completely separate, slimmed down version of Hydra, it would make sense to have these side effect features something that are disabled by default and opt-in through Hydra config, rather than the current situation in which they are on by default and must be opted out of?