[Feature Request] Include config files into variables of other config files. (Question)

[paudom]

🚀 Feature Request

Hi! I was wondering if with Hydra 1.1 it's possible to include certain config files into variables of another config file. I've looked at the documentation Extending Configs but I can't figure out how to make it work for my use case.

Pitch

Giving more context, given the following folder structure:

├── conf
│   ├── foo
│   │   ├── foo1.yaml
│   │   └── foo2.yaml
│   ├── bar
│   │   └── bar1.yaml
│   └── config.yaml
└── test.py

And the following content on these files:

# conf/config.yaml
defaults:
 - _self_
 - foo: foo2
 - bar: bar1

# conf/foo/foo1.yaml
a: 4
b: 5
c: 6

# conf/foo/foo2.yaml
f: 7
g: 3
h: 1

Is there any way (using extensions or interpolations), that in the conf/bar/bar1.yaml I could have a variable named foo under where I could 'copy' the same content as conf/foo/foo1.yaml like this:

# conf/bar/bar1.yaml
foo:
 a: 4
 b: 5
 c: 6

I know that in the case of using foo1 instead of foo2 in the defaults list from the config.yaml file I could interpolate the values using:

# conf/bar/bar1.yaml
foo: ${foo}

But in my use case, I want foo to be foo2 by default while in conf/bar/bar1.yaml I would want a variable that is the same (including/copying) content as in conf/foo/foo1.yaml.

Thanks in advance!!

Additional context

Using Hydra 1.1, OmegaConf 2.1, and Python 3.8

[Jasha10]

This can be achieved using defaults lists (or the command-line override syntax).

To fix an example, here is a minimal test.py file:

$ cat test.py
from omegaconf import OmegaConf
import hydra

@hydra.main(config_path="conf", config_name="config")
def my_app(cfg):
    print(OmegaConf.to_yaml(cfg))

if __name__ == "__main__":
    my_app()

With the configs you gave above, I'm getting this output when I run test.py:

$ python test.py
foo:
  f: 7
  g: 3
  h: 1
bar: {}

If I understand correctly, your goal is to produce this output:

foo:
  f: 7
  g: 3
  h: 1
bar:
  foo:
    a: 4
    b: 5
    c: 6

Solution using defaults list

This can be achieved using an absolute default in the defaults list for bar1.yaml:

# conf/bar/bar1.yaml
defaults:
  - /foo: foo1

Solution using the command-line

Alternatively, you can achieve the same result using an @package option at the command line:

$ python test.py +foo@bar.foo=foo1

Equivalently, you could add an element to the end of your primary config's defaults list:

# conf/config.yaml
defaults:
 - _self_
 - foo: foo2
 - bar: bar1
 - foo@bar.foo: foo1

A bit of explanation

The first solution uses '/foo' with a leading slash '/' to select the 'foo' config group inside of the 'conf' folder. If not for the leading slash, hydra would look for a group named 'foo' nested inside of 'bar' (such a group does not exist). Having selected the foo/foo1 config, this config is placed in the bar.foo package.

As for the second solution: +foo@bar.foo=foo1 This boils down to:

• the leading + symbol means that an entry should be appended to the top-level defaults list
• the selected group is foo. This means hydra will search within the conf/foo directory for a config.
• the specified package is bar.foo. This means that, once a config is located, it will be merged into the output config at the location bar.foo.
• the selected config within the foo group is foo1.

References:

• Terminology
• The Defaults List
• Basic Override syntax
• Packages

[paudom]

Everything is much clear now, thanks a lot!!

[Yevgnen]

@Jasha10 Hi, can I include it as a list item? e.g.

---
foo:
  f: 7
  g: 3
  h: 1
bar:
  - a: 4
    b: 5
    c: 6

[Jasha10]

Hi @Yevgnen, Hydra has first-class support for merging dicts together, but not for adding to lists.

There are some ways to achieve what you want (using interpolation, e.g. bar: ["${..my_temporary_variable}"]), but these methods are difficult to work with. I'd recommend working with dictionaries if possible (instead of lists).

If you change bar1.yaml to look like this:

# conf/bar/bar1.yaml
defaults:
  - /foo@0: foo1

Then you can get the following as output:

$ python test.py
foo:
  f: 7
  g: 3
  h: 1
bar:
  '0':
    a: 4
    b: 5
    c: 6

This is using the string "0" so that the dictionary will be more similar to a list. Would something like that work for you?

[Yevgnen]

@Jasha10 Hi, thanks for the details. My orignal X problem is trying to instantiate recursive objects like this example

car:
  _target_: my_app.Car
  driver:
    _target_: my_app.Driver
    name: James Bond
    age: 7
  wheels:
    - _target_: my_app.Wheel
      radius: 20
      width: 1
    - _target_: my_app.Wheel
      radius: 20
      width: 1
    - _target_: my_app.Wheel
      radius: 20
      width: 1
    - _target_: my_app.Wheel
      radius: 20
      width: 1

but my wheels are in other config group so I want to include them in car.

The real world example is here. Since the logger configs are not included in trainer, so one has to first instantiate the loggers then the trainer. I'm trying to find a way to get it work but not sure if it's idiomatic.

[Jasha10]

Aah, thanks for the context :)

I think you are confused about the lightning-hydra template -- you do not need a list of loggers, you need a dict of loggers. Take a look at line 54 from the link you gave: for _, lg_conf in config.logger.items(): ...

The for loop is iterating over config.logger.items(). This means that config.logger should be a mapping, not a sequence.

Take a look also at the lightning-hydra many_loggers config, as it is relevant to your use-case. You can run the lightning-hydra template with many loggers like this:

$ python run.py logger=many_loggers

---

For future reference, there is an idiomatic way to get a list of values from a mapping: you can use OmegaConf's oc.dict.values resolver.

In yaml, that would look like this:

# config.yaml
car:
  _target_: my_app.Car
  driver:
    _target_: my_app.Driver
    name: James Bond
    age: 7
  wheels: "${oc.dict.values: _wheel_dict}"  # gets values from _wheel_dict
_wheel_dict:
  wheel0:
    _target_: my_app.Wheel
    radius: 20
    width: 1
  wheel1:
    _target_: my_app.Wheel
    radius: 20
    width: 1
  wheel2:
    _target_: my_app.Wheel
    radius: 20
    width: 1
  wheel3:
    _target_: my_app.Wheel
    radius: 20
    width: 1

# run_app.py
from omegaconf import OmegaConf, ListConfig

cfg = OmegaConf.load("config.yaml")
assert isinstance(cfg.car.wheels, ListConfig)
assert len(cfg.car.wheels) == 4
assert cfg.car.wheels[0].radius == 20

[Yevgnen]

@Jasha10 Hi

Take a look at line 54 from the link you gave for _, lg_conf in config.logger.items(): ...

That's exactly what I'm trying to avoid.

I'd like to build the loggers and trainer together using single recursive instantiate call like this example instead of the two indivdual calls like the lightning-hydra template. e.g. to use hydra.utils.instantiate(config.trainer) instead of for ... hydra.utils.instantiate(logger_conf) and hydra.utils.instantiate(config.trainer, logger=loggers).

Since I also want to put the logger config alone (I don't want to put all logger configs as a list into the trainer part), so I am looking for a way to include them into the trainer config.

The future reference looks good to me. Thanks for letting me know that!

[Jasha10]

That's exactly what I'm trying to avoid.

I see. In that case my advice would be to create a mapping whose values have a _target_, then call ${oc.dict.values: ...} on that mapping.