Closed paudom closed 3 years ago
This can be achieved using defaults lists (or the command-line override syntax).
To fix an example, here is a minimal test.py
file:
$ cat test.py
from omegaconf import OmegaConf
import hydra
@hydra.main(config_path="conf", config_name="config")
def my_app(cfg):
print(OmegaConf.to_yaml(cfg))
if __name__ == "__main__":
my_app()
With the configs you gave above, I'm getting this output when I run test.py
:
$ python test.py
foo:
f: 7
g: 3
h: 1
bar: {}
If I understand correctly, your goal is to produce this output:
foo:
f: 7
g: 3
h: 1
bar:
foo:
a: 4
b: 5
c: 6
This can be achieved using an absolute default in the defaults list for bar1.yaml
:
# conf/bar/bar1.yaml
defaults:
- /foo: foo1
Alternatively, you can achieve the same result using an @package
option at the command line:
$ python test.py +foo@bar.foo=foo1
Equivalently, you could add an element to the end of your primary config's defaults list:
# conf/config.yaml
defaults:
- _self_
- foo: foo2
- bar: bar1
- foo@bar.foo: foo1
The first solution uses '/foo'
with a leading slash '/'
to select the 'foo'
config group inside of the 'conf'
folder. If not for the leading slash, hydra would look for a group named 'foo'
nested inside of 'bar'
(such a group does not exist). Having selected the foo/foo1
config, this config is placed in the bar.foo
package.
As for the second solution: +foo@bar.foo=foo1
This boils down to:
+
symbol means that an entry should be appended to the top-level defaults listfoo
. This means hydra will search within the conf/foo
directory for a config.bar.foo
. This means that, once a config is located, it will be merged into the output config at the location bar.foo
.foo
group is foo1
.References:
Everything is much clear now, thanks a lot!!
@Jasha10 Hi, can I include it as a list item? e.g.
---
foo:
f: 7
g: 3
h: 1
bar:
- a: 4
b: 5
c: 6
Hi @Yevgnen, Hydra has first-class support for merging dicts together, but not for adding to lists.
There are some ways to achieve what you want (using interpolation, e.g. bar: ["${..my_temporary_variable}"]
), but these methods are difficult to work with. I'd recommend working with dictionaries if possible (instead of lists).
If you change bar1.yaml
to look like this:
# conf/bar/bar1.yaml
defaults:
- /foo@0: foo1
Then you can get the following as output:
$ python test.py
foo:
f: 7
g: 3
h: 1
bar:
'0':
a: 4
b: 5
c: 6
This is using the string "0"
so that the dictionary will be more similar to a list. Would something like that work for you?
@Jasha10 Hi, thanks for the details. My orignal X problem is trying to instantiate recursive objects like this example
car:
_target_: my_app.Car
driver:
_target_: my_app.Driver
name: James Bond
age: 7
wheels:
- _target_: my_app.Wheel
radius: 20
width: 1
- _target_: my_app.Wheel
radius: 20
width: 1
- _target_: my_app.Wheel
radius: 20
width: 1
- _target_: my_app.Wheel
radius: 20
width: 1
but my wheels
are in other config group so I want to include them in car
.
The real world example is here. Since the logger configs are not included in trainer, so one has to first instantiate the loggers then the trainer. I'm trying to find a way to get it work but not sure if it's idiomatic.
Aah, thanks for the context :)
I think you are confused about the lightning-hydra template -- you do not need a list of loggers, you need a dict of loggers. Take a look at line 54 from the link you gave:
for _, lg_conf in config.logger.items(): ...
The for loop is iterating over config.logger.items()
. This means that config.logger
should be a mapping, not a sequence.
Take a look also at the lightning-hydra many_loggers
config, as it is relevant to your use-case.
You can run the lightning-hydra template with many loggers like this:
$ python run.py logger=many_loggers
For future reference, there is an idiomatic way to get a list of values from a mapping: you can use OmegaConf's oc.dict.values
resolver.
In yaml
, that would look like this:
# config.yaml
car:
_target_: my_app.Car
driver:
_target_: my_app.Driver
name: James Bond
age: 7
wheels: "${oc.dict.values: _wheel_dict}" # gets values from _wheel_dict
_wheel_dict:
wheel0:
_target_: my_app.Wheel
radius: 20
width: 1
wheel1:
_target_: my_app.Wheel
radius: 20
width: 1
wheel2:
_target_: my_app.Wheel
radius: 20
width: 1
wheel3:
_target_: my_app.Wheel
radius: 20
width: 1
# run_app.py
from omegaconf import OmegaConf, ListConfig
cfg = OmegaConf.load("config.yaml")
assert isinstance(cfg.car.wheels, ListConfig)
assert len(cfg.car.wheels) == 4
assert cfg.car.wheels[0].radius == 20
@Jasha10 Hi
Take a look at line 54 from the link you gave for _, lg_conf in config.logger.items(): ...
That's exactly what I'm trying to avoid.
I'd like to build the loggers and trainer together using single recursive instantiate call like this example instead of the two indivdual calls like the lightning-hydra template. e.g. to use hydra.utils.instantiate(config.trainer)
instead of for ... hydra.utils.instantiate(logger_conf)
and hydra.utils.instantiate(config.trainer, logger=loggers)
.
Since I also want to put the logger config alone (I don't want to put all logger configs as a list into the trainer part), so I am looking for a way to include them into the trainer config.
The future reference looks good to me. Thanks for letting me know that!
That's exactly what I'm trying to avoid.
I see. In that case my advice would be to create a mapping whose values have a _target_
, then call ${oc.dict.values: ...}
on that mapping.
π Feature Request
Hi! I was wondering if with Hydra 1.1 it's possible to include certain config files into variables of another config file. I've looked at the documentation Extending Configs but I can't figure out how to make it work for my use case.
Pitch
Giving more context, given the following folder structure:
And the following content on these files:
Is there any way (using extensions or interpolations), that in the
conf/bar/bar1.yaml
I could have a variable namedfoo
under where I could 'copy' the same content asconf/foo/foo1.yaml
like this:I know that in the case of using
foo1
instead offoo2
in the defaults list from theconfig.yaml
file I could interpolate the values using:But in my use case, I want
foo
to befoo2
by default while inconf/bar/bar1.yaml
I would want a variable that is the same (including/copying) content as inconf/foo/foo1.yaml
.Thanks in advance!!
Additional context
Using Hydra 1.1, OmegaConf 2.1, and Python 3.8