[Question] Is it possible to put config group and defaults in subfolder?

[arisliang]

I have a folder structure like below:

├── conf-hydra
│   ├── common
│   │   └── config-common.yaml
│   └── sub_conf
│       ├── config-default.yaml
│       └── db
│           ├── mysql.yaml
│           └── postgresql.yaml
└── hydra_app.py

The config path is: @hydra.main(config_path="conf-hydra", config_name="sub_conf/config-default")

I'm trying to specify the config group db to use mysql by doing:

python hydra_app.py -m +db=mysql

And the result is:

sub_conf:
  common:
    year: '2021'
db: mysql

It doesn't get to the entries in mysql.yaml.

So the question, is there possible to get the mysql.yaml entries if the config group db and default config-default.yaml are in a subfolder?

hydra version: '1.1.0.rc1'

[omry]

You don't have a config group db, you have a config group sub_conf/db.

[arisliang]

My bad, tried the follows:

python hydra_app.py -m +sub_conf/db=mysql

gives this error:

In 'sub_conf/config-default': Could not find 'sub_conf/sub_conf/db/mysql' Config search path: provider=hydra, path=pkg://hydra.conf provider=main, path=file://C:\hydra-example\conf-hydra provider=schema, path=structured://

Do I need to configure search path somehow? How to do that?

[omry]

The error says: In 'sub_conf/config-default': Could not find 'sub_conf/sub_conf/db/mysql'

It means that in teh file sub_conf/config-default you have sub_conf/db/mysql in your defaults list. However, the defaults list entries are relative. In that file, you should have db: mysql. From the command line, you should have sub_conf/db=mysql.

Please read the defaults list carefully.

Specific section:

CONFIG_GROUP : A path to a set of configs. The path is relative to the containing config. It can be made absolute by prefixing it with a /. The path separator is / regardless of the operating system.

[arisliang]

Thanks for the pointer, it works.

config-defaults.yaml:

defaults:
  - /common: config-common.yaml
  - db: mysql.yaml

command line: python hydra_app.py sub_conf/db=postgresql

[RemyLau]

I have a similar but not exactly the same question. I will post it here in case anyone is looking for the answer (hi, future me).

Suppose I have the configs structured as follow:

├── config
│   ├── base_config.yaml
│   └── model
│       ├── model1.yaml
│       ├── model2.yaml
│       ├── model3.yaml
│       └── v2
│           ├── model1.yaml
│           ├── model2.yaml
│           └── model3.yaml
└── main.py

Usually I would just run:

python main.py model=model1

Now suppose I want to run v2's model1 (I could have make it v2_model1.yaml and call it a day, but this is just for organizational sake), and similarly v3, v4, etc., it is possible to do:

python main.py model=v2/model1

Amazing!

(hydra-core 1.3.2)

[odelalleau]

Now suppose I want to run v2's model1 (I could have make it v2_model1.yaml and call it a day, but this is just for organizational sake), and similarly v3, v4, etc., it is possible to do:

python main.py model=v2/model1

Interesting! I'm not sure this is even tested in Hydra, may be worth making sure it is to avoid breaking it in the future...

[rhajou]

Hello @RemyLau , @omry

I have a specific problem regarding this.

in my default config I have something similar to this:

├── config │ ├── base_config.yaml

│ └── model │ ├── model1.yaml │ ├── model2.yaml │ ├── model3.yaml │ └── dataset │ ├── dataset1.yaml │ ├── dataset2.yaml │ ├── dataset3.yaml │ └── v2 │ ├── dataset1.yaml │ ├── dataset2.yaml │ └── dataset3.yaml └── main.py

defaults:
  - _self_
  - model: model1
  - dataset@annotations: dataset1
  - dataset@predictions: v2/dataset1

I am trying to override the the predictions (to v2/dataset2), but it converts the value to string. how to solve this?

with hydra.initialize(version_base="1.3", config_path="..."):
    cfgHydra = hydra.compose(
        config_name="base_config",
        overrides=[
            "dataset@annotations=dataset2", # this works well
            "dataset@predictions=v2/dataset2" , # this does not work and reads v2/dataset2 as a string not the config file
        ],
    )

How to solve this? Thanks!