[Feature Request] Tuple Sweeper OR Top-level dictionary overrides

[goens]

🚀 Feature Request

For multiple keys, the choice/range sweepers will iterate over the (Cartesian) product of the lists. It is useful to sometimes be able to iterate over distinct pairs. This works with the dictionary override syntax, but only over a single key. I would suggest to either add a tuple sweeper (e.g. key1,key2=(value1_key1,value1_key2),(value2_key1,value2_key2)) or add top-level dictionary override (e.g. job={key1:value1_key1,key2:value1_key2},{key1:value2_key1,key2:value2_key2}).

The syntax examples are just to make the point clear. This request is not about the concrete syntax but rather about having something like this in the grammar (with a syntax that you consider proper).

Motivation

It is common that some pairs of configurations that do not make sense together. Suppose you have two keys in the configuration, key1, key2, and two options for each, value1_key1, value2_key1 and value1_key2,value2_key2. However, due to some interaction in the logic of your program, it does not make sense to call the program with the configuration key1=value1_key1,key2=value2_key2. If you call hydra with the override string key1=value1_key1,value2_key1 key2=value1_key2,value2_key2 then all four combinations of the values will be included in the sweep, including the one that makes no sense (which in the worst case causes a runtime error and crashes the whole sweep).

Pitch

Describe the solution you'd like

I think it would make most sense to have the option to submit pairs (or rather, arbitrary tuples) of key-value pairs to construct a specific subset of the Cartesian product of lists as intended. Dictionary override syntax already allows this, but not at the top level. You would have to put all options in a single key to use dictionary override syntax, which kind of makes everything unnecessarily verbose.

Even nicer would be other combinators like in itertools (i.e. a zip instead of prod), but that would be unnecessarily complicated and perhaps only marginally more useful.

Describe alternatives you've considered

The easiest alternative is just defining a top-level configuration key (e.g. top) and using dictionary override syntax. I think it's a bad alternative because then every configuration has to be changed to that, in the example: top = {key1:value1_key1,key2:value1_key2},{key1:value2_key1,key2:value2_key2} works, but then to just set the first option you need to call it as top.key1=value1_key1 top.key2=value1_key2.

The other two alternatives are the top-level dictionary overrides and the tuple sweeper combinator as mentioned. I tend to think both are better than just one of them, but the top-level dictionary override one is more general (but the syntax is less obvious).

Are you willing to open a pull request? (See CONTRIBUTING) Yes, I am happy to implement this and open a pull request if you think it is a welcome addition.

Additional context

Add any other context or screenshots about the feature request here.

[jieru-hu]

Thanks for the feature request! Right now to get around this, you can group the parameter's together and override that way. This could also be a standalone tuple sweeper plugin, since plugins have more control over the config syntax and on how to parse the overrides.

cc @omry

[omry]

Hi @goens, Thanks for your feature request.

The problem with both of your proposals is that they are very different than anything else and will have a significant surface area For example, I don't think optimizing sweepers like Optuna, Ax or Nevergrad will deal with this in the form you are suggesting.

What you are asking for is already possible if you make a small change to your config and move the swept parameters into a specific path:

$ python my_app.py -m '+top={a:10,b:20},{a:100,b:200}'
[2021-01-04 18:13:31,694][HYDRA] Launching 2 jobs locally
[2021-01-04 18:13:31,694][HYDRA]        #0 : +top={a:10,b:20}
top:
  a: 10
  b: 20

[2021-01-04 18:13:31,837][HYDRA]        #1 : +top={a:100,b:200}
top:
  a: 100
  b: 200

This can be combined with an interpolation at the root to achieve a similar final result:

a: ${top.a}
b: ${top.b}

Overall, I am not convinced that supporting tuple sweeps are worth the added complexity. I will keep this open for a while to see what kind of interest this generates for other people before making my final decision.

[goens]

Hi @omry, thanks for the feedback! I'm aware of the alternative with the shared directory, that's why I called the first alternative a top-level dictionary override (because at other levels it works). It's just not very 'idiomatic' in a sense, that's why I proposed the tuple sweeper.

I'd honestly be surprised if this is not/has not been an issue for others, but I guess it depends a lot on the use cases. I'd certainly understand that the added complexity might not be worth it if people don't need this. I'd be happy to try and write a tuple sweeper myself, as a plugin as @jieru-hu suggests, if that changes anything.

[omry]

Hi @goens, It's certainly not the most intuitive solution.

Most people leverage the hierarchical nature of the config and do not have many top level elements so this problem is less of an issue for them. You don't need to ask for permission to create a plugin. You can create one and host it in your own repo. there is no requirement that all plugins are a part of this repo.

I suspect that a proper solution would have to go deeper than a sweeper plugin though, for example - a plugin cannot extend the command line grammar in Hydra so you will need to parse manually or create your own grammar.

As I said, I am keeping this open to get feedback from other interested people. Hydra 1.1 already has some major changes coming and I am certainly not shopping for additional low level changes beyond what is already in the milestone (If anything, I will probably reduce the scope of the milestone). Even we decide to go on with it it will take a while before it's available to users.

[timothylimyl]

Recently, I had a need to run multirun with constrained combination instead of running all combinations. I believe that tuple sweep will be very useful as there are many use cases for separating the combination, e.g, you want to test multiple optimizers while renaming the log file accordingly:

python3 training.py -m optimizers.option= SGD,Adam, AdamW  logger.name=sgd,adam,adamw

Running all combinations does not make any sense here.

[praksharma]

Me too, I think this might be a useful features.

[praksharma]

I can't reproduce the same thing with a config file.

network:
    inputs: 2
    outputs: 1
    layer_size: 60
    nr_layers: 8

optimiser:
    lr: 1e-3
    scheduler: False
    iter: 5

hydra:
    mode: MULTIRUN
    sweeper:
        params:
          +n: 5,10,15
          +a_lims: {'a_lower' : 0.5, 'a_upper' : 1.1, 'a' : 0.91},{'a_lower' : 0.7, 'a_upper' : 1.15, 'a' : 1.05}, {'a_lower' : 0.9, 'a_upper' : 1.15, 'a' : 1.09}
          +pde_coeff: 0.1, 1.0

The python file:

import hydra
from omegaconf import DictConfig

@hydra.main(version_base="1.2", config_path="saved_data", config_name="conf")
def main(cfg: DictConfig) -> None:
    print(cfg.n)

if __name__ == "__main__":
   main()

The error:

[s.1915438@sl1(sunbird) hydra test dir]$ python 1.single_notebook.py -m
Traceback (most recent call last):
  File "/scratch/s.1915438/env/modulus/lib/python3.9/site-packages/hydra/_internal/utils.py", line 213, in run_and_report
    return func()
  File "/scratch/s.1915438/env/modulus/lib/python3.9/site-packages/hydra/_internal/utils.py", line 461, in <lambda>
    lambda: hydra.multirun(
  File "/scratch/s.1915438/env/modulus/lib/python3.9/site-packages/hydra/_internal/hydra.py", line 143, in multirun
    cfg = self.compose_config(
  File "/scratch/s.1915438/env/modulus/lib/python3.9/site-packages/hydra/_internal/hydra.py", line 594, in compose_config
    cfg = self.config_loader.load_configuration(
  File "/scratch/s.1915438/env/modulus/lib/python3.9/site-packages/hydra/_internal/config_loader_impl.py", line 141, in load_configuration
    return self._load_configuration_impl(
  File "/scratch/s.1915438/env/modulus/lib/python3.9/site-packages/hydra/_internal/config_loader_impl.py", line 235, in _load_configuration_impl
    self._process_config_searchpath(config_name, parsed_overrides, caching_repo)
  File "/scratch/s.1915438/env/modulus/lib/python3.9/site-packages/hydra/_internal/config_loader_impl.py", line 158, in _process_config_searchpath
    loaded = repo.load_config(config_path=config_name)
  File "/scratch/s.1915438/env/modulus/lib/python3.9/site-packages/hydra/_internal/config_repository.py", line 349, in load_config
    ret = self.delegate.load_config(config_path=config_path)
  File "/scratch/s.1915438/env/modulus/lib/python3.9/site-packages/hydra/_internal/config_repository.py", line 92, in load_config
    ret = source.load_config(config_path=config_path)
  File "/scratch/s.1915438/env/modulus/lib/python3.9/site-packages/hydra/_internal/core_plugins/file_config_source.py", line 31, in load_config
    cfg = OmegaConf.load(f)
  File "/scratch/s.1915438/env/modulus/lib/python3.9/site-packages/omegaconf/omegaconf.py", line 190, in load
    obj = yaml.load(file_, Loader=get_yaml_loader())
  File "/scratch/s.1915438/env/modulus/lib/python3.9/site-packages/yaml/__init__.py", line 81, in load
    return loader.get_single_data()
  File "/scratch/s.1915438/env/modulus/lib/python3.9/site-packages/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()
  File "/scratch/s.1915438/env/modulus/lib/python3.9/site-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
  File "/scratch/s.1915438/env/modulus/lib/python3.9/site-packages/yaml/composer.py", line 55, in compose_document
    node = self.compose_node(None, None)
  File "/scratch/s.1915438/env/modulus/lib/python3.9/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/scratch/s.1915438/env/modulus/lib/python3.9/site-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/scratch/s.1915438/env/modulus/lib/python3.9/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/scratch/s.1915438/env/modulus/lib/python3.9/site-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/scratch/s.1915438/env/modulus/lib/python3.9/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/scratch/s.1915438/env/modulus/lib/python3.9/site-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/scratch/s.1915438/env/modulus/lib/python3.9/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/scratch/s.1915438/env/modulus/lib/python3.9/site-packages/yaml/composer.py", line 127, in compose_mapping_node
    while not self.check_event(MappingEndEvent):
  File "/scratch/s.1915438/env/modulus/lib/python3.9/site-packages/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
  File "/scratch/s.1915438/env/modulus/lib/python3.9/site-packages/yaml/parser.py", line 438, in parse_block_mapping_key
    raise ParserError("while parsing a block mapping", self.marks[-1],
yaml.parser.ParserError: while parsing a block mapping
  in "/scratch/s.1915438/2. inverse+forward/7. working/hydra test dir/saved_data/conf.yaml", line 16, column 11
expected <block end>, but found ','
  in "/scratch/s.1915438/2. inverse+forward/7. working/hydra test dir/saved_data/conf.yaml", line 17, column 66

[Jasha10]

@praksharma the traceback shows a yaml.parser.ParserError. The config file you gave is not valid yaml. You need to replace this:

          +a_lims: {'a_lower' : 0.5, 'a_upper' : 1.1, 'a' : 0.91},{'a_lower' : 0.7, 'a_upper' : 1.15, 'a' : 1.05}, {'a_lower' : 0.9, 'a_upper' : 1.15, 'a' : 1.09}

Replace it with this:

          +a_lims: "{a_lower: 0.5, a_upper: 1.1, a: 0.91},{a_lower: 0.7, a_upper: 1.15, a: 1.05},{a_lower: 0.9, a_upper: 1.15, a: 1.09}"

Ref: Hydra's Override Grammar

[praksharma]

Lovely thank you.

[nuomizai]

I met the same request. But I can't get the right feature. In my settings, the file tree is as follows

cfgs/
----task/
--------cheetah_run.yaml
--------finger_spin.yaml
----config.yaml

In cheetah_run.yaml, the content is as follows:

defaults:
  - _self_

task_name: cheetah_run
action_repeat: -1

In finger_spin.yaml, the content is as follows:

defaults:
  - _self_

task_name: finger_spin
action_repeat: -1

I want to run the task cheetah_run and finger_spin sequentially without changing the default yaml files. So I wrote the command as follows:

python src/train.py --multirun "+task={task_name: cheetah_run, action_repeat:4}, {task_name: finger_spin, action_repeat:2}" eval_mode=video_hard agent=drq seed=0 device=0

However, it doesn't work. I print the task_name and action_repeat on the terminal, but it only showed the default values.

My config YAML file is as follows:

defaults:
  - _self_
  - agent@_global_: svea
  - task@_global_: cartpole_swingup

[arzaatri]

Hi I'm looking for advice on my use case. I want to sweep over several different model architectures each of which may have its own set of hyperparameters. Here's a toy example, to avoid the particulars:

model:
    __target__: model1Class
    num_layers: 4
    hidden_dim: 32

model:
    __target__: model2Class
    num_layers: 8
    input_dim: (256, 256)

I could bundle all of these into a single param that is a dictionary, but then I can't take advantage of hydra.utils.instantiate. I can imagine workarounds using python functions, but I'm wondering if there's any way I can do this in a more hydra-centric way. For example if I could leave "model" out of my main config file, and instead sweep over multiple other config files, each of which defines the full hierarchy of the "model" config parameter?

[odelalleau]

I met the same request. But I can't get the right feature. In my settings, the file tree is as follows (...) However, it doesn't work. I print the task_name and action_repeat on the terminal, but it only showed the default values.

What happens is when you do this, you are adding a new task node with the values you provide on the command line, but your task_name and action_repeat options are actually at the root level.

One potential way to achieve what you're trying to do is the following:

# Your Python script.
OmegaConf.register_new_resolver("map", lambda d, k: d[k])

# config.yaml
defaults:
  - _self_
  - agent@_global_: svea
  - task@_global_: cartpole_swingup

chosen_task: ${hydra:runtime.choices.task@_global_}

Command line:

python src/train.py --multirun 'action_repeat="${map:{cheetah_run: 4, finger_spin: 2}, ${chosen_task}}"' task@_global_=finger_spin,cheetah_run eval_mode=video_hard agent=drq seed=0 device=0

[odelalleau]

Hi I'm looking for advice on my use case. I want to sweep over several different model architectures each of which may have its own set of hyperparameters. (...) For example if I could leave "model" out of my main config file, and instead sweep over multiple other config files, each of which defines the full hierarchy of the "model" config parameter?

If you just want to sweep over mutliple config files associated to different models, this is easily achieved by making model a config group, so you can use model=model_1,model_2,model_3 on the command line.

[PhilippDahlinger]

I created a plugin, which exactly implements the desired tuple sweeping. It is publicly available here:

https://github.com/ALRhub/hydra_list_sweeper

You can set a list_params and the plugin sweeps over the zipped lists here. Also, you can combine it with a grid search specified by grid_params or command line overrides. If you encounter bugs, or request features, please let me know!

If the Hydra team is interested in adding this to hydra-core, I'd happily contribute.

[zuble]

Hi I'm looking for advice on my use case. I want to sweep over several different model architectures each of which may have its own set of hyperparameters. (...) For example if I could leave "model" out of my main config file, and instead sweep over multiple other config files, each of which defines the full hierarchy of the "model" config parameter?

If you just want to sweep over mutliple config files associated to different models, this is easily achieved by making model a config group, so you can use model=model_1,model_2,model_3 on the command line.

so i am tryng to achieve something similiar and basic

i have this config struct:

cfg
├── data
├── exp
├── hydra
├── model
     m1.yaml
     m2.yaml
├── path
 orig.yaml

in orig.yaml:

defaults:
  - _self_
  - data: ds1
  - model: m1
  - path: dflt
  - hydra: dflt
  - exp: null

then i setup exp/model.yaml as :

# @package _global_

hydra:
  mode: MULTIRUN
  sweeper:
    params:
      model: m1, m2

when running with python main.py exp=model and python main.py model=m1,m2 i get this error:

  File "...hydra/_internal/utils.py", line 466, in <lambda>
    lambda: hydra.multirun(
            ^^^^^^^^^^^^^^^
  File "...hydra/_internal/hydra.py", line 162, in multirun
    ret = sweeper.sweep(arguments=task_overrides)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../hydra/_internal/core_plugins/basic_sweeper.py", line 162, in sweep
    sweep_dir = Path(self.config.hydra.sweep.dir)
    ......
    raise ValueError("HydraConfig was not set")
omegaconf.errors.InterpolationResolutionError: ValueError raised while resolving interpolation: HydraConfig was not set
    full_key: hydra.sweep.dir
    reference_type=SweepDir
    object_type=SweepDir

-c hydra gives this:

hydra:
  run:
    dir: ${hydra:runtime.cwd}/log/runs/${now:%d-%m}_${now:%H-%M-%S}
  sweep:
    dir: ${hydra:runtime.cwd}/log/multiruns/${now:%d-%m}_${now:%H-%M-%S}
    subdir: ${hydra.job.num}
  launcher:
    _target_: hydra._internal.core_plugins.basic_launcher.BasicLauncher
  sweeper:
    _target_: hydra._internal.core_plugins.basic_sweeper.BasicSweeper
    max_batch_size: null
    params:
      model: m1,m2

any idea of what i am doing wrong here ? many thanks in advance