sweep parameters in a paired manner

[hukz18]

🚀 Feature Request

Support sweep parameters in pairs(triplets, etc) rather than the full combination of them

Motivation

Is your feature request related to a problem? Please describe.

Sometimes several sets of parameters always come in pairs, like (a=a₁, b=b₁), (a=a₂, b=b₂), etc. And we know that (a=a₁, b=b₂) or (a=a₂, b=b₁) would not be the case. The current sweep syntax make it impossible to sweep paired parameters

Pitch

Describe the solution you'd like

use brackets to wrap paired parameters, like python sweep.py -m (a, b)=(a1, b1), (a2, b2) c=c1, c2, would make a combination of (a₁, b₁, c₁), (a₁, b₁, c₂), (a₂, b₂, c₁), and (a₂, b₂, c₂),

Describe alternatives you've considered

I've been using a shell tool parallel, where you could add a --link argument to sweep all parameters in pairs, however (as far as I know) you could not pair some parameters and combine others

[frederikschubert]

You can use https://hydra.cc/docs/patterns/configuring_experiments/ for that. Just define the paired params in a subdirectory of your configuration, e.g. called param_group and create a file for each paired parameter group. Then you can sweep over them like +param_group=1,2,3 or +param_group=glob(*).

[Jasha10]

I've put this on the wishlist. I think it would be difficult to implement at the moment.

[Jasha10]

You can use https://hydra.cc/docs/patterns/configuring_experiments/ for that. Just define the paired params in a subdirectory of your configuration, e.g. called param_group and create a file for each paired parameter group. Then you can sweep over them like +param_group=1,2,3 or +param_group=glob(*).

Thanks @frederikschubert. Let me give a concrete example based on this comment:

$ python main.py -m +a_b=a1_b1,a2_b2 +c=c1,c2
{'a': {'foo': 'a1'}, 'b': {'bar': 'b1'}, 'c': {'baz': 'c1'}}
[2021-12-19 18:48:47,236][HYDRA]        #1 : +a_b=a1_b1 +c=c2
{'a': {'foo': 'a1'}, 'b': {'bar': 'b1'}, 'c': {'baz': 'c2'}}
[2021-12-19 18:48:47,327][HYDRA]        #2 : +a_b=a2_b2 +c=c1
{'a': {'foo': 'a2'}, 'b': {'bar': 'b2'}, 'c': {'baz': 'c1'}}
[2021-12-19 18:48:47,420][HYDRA]        #3 : +a_b=a2_b2 +c=c2
{'a': {'foo': 'a2'}, 'b': {'bar': 'b2'}, 'c': {'baz': 'c2'}}

Here are the files:

$ tree
.
├── conf
│   ├── a
│   │   ├── a1.yaml
│   │   └── a2.yaml
│   ├── a_b
│   │   ├── a1_b1.yaml
│   │   └── a2_b2.yaml
│   ├── b
│   │   ├── b1.yaml
│   │   └── b2.yaml
│   ├── c
│   │   ├── c1.yaml
│   │   └── c2.yaml
│   └── config.yaml
└── main.py

# main.py
import hydra

@hydra.main(config_path="conf", config_name="config")
def app(cfg) -> None:
    print(cfg)

app()

# conf/config.yaml (empty)

# conf/a/a2.yaml
foo: a2

# conf/a/a1.yaml
foo: a1

# @package _global_
# conf/a_b/a1_b1.yaml
defaults:
  - /a: a1
  - /b: b1

# @package _global_
# conf/a_b/a2_b2.yaml
defaults:
  - /a: a2
  - /b: b2

# conf/c/c2.yaml
baz: c2

# conf/c/c1.yaml
baz: c1

# conf/b/b2.yaml
bar: b2

# conf/b/b1.yaml
bar: b1

[hukz18]

not so convenient, but thanks for the detailed reply

[jieru-hu]

this is similar to #1258 maybe also checkout omry's suggestion there https://github.com/facebookresearch/hydra/issues/1258#issuecomment-754341537

[timothylimyl]

seconded for this feature. My current workaround is to just put the individual terminal commands in a bash script, e.g, for a optimiser sweep test for deep learning:

python3 ../training.py -m optimizers.option=SGD   logger.experiment_name=optimizer_test logger.experiment_version_tag=sgd
python3 ../training.py -m optimizers.option=Adam  logger.experiment_name=optimizer_test logger.experiment_version_tag=adam 
python3 ../training.py -m optimizers.option=AdamW logger.experiment_name=optimizer_test logger.experiment_version_tag=AdamW
python3 ../training.py -m optimizers.option=Nadam logger.experiment_name=optimizer_test logger.experiment_version_tag=Nadam
python3 ../training.py -m optimizers.option=AdamP  logger.experiment_name=optimizer_test logger.experiment_version_tag=AdamP 

The ideal solution would be to pair the optimizers.option and logger.experiment_version_tag while keeping a single same change logger.experiment_name . Working around this via changing the configuration file is not desirable as it changes the overall functionality of the configs, having the ability to do some custom sweeping will be great.

[odelalleau]

Note that you can use interpolations to achieve this:

+sweep='{a: 0, b: 0},{a: 1, b: 1}' a='${sweep.a}' b='${sweep.b}'

Edit: just realized this is actually the same as Omry's comment linked above...

[alvitawa]

I would like this feature as well. Maybe it could be simply a custom sweeper plugin that pairs everything?

[swertz]

If it helps you, I ended up creating a minimalistic custom sweeper pluging to iterate over multiple parameters in parallel: https://github.com/b12consulting/hydra_paired_sweeper (soon on PyPI...).

It's not as general as what was proposed in #2851 , but we could also easily implement the proposed syntax of #2851 in there...

EDIT: I only now realize there exists a very similar implementation here: https://github.com/ALRhub/hydra_list_sweeper ...