[Feature Request] Concatenating parameter sweeps

[Sola85]

🚀 Feature Request: Concatenating parameter sweeps

Issues #1918 and #1258 have already proposed adding "tuple sweeps" where a fixed list of parameter combinations is used for a sweep instead of a cartesian product of all parameter lists. These issues are still open because work arounds exist, like defining a top-level dict of all desired parameter combination and using interpolation.

However, to my understanding, this means that you have to manually define all parameter combinations you want so sweep over. This means that there are three options:

• Either using the default cartesian product to sweep over all parameter combinations
• Use the top-level dict approach for defining all sweep parameter combinations manually
• Start hydra multiple times with different sweep configurations (which sort of defeats the point of a tool like hydra)

I would like to propose the following new option, which would solve the two issues mentioned above, while at the same time being more flexible than the proposed workarounds: E.g.:

hydra:
  mode: MULTIRUN
  sweeper:
    params1:
      variable1: 1, 5, 8, 19, 32
      variable2: 0, 1, 800, 1000, 3000
      variable3: true
    params2:
      variable1: 0
      variable2: -1, -2, -3
      variable3: false

Here we have multiple params fields, that are treated independently from each other. Within one params field, everything behaves as usual with the cartesian product. The above example would generate 5*5+3 jobs. You could also call this "sweeping over the sweeper params".

This has a few advantages:

• It allows nonsensical parameter combinations to be excluded (as requested by Issues #1918 and #1258), e.g. here variable1=0 and variable2=1 might not make sense.
• We can still use cartesian products to generate a lot of parameter combinations without having to manually define them
• This is conceptually very easy and behaves similar to starting hydra multiple times with different configs. Just with the advantage of having it all in a single file.
• As far as I can tell this would even be more general than this fairly new list-sweeper plugin mentioned here (At least I dont see how one would generate the above jobs with list-sweeper, but the list-sweeper example can easily be recreated with the above approach).

In case I missed something and a solution to this already exists, I'd be happy to close this issue.

[gil2rok]

Great idea! +1

[libokj]

It does sound like a pretty general solution to all those paired sweep problems. It's been a pain to create extra sweep configs to accommodate those sweeps for me, where I need to pair a pytorch lightning model checkpoint to its model config file. For better parsing and look, maybe using a list of dictionaries should be considered? Also, correct me if I'm wrong - I thought spaces were not allowed in a sweep variable, that it must be purely comma-seperated? If this solution becomes widely accepted, I'd be more than happy to help with implementation.

hydra:
  mode: MULTIRUN
  sweeper:
    params:
      - variable1: 1,5,8,19,32
        variable2: 0,1,800,1000,3000
        variable3: true
      - variable1: 0
        variable2: -1,-2,-3
        variable3: false

[stefangstark]

this would be great, and it would be nice if there is a way to include variables that are shared across all concatenated sweeps

[florian6973]

Any update by any chance?