Hierarchical values interrupt orion overrides of other variables

[jerpint]

It appears that once a hierarchical value is passed to orion, it stops updating the parameters listed thereafter in the config (ordering seems to matter). Here is a small experiment to show this with three configs that are identical except for the ordering under hydra.sweeper.params. All configs are taken from the example folder and only the ordering is different.

Example 1

(Identical to what is current under example/)

hydra:
  sweeper:
    # default parametrization of the search space
    params:
      optimizer:
        name: "choices(['Adam', 'SGD'])"
        lr: "uniform(0, 1)"
      dropout: "uniform(0, 1)"
      batch_size: "uniform(4, 16, discrete=True)"
...

[2022-07-26 11:29:17,230][HYDRA] Orion Optimizer {'type': 'random', 'config': {'seed': 1}}
[2022-07-26 11:29:17,230][HYDRA] with parametrization {'optimizer.lr': 'uniform(0, 1)', 'optimizer.name': "choices(['Adam', 'SGD'])"}
[2022-07-26 11:29:18,092][HYDRA] Launching 8 jobs locally
[2022-07-26 11:29:18,092][HYDRA]        #0 : optimizer.name=SGD optimizer.lr=0.2878

result: optimizer params get overridden, but dropout and batch_size don't get overridden:

Example 2

Different ordering, dropout come before hierarchical att so gets overriden, batch size comes after so it does not.

hydra:
  sweeper:
    # default parametrization of the search space
    params:
      dropout: "uniform(0, 1)"
      optimizer:
        name: "choices(['Adam', 'SGD'])"
        lr: "uniform(0, 1)"
      batch_size: "uniform(4, 16, discrete=True)"
...

2022-07-26 11:29:52,283][HYDRA] Orion Optimizer {'type': 'random', 'config': {'seed': 1}}
[2022-07-26 11:29:52,283][HYDRA] with parametrization {'dropout': 'uniform(0, 1)', 'optimizer.lr': 'uniform(0, 1)', 'optimizer.name': "choices(['Adam', 'SGD'])"}
[2022-07-26 11:29:53,153][HYDRA] Launching 8 jobs locally
[2022-07-26 11:29:53,153][HYDRA]        #0 : optimizer.name=Adam optimizer.lr=0.8581 dropout=0.2878

result: dropout and optimizer get overriden, but not batch_size:

Example 3

Here, the hieararchical att is last so everything seems to be overridden correctly.

hydra:
  sweeper:
    # default parametrization of the search space
    params:
      dropout: "uniform(0, 1)"
      batch_size: "uniform(4, 16, discrete=True)"
      optimizer:
        name: "choices(['Adam', 'SGD'])"
        lr: "uniform(0, 1)"
...

[2022-07-26 11:30:22,118][HYDRA] Orion Optimizer {'type': 'random', 'config': {'seed': 1}}
[2022-07-26 11:30:22,118][HYDRA] with parametrization {'batch_size': 'uniform(4, 16, discrete=True)', 'dropout': 'uniform(0, 1)', 'optimizer.lr': 'uniform(0, 1)', 'optimizer.name': "choices(['Adam', 'SGD'])"}
[2022-07-26 11:30:22,994][HYDRA] Launching 8 jobs locally
[2022-07-26 11:30:22,994][HYDRA]        #0 : optimizer.name=SGD optimizer.lr=0.4874 dropout=0.8581 batch_size=7

result: dropout, batch_size and optimizer get overriden correctly.

[jerpint]

I just wrote a fix

[jerpint]

nevermind, i see you already opened a PR for it, that was quick :)

[Delaunay]

I found the issue while I was adding tests :)