Array unpacking is not supported

[timruhkopf]

Hey Fredierk, As discussed earlier the concept of the issue:

I am currently trying to optimize the neurons of a (in terms of layers) fixed sized mlp, which I wrote in form of a list:

mlp(layerneurons=[neurons1, neurons2, ....]):

consequently, I would want to write a config file model/mlp.yaml file containing the model's layerneurons as such:

layerneurons:
  - 100
  - 200

and subsequently define the sweep experiment

# @package _global_
defaults:
  - override /hydra/sweeper: SMAC  # be carefull to make it /hydra
  - override /hydra/launcher: submitit_smac_slurm

# smac parameter defaults:
max_epochs: 500

# define smac mf
hydra:
  launcher:
    partition: cpu_normal
  sweeper:
    budget_variable: ${max_epochs}
    seed: ${seed}
    n_jobs: 4
    scenario:
      run_obj: quality
      deterministic: true
      #      wallclock_limit: 100
      runcount_limit: 50
      cutoff: 30 # runtime limit for target algorithm
      memory_limit: 3072 # adapt this to reasonable value for your hardware
    intensifier:
      initial_budget: 5
      max_budget: ${max_epochs}
      eta: 3
    search_space:
        hyperparameters:
          model.layerneurons.0:
            type: uniform_int
            lower: 100
            upper: 400
            default: ${model.layerneurons.0}
          model.layerneurons.1:
            type: uniform_int
            lower: 100
            upper: 400
            default: ${model.layerneurons.1}

  run:
    dir: ./tmp/${now:%Y-%m-%d}/${now:%H-%M-%S}
  sweep:
    dir: ./tmp/${now:%Y-%m-%d}/${now:%H-%M-%S}

but unfortunately, the array unpacking is not supported.

/home/ruhkopf/PycharmProjects/AlgoSelectionMF/mf_gravitas/main.py +experiment=search_spaces/ssp_rank_ensemble --multirun
Connected to pydev debugger (build 213.6777.50)
[2022-05-04 16:41:57,406][HYDRA] Sweep overrides: +experiment=search_spaces/ssp_rank_ensemble
[2022-05-04 16:41:57,417][HYDRA] Output to ./tmp/2022-05-04/16-41-57
[2022-05-04 16:41:57,418][HYDRA] Optimizing a deterministic scenario for quality without a tuner timeout - will make SMAC deterministic and only evaluate one configuration per iteration!
[2022-05-04 16:41:57,499][HYDRA] Running initial design for 12 configurations
[2022-05-04 16:41:57,500][HYDRA] <class 'smac.facade.smac_mf_facade.SMAC4MF'>
[2022-05-04 16:41:57,500][HYDRA] <class 'smac.facade.smac_mf_facade.SMAC4MF'>
Traceback (most recent call last):
  File "/home/ruhkopf/miniconda3/envs/gravitas/lib/python3.9/site-packages/hydra/_internal/utils.py", line 385, in _run_hydra
    run_and_report(
  File "/home/ruhkopf/miniconda3/envs/gravitas/lib/python3.9/site-packages/hydra/_internal/utils.py", line 214, in run_and_report
    raise ex
  File "/home/ruhkopf/miniconda3/envs/gravitas/lib/python3.9/site-packages/hydra/_internal/utils.py", line 211, in run_and_report
    return func()
  File "/home/ruhkopf/miniconda3/envs/gravitas/lib/python3.9/site-packages/hydra/_internal/utils.py", line 386, in <lambda>
    lambda: hydra.multirun(
  File "/home/ruhkopf/miniconda3/envs/gravitas/lib/python3.9/site-packages/hydra/_internal/hydra.py", line 140, in multirun
    ret = sweeper.sweep(arguments=task_overrides)
  File "/home/ruhkopf/PycharmProjects/hydra-smac-sweeper/hydra_plugins/hydra_smac_sweeper/smac_sweeper.py", line 32, in sweep
    return self.sweeper.sweep(arguments)
  File "/home/ruhkopf/PycharmProjects/hydra-smac-sweeper/hydra_plugins/hydra_smac_sweeper/smac_sweeper_backend.py", line 95, in sweep
    incumbent = smac.optimize()
  File "/home/ruhkopf/miniconda3/envs/gravitas/lib/python3.9/site-packages/smac/facade/smac_ac_facade.py", line 712, in optimize
    incumbent = self.solver.run()
  File "/home/ruhkopf/miniconda3/envs/gravitas/lib/python3.9/site-packages/smac/optimizer/smbo.py", line 272, in run
    self.tae_runner.submit_run(run_info=run_info)
  File "/home/ruhkopf/PycharmProjects/hydra-smac-sweeper/hydra_plugins/hydra_smac_sweeper/submitit_runner.py", line 89, in submit_run
    overrides = self._diff_overrides(run_info)
  File "/home/ruhkopf/PycharmProjects/hydra-smac-sweeper/hydra_plugins/hydra_smac_sweeper/submitit_runner.py", line 105, in _diff_overrides
    diff_overrides = [tuple(f"{key}={val1}" for key, val1 in run_info_cfg_flat.items(
  File "/home/ruhkopf/PycharmProjects/hydra-smac-sweeper/hydra_plugins/hydra_smac_sweeper/submitit_runner.py", line 106, in <genexpr>
    ) if val1 != self.base_cfg_flat[key])]
KeyError: 'model.layerneurons.0'

Process finished with exit code 1

Expected behavior Regardless of where the list is nested, I would expect, that I can index and override it. For now, it would suffice to have the last index to be unpacked properly

[timruhkopf]

While not neat in the sence of supporting all recursive cases, the following method at least supports the cases, where the final node is a list

    def _diff_overrides(self, run_info: RunInfo):
        run_info_cfg_flat = flatten_dict(run_info.config.get_dictionary())

        diff_overrides = []
        for key, val in run_info_cfg_flat.items():
            if key in self.base_cfg_flat.keys():
                if val != self.base_cfg_flat[key]:
                    diff_overrides.append(f"{key}={val}")

            else:
                # list valued key?
                components = [i if not i.isnumeric() else int(i) for i in key.split('.')]
                key_intermediate = '.'.join(components[:-1])
                index = components[-1]
                val1 = self.base_cfg_flat[key_intermediate][index]
                if val != val1:
                    diff_overrides.append(f"{key}={val1}")

        # diff_overrides = [
        #     tuple(f"{key}={val1}"
        #           for key, val1 in run_info_cfg_flat.items()
        #           if key in self.base_cfg_flat.keys() and val1 != self.base_cfg_flat[key])]

        return [tuple(diff_overrides)]