[BUG] Hypersweeper is throwing an error if argument is not in the config

becktepe commented 6 months ago

HyperSweeper version: 0.1.1
Python version: 3.10
Operating System: MacOS/Linux

Description

For an experiment, I want to override certain values in my configuration using a config file specified as an argument. However, the config only overrides existing attributes, it is not included in the config. This seems to crash the hypersweeper when building the final config once all runs are done.

Steps/Code to Reproduce

In addition to some configurations in configs/environment and configs/algorithm, I am using this file (configs/experiments/dqn_cartpole.yaml):

# @package _global_
defaults:
  - override /environment: cartpole  
  - override /algorithm: dqn   

np_config:
  activation: tanh
  hidden_size: 64

I am running the hyper sweeper: python runscripts/run.py -m experiments=dqn_cartpole

Expected Results

It should just run successfully :-)

Actual Results

After all runs are completed, the hypersweeper throws this error:

Traceback (most recent call last):
  File "/bigwork/nhwpbecj/nhwpbecj/.conda/envs/arlb/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/bigwork/nhwpbecj/nhwpbecj/.conda/envs/arlb/lib/python3.10/site-packages/hydra/_internal/utils.py", line 466, in <lambda>
    lambda: hydra.multirun(
  File "/bigwork/nhwpbecj/nhwpbecj/.conda/envs/arlb/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 162, in multirun
    ret = sweeper.sweep(arguments=task_overrides)
  File "/bigwork/nhwpbecj/git_projects/hypersweeper/hydra_plugins/hypersweeper/hypersweeper.py", line 37, in sweep
    return self.sweeper.sweep(arguments)
  File "/bigwork/nhwpbecj/git_projects/hypersweeper/hydra_plugins/hypersweeper/hypersweeper_backend.py", line 173, in sweep
    reduce(operator.getitem, key_parts[:-1], final_config)[key_parts[-1]] = v
  File "/bigwork/nhwpbecj/nhwpbecj/.conda/envs/arlb/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 310, in __setitem__
    self._format_and_raise(
  File "/bigwork/nhwpbecj/nhwpbecj/.conda/envs/arlb/lib/python3.10/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/bigwork/nhwpbecj/nhwpbecj/.conda/envs/arlb/lib/python3.10/site-packages/omegaconf/_utils.py", line 819, in format_and_raise
    _raise(ex, cause)
  File "/bigwork/nhwpbecj/nhwpbecj/.conda/envs/arlb/lib/python3.10/site-packages/omegaconf/_utils.py", line 797, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
  File "/bigwork/nhwpbecj/nhwpbecj/.conda/envs/arlb/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 308, in __setitem__
    self.__set_impl(key=key, value=value)
  File "/bigwork/nhwpbecj/nhwpbecj/.conda/envs/arlb/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 318, in __set_impl
    self._set_item_impl(key, value)
  File "/bigwork/nhwpbecj/nhwpbecj/.conda/envs/arlb/lib/python3.10/site-packages/omegaconf/basecontainer.py", line 549, in _set_item_impl
    self._validate_set(key, value)
  File "/bigwork/nhwpbecj/nhwpbecj/.conda/envs/arlb/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 180, in _validate_set
    target = self._get_node(key) if key is not None else self
  File "/bigwork/nhwpbecj/nhwpbecj/.conda/envs/arlb/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 475, in _get_node
    self._validate_get(key)
  File "/bigwork/nhwpbecj/nhwpbecj/.conda/envs/arlb/lib/python3.10/site-packages/omegaconf/dictconfig.py", line 164, in _validate_get
    self._format_and_raise(
  File "/bigwork/nhwpbecj/nhwpbecj/.conda/envs/arlb/lib/python3.10/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/bigwork/nhwpbecj/nhwpbecj/.conda/envs/arlb/lib/python3.10/site-packages/omegaconf/_utils.py", line 899, in format_and_raise
    _raise(ex, cause)
  File "/bigwork/nhwpbecj/nhwpbecj/.conda/envs/arlb/lib/python3.10/site-packages/omegaconf/_utils.py", line 797, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
omegaconf.errors.ConfigKeyError: Key 'experiments' is not in struct
    full_key: experiments
    object_type=dict

Additional Info

Did you try upgrading to the most current version? yes
Are you using a supported operating system (version)? yes
How did you install this package (e.g. GitHub, pip, etc.)? GitHub

TheEimer commented 5 months ago

This is expected behavior of hydra, you should use "+" in front of arguments that don't exist. I can't explain what's going on here though in the first place - is "experiments" a global config you have somewhere? I assume on the ARLBench experiments branch?

TheEimer commented 5 months ago

The issue here was that the config was previously resolved at some point and then if the sweeper tries to do this again to write the final config, the global experiments keyword is gone. I simply wrapped this case, the final config ends up looking correct in your case.

automl / hypersweeper