Open Entodi opened 1 year ago
Can you give me your hydra configuration ? Is Fold an hyperparameter ?
The fold is not a hyperparameter, but I want to set it using the SLURM_ARRAY_TASK.
Here is the modified yaml:
defaults:
- dataset: cifar10
- override hydra/sweeper: orion
hydra:
sweep:
dir: multirun/${now:%Y-%m-%d}/${now:%H-%M-%S}
subdir: ${hydra.sweeper.experiment.name}/${hydra.sweeper.experiment.uuid}/${hydra.job.id}
sweeper:
params:
a: "uniform(0, 1)"
b: "uniform(0, 1)"
experiment:
name: 'experiment'
version: '1'
algorithm:
type: random
config:
seed: 1
worker:
n_workers: -1
max_broken: 3
max_trials: 100
storage:
type: legacy
database:
type: pickleddb
host: 'database.pkl'
# Default values
a: 0
b: 0
The cifar10.yaml
in the directory dataset
:
object:
fold: 0
If I run:
HYDRA_FULL_ERROR=1 python main.py dataset=cifar10 dataset.object.fold=3 --multirun
I get:
<my_dir>/lib/python3.8/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'config': Defaults list is missing `_self_`. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/default_composition_order for more information
warnings.warn(msg, UserWarning)
[2023-06-07 13:59:37,540][HYDRA] Orion Optimizer {'type': 'random', 'config': {'seed': 1}}
[2023-06-07 13:59:37,541][HYDRA] with parametrization {'a': 'uniform(0, 1)', 'b': 'uniform(0, 1)'}
[2023-06-07 13:59:40,153][HYDRA] algorithms is deprecated and will be removed in v0.4.0. Use algorithm instead.
<my_dir>/lib/python3.8/site-packages/hydra/_internal/defaults_list.py:251: UserWarning:In 'config': Defaults list is missing `_self_`. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/default_composition_order for more information
[2023-06-07 14:00:02,842][HYDRA] Launching 128 jobs locally
{'dataset': {'object': {'fold': 0}}, 'a': 0.2878, 'b': 0.8581}
[2023-06-07 14:06:53,268][HYDRA] #1 : dataset=cifar10 b=0.72 a=0.4934
{'dataset': {'object': {'fold': 0}}, 'a': 0.4934, 'b': 0.72}
[2023-06-07 14:06:53,392][HYDRA] #2 : dataset=cifar10 b=0.5606 a=0.9892
{'dataset': {'object': {'fold': 0}}, 'a': 0.9892, 'b': 0.5606}
[2023-06-07 14:06:53,610][HYDRA] #3 : dataset=cifar10 b=0.9306 a=0.06306
If I run:
HYDRA_FULL_ERROR=1 python main.py dataset.object.fold=3 --multirun
I get the error:
<my_dir>/lib/python3.8/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'config': Defaults list is missing `_self_`. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/default_composition_order for more information
warnings.warn(msg, UserWarning)
[2023-06-07 14:00:40,087][HYDRA] Orion Optimizer {'type': 'random', 'config': {'seed': 1}}
[2023-06-07 14:00:40,088][HYDRA] with parametrization {'a': 'uniform(0, 1)', 'b': 'uniform(0, 1)'}
[2023-06-07 14:00:42,677][HYDRA] algorithms is deprecated and will be removed in v0.4.0. Use algorithm instead.
Traceback (most recent call last):
File "main.py", line 12, in <module>
main()
File "<my_dir>/lib/python3.8/site-packages/hydra/main.py", line 94, in decorated_main
_run_hydra(
File "<my_dir>/lib/python3.8/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
_run_app(
File "<my_dir>/lib/python3.8/site-packages/hydra/_internal/utils.py", line 465, in _run_app
run_and_report(
File "<my_dir>/lib/python3.8/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
raise ex
File "<my_dir>/lib/python3.8/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
return func()
File "<my_dir>/lib/python3.8/site-packages/hydra/_internal/utils.py", line 466, in <lambda>
lambda: hydra.multirun(
File "<my_dir>/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 162, in multirun
ret = sweeper.sweep(arguments=task_overrides)
File "<my_dir>/lib/python3.8/site-packages/hydra_plugins/hydra_orion_sweeper/orion_sweeper.py", line 79, in sweep
return self.sweeper.sweep(arguments)
File "<my_dir>/lib/python3.8/site-packages/hydra_plugins/hydra_orion_sweeper/implementation.py", line 500, in sweep
raise e
File "<my_dir>/lib/python3.8/site-packages/hydra_plugins/hydra_orion_sweeper/implementation.py", line 497, in sweep
self.optimize(self.client)
File "<my_dir>/lib/python3.8/site-packages/hydra_plugins/hydra_orion_sweeper/implementation.py", line 517, in optimize
returns = self.execute_trials(trials)
File "<my_dir>/lib/python3.8/site-packages/hydra_plugins/hydra_orion_sweeper/implementation.py", line 557, in execute_trials
self.validate_batch_is_legal(overrides)
File "<my_dir>/lib/python3.8/site-packages/hydra/plugins/sweeper.py", line 63, in validate_batch_is_legal
config_loader.load_sweep_config(
File "<my_dir>/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 319, in load_sweep_config
sweep_config = self.load_configuration(
File "<my_dir>/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 142, in load_configuration
return self._load_configuration_impl(
File "<my_dir>/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 244, in _load_configuration_impl
parsed_overrides, caching_repo = self._parse_overrides_and_create_caching_repo(
File "<my_dir>/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 228, in _parse_overrides_and_create_caching_repo
parsed_overrides = parser.parse_overrides(overrides=overrides)
File "<my_dir>/lib/python3.8/site-packages/hydra/core/override_parser/overrides_parser.py", line 96, in parse_overrides
raise OverrideParseException(
hydra.errors.OverrideParseException: no viable alternative at input '{'object''
See https://hydra.cc/docs/1.2/advanced/override_grammar/basic for details
My versions of hydra and orion packages:
(my_env) <my_dir>/orion_test$ pip show hydra-core
Name: hydra-core
Version: 1.3.2
Summary: A framework for elegantly configuring complex applications
Home-page: https://github.com/facebookresearch/hydra
Author: Omry Yadan
Author-email: omry@fb.com
License: MIT
Location: <my_dir>/anaconda3/envs/my_env/lib/python3.8/site-packages
Requires: antlr4-python3-runtime, importlib-resources, omegaconf, packaging
Required-by: hydra-orion-sweeper
(my_env) <my_dir>/orion_test$ pip show hydra-orion-sweeper
Name: hydra-orion-sweeper
Version: 1.6.2
Summary: Hydra Orion Sweeper plugin
Home-page: https://orion.readthedocs.io/
Author: Pierre Delaunay
Author-email: pierre.delaunay@mila.quebec
License:
Location: <my_dir>/anaconda3/envs/my_env/lib/python3.8/site-packages
Requires: hydra-core, omegaconf, orion, typing-extensions
Required-by:
Can you try with the latest version ? The issue should be resolved, feel free to reopen this if this is not the case
It worked for the command ✅:
HYDRA_FULL_ERROR=1 python main.py dataset=cifar10 dataset.object.fold=3 --multirun
But it does not work if ❌:
HYDRA_FULL_ERROR=1 python main.py dataset.object.fold=3 --multirun
The error:
(orion) <node>:<my_dir>/orion_test$ HYDRA_FULL_ERROR=1 python main.py dataset.object.fold=3 --multirun
<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'config': Defaults list is missing `_self_`. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/default_composition_order for more information
warnings.warn(msg, UserWarning)
[2023-06-07 15:17:18,714][HYDRA] Orion Optimizer {'type': 'random', 'config': {'seed': 1}}
[2023-06-07 15:17:18,714][HYDRA] with parametrization {'a': 'uniform(0, 1)', 'b': 'uniform(0, 1)'}
[2023-06-07 15:17:21,178][HYDRA] algorithms is deprecated and will be removed in v0.4.0. Use algorithm instead.
Traceback (most recent call last):
File "main.py", line 13, in <module>
main()
File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/main.py", line 94, in decorated_main
_run_hydra(
File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
_run_app(
File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/utils.py", line 465, in _run_app
run_and_report(
File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
raise ex
File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
return func()
File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/utils.py", line 466, in <lambda>
lambda: hydra.multirun(
File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 162, in multirun
ret = sweeper.sweep(arguments=task_overrides)
File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra_plugins/hydra_orion_sweeper/orion_sweeper.py", line 79, in sweep
return self.sweeper.sweep(arguments)
File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra_plugins/hydra_orion_sweeper/implementation.py", line 510, in sweep
raise e
File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra_plugins/hydra_orion_sweeper/implementation.py", line 507, in sweep
self.optimize(self.client)
File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra_plugins/hydra_orion_sweeper/implementation.py", line 527, in optimize
returns = self.execute_trials(trials)
File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra_plugins/hydra_orion_sweeper/implementation.py", line 567, in execute_trials
self.validate_batch_is_legal(overrides)
File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/plugins/sweeper.py", line 63, in validate_batch_is_legal
config_loader.load_sweep_config(
File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 319, in load_sweep_config
sweep_config = self.load_configuration(
File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 142, in load_configuration
return self._load_configuration_impl(
File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 244, in _load_configuration_impl
parsed_overrides, caching_repo = self._parse_overrides_and_create_caching_repo(
File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 228, in _parse_overrides_and_create_caching_repo
parsed_overrides = parser.parse_overrides(overrides=overrides)
File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/core/override_parser/overrides_parser.py", line 96, in parse_overrides
raise OverrideParseException(
hydra.errors.OverrideParseException: no viable alternative at input '{'object''
See https://hydra.cc/docs/1.2/advanced/override_grammar/basic for details
Here's an example of a script I run:
where FOLD is a bash variable. When it runs in a multi-run mode the sweep does not override the default value that was specified by the
script.dataset.object.fold=${FOLD}
. So, if my default fold was 0, but FOLD=4, it will just use the default value in the sweep.To make a small fix, I added a
fold
parameter in the config file and pointed it inscript.dataset.object.fold=${fold}
.I'm not sure if this is the expected behavior to override only the first-order parameters. Because it does override
script/dataset=dataset2
, for example. Could you clarify this?Thank you, Alex