search

Eclectic-Sheep / sheeprl

Distributed Reinforcement Learning accelerated by Lightning Fabric
https://eclecticsheep.ai
Apache License 2.0
303 stars 31 forks source link

Agent evaluation hydra configuration error #259

Closed geranim0 closed 5 months ago

geranim0 commented 5 months ago

Hi,

Wanted to evaluate an agent with simple command

HYDRA_FULL_ERROR=1 python sheeprl_eval.py fabric.accelerator=gpu  checkpoint=logs/runs/dreamer_v3/MsPacmanNoFrameskip-v4/2024-03-24_20-28-49_dreamer_v3_MsPacmanNoFrameskip-v4_5/version_0/checkpoint/ckpt_600000_0.ckpt

However, it gets this error message:

Traceback (most recent call last):
  File "/home/sam/dev/sheeprl/sheeprl_eval.py", line 4, in <module>
    evaluation()
  File "/home/sam/dev/sheeprl/.venv/lib/python3.10/site-packages/hydra/main.py", line 90, in decorated_main
    _run_hydra(
  File "/home/sam/dev/sheeprl/.venv/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/home/sam/dev/sheeprl/.venv/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/home/sam/dev/sheeprl/.venv/lib/python3.10/site-packages/hydra/_internal/utils.py", line 222, in run_and_report
    raise ex
  File "/home/sam/dev/sheeprl/.venv/lib/python3.10/site-packages/hydra/_internal/utils.py", line 219, in run_and_report
    return func()
  File "/home/sam/dev/sheeprl/.venv/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/home/sam/dev/sheeprl/.venv/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 105, in run
    cfg = self.compose_config(
  File "/home/sam/dev/sheeprl/.venv/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 594, in compose_config
    cfg = self.config_loader.load_configuration(
  File "/home/sam/dev/sheeprl/.venv/lib/python3.10/site-packages/hydra/_internal/config_loader_impl.py", line 142, in load_configuration
    return self._load_configuration_impl(
  File "/home/sam/dev/sheeprl/.venv/lib/python3.10/site-packages/hydra/_internal/config_loader_impl.py", line 253, in _load_configuration_impl
    defaults_list = create_defaults_list(
  File "/home/sam/dev/sheeprl/.venv/lib/python3.10/site-packages/hydra/_internal/defaults_list.py", line 753, in create_defaults_list
    overrides.ensure_overrides_used()
  File "/home/sam/dev/sheeprl/.venv/lib/python3.10/site-packages/hydra/_internal/defaults_list.py", line 168, in ensure_overrides_used
    raise ConfigCompositionException(msg)
hydra.errors.ConfigCompositionException: Could not override 'checkpoint'. No match in the defaults list.
To append to your default list use +checkpoint=logs/runs/dreamer_v3/MsPacmanNoFrameskip-v4/2024-03-24_20-28-49_dreamer_v3_MsPacmanNoFrameskip-v4_5/version_0/checkpoint/ckpt_600000_0.ckpt

Then if I try the suggested command, it does

HYDRA_FULL_ERROR=1 python sheeprl_eval.py fabric.accelerator=gpu  +checkpoint=logs/runs/dreamer_v3/MsPacmanNoFrameskip-v4/2024-03-24_20-28-49_dreamer_v3_MsPacmanNoFrameskip-v4_5/version_0/checkpoint/ckpt_600000_0.ckpt
Traceback (most recent call last):
  File "/home/sam/dev/sheeprl/sheeprl_eval.py", line 4, in <module>
    evaluation()
  File "/home/sam/dev/sheeprl/.venv/lib/python3.10/site-packages/hydra/main.py", line 90, in decorated_main
    _run_hydra(
  File "/home/sam/dev/sheeprl/.venv/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/home/sam/dev/sheeprl/.venv/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/home/sam/dev/sheeprl/.venv/lib/python3.10/site-packages/hydra/_internal/utils.py", line 222, in run_and_report
    raise ex
  File "/home/sam/dev/sheeprl/.venv/lib/python3.10/site-packages/hydra/_internal/utils.py", line 219, in run_and_report
    return func()
  File "/home/sam/dev/sheeprl/.venv/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/home/sam/dev/sheeprl/.venv/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 105, in run
    cfg = self.compose_config(
  File "/home/sam/dev/sheeprl/.venv/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 594, in compose_config
    cfg = self.config_loader.load_configuration(
  File "/home/sam/dev/sheeprl/.venv/lib/python3.10/site-packages/hydra/_internal/config_loader_impl.py", line 142, in load_configuration
    return self._load_configuration_impl(
  File "/home/sam/dev/sheeprl/.venv/lib/python3.10/site-packages/hydra/_internal/config_loader_impl.py", line 253, in _load_configuration_impl
    defaults_list = create_defaults_list(
  File "/home/sam/dev/sheeprl/.venv/lib/python3.10/site-packages/hydra/_internal/defaults_list.py", line 746, in create_defaults_list
    defaults, tree = _create_defaults_list(
  File "/home/sam/dev/sheeprl/.venv/lib/python3.10/site-packages/hydra/_internal/defaults_list.py", line 716, in _create_defaults_list
    defaults_tree = _create_defaults_tree(
  File "/home/sam/dev/sheeprl/.venv/lib/python3.10/site-packages/hydra/_internal/defaults_list.py", line 356, in _create_defaults_tree
    ret = _create_defaults_tree_impl(
  File "/home/sam/dev/sheeprl/.venv/lib/python3.10/site-packages/hydra/_internal/defaults_list.py", line 457, in _create_defaults_tree_impl
    return _expand_virtual_root(repo, root, overrides, skip_missing)
  File "/home/sam/dev/sheeprl/.venv/lib/python3.10/site-packages/hydra/_internal/defaults_list.py", line 280, in _expand_virtual_root
    subtree = _create_defaults_tree_impl(
  File "/home/sam/dev/sheeprl/.venv/lib/python3.10/site-packages/hydra/_internal/defaults_list.py", line 573, in _create_defaults_tree_impl
    add_child(children, new_root)
  File "/home/sam/dev/sheeprl/.venv/lib/python3.10/site-packages/hydra/_internal/defaults_list.py", line 520, in add_child
    subtree_ = _create_defaults_tree_impl(
  File "/home/sam/dev/sheeprl/.venv/lib/python3.10/site-packages/hydra/_internal/defaults_list.py", line 488, in _create_defaults_tree_impl
    config_not_found_error(repo=repo, tree=root)
  File "/home/sam/dev/sheeprl/.venv/lib/python3.10/site-packages/hydra/_internal/defaults_list.py", line 800, in config_not_found_error
    raise MissingConfigException(
hydra.errors.MissingConfigException: In 'eval_config': Could not find 'checkpoint/logs/runs/dreamer_v3/MsPacmanNoFrameskip-v4/2024-03-24_20-28-49_dreamer_v3_MsPacmanNoFrameskip-v4_5/version_0/checkpoint/ckpt_600000_0.ckpt'

Available options in 'checkpoint':
    default
Config search path:
    provider=hydra, path=pkg://hydra.conf
    provider=main, path=pkg://sheeprl.configs
    provider=schema, path=structured://

Am I missing something or it's a bug?

belerico commented 5 months ago

Hi @geranim0, you should call the evaluation script with python sheeprl_eval.py checkpoint_path=/path/to/checkpoint .... Can you try this?

geranim0 commented 5 months ago

Ah... Sorry... Missed the _path! It works