google-deepmind / acme

A library of reinforcement learning components and agents
Apache License 2.0
3.52k stars 426 forks source link

Missleading errors when using discrete action space with `CanonicalSpecWrapper` #294

Open Whojo opened 1 year ago

Whojo commented 1 year ago

Problem

Using an environment which uses a DiscreteArray as action's spec (e.g. gym's CartPole) with the CanonicalSpecWrapper (which, to the best of my knowledge, should be used with continuous action space only) returns the following error, which is quite missleading.

Traceback (most recent call last):
  File "run_sac.py", line 86, in <module>
    app.run(main)
  File "/usr/local/lib/python3.8/dist-packages/absl/app.py", line 308, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.8/dist-packages/absl/app.py", line 254, in _run_main
    sys.exit(main(argv))
  File "run_sac.py", line 73, in main
    config = build_experiment_config()
  File "run_sac.py", line 47, in build_experiment_config
    environment_spec = specs.make_environment_spec(environment)
  File "/usr/local/lib/python3.8/dist-packages/acme/specs.py", line 46, in make_environment_spec
    actions=environment.action_spec(),
  File "/usr/local/lib/python3.8/dist-packages/acme/wrappers/single_precision.py", line 42, in action_spec
    return _convert_spec(self._environment.action_spec())
  File "/usr/local/lib/python3.8/dist-packages/acme/wrappers/canonical_spec.py", line 54, in action_spec
    return _convert_spec(self._environment.action_spec())
  File "/usr/local/lib/python3.8/dist-packages/acme/wrappers/canonical_spec.py", line 68, in _convert_spec
    return tree.map_structure(_convert_single_spec, nested_spec)
  File "/usr/local/lib/python3.8/dist-packages/tree/__init__.py", line 430, in map_structure
    [func(*args) for args in zip(*map(flatten, structures))])
  File "/usr/local/lib/python3.8/dist-packages/tree/__init__.py", line 430, in <listcomp>
    [func(*args) for args in zip(*map(flatten, structures))])
  File "/usr/local/lib/python3.8/dist-packages/acme/wrappers/canonical_spec.py", line 63, in _convert_single_spec
    return spec.replace(
  File "/usr/local/lib/python3.8/dist-packages/dm_env/specs.py", line 146, in replace
    return type(self)(**all_kwargs)
TypeError: __init__() got an unexpected keyword argument 'minimum'

Expected behavior

Do not raise an error.

Why is it important to solve ?

Solution

The following change should do the work, from: https://github.com/deepmind/acme/blob/d1e69c92000079b118b868ce9303ee6d39c4a0b6/acme/wrappers/canonical_spec.py#L62

To:

 if type(spec) == specs.BoundedArray: