cannot run GTrXL demo since v0.5.0

[klcheungaj]

• [ ] I have marked all applicable categories:
  • [x] exception-raising bug
  • [ ] RL algorithm bug
  • [ ] system worker bug
  • [ ] system utils bug
  • [ ] code design/refactor
  • [ ] documentation request
  • [ ] new feature request
• [x] I have visited the readme and doc
• [x] I have searched through the issue tracker and pr tracker
• [x] I have mentioned version numbers, operating system and environment, where applicable:

they worked fine in v0.4.8 and v0.4.9 but not since v0.5.0

cartpole:

  /home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/treevalue/tree/integration/torch.py:21: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
    register_for_torch(TreeValue)
  /home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/treevalue/tree/integration/torch.py:22: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
    register_for_torch(FastTreeValue)
  Traceback (most recent call last):
    File "/home/kenny/repo/DI-engine/dizoo/classic_control/cartpole/config/./cartpole_r2d2_gtrxl_config.py", line 78, in <module>
      serial_pipeline((main_config, create_config), seed=0)
    File "/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/ding/entry/serial_entry.py", line 61, in serial_pipeline
      policy = create_policy(cfg.policy, model=model, enable_field=['learn', 'collect', 'eval', 'command'])
    File "/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/ding/policy/base_policy.py", line 844, in create_policy
      return POLICY_REGISTRY.build(cfg.type, cfg=cfg, **kwargs)
    File "/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/ding/utils/registry.py", line 96, in build
      raise e
    File "/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/ding/utils/registry.py", line 82, in build
      return build_fn(*obj_args, **obj_kwargs)
    File "/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/ding/policy/base_policy.py", line 134, in __init__
      model = self._create_model(cfg, model)
    File "/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/ding/policy/base_policy.py", line 219, in _create_model
      return create_model(model_cfg)
    File "/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/ding/model/common/utils.py", line 31, in create_model
      return MODEL_REGISTRY.build(cfg.pop("type"), **cfg)
    File "/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/ding/utils/registry.py", line 96, in build
      raise e
    File "/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/ding/utils/registry.py", line 82, in build
      return build_fn(*obj_args, **obj_kwargs)
    File "/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/ding/model/template/q_learning.py", line 1117, in __init__
      raise NotImplementedError("not support obs_shape for pre-defined encoder: {}".format(obs_shape))
  NotImplementedError: not support obs_shape for pre-defined encoder: 4

Pong

  /home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/treevalue/tree/integration/torch.py:21: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
    register_for_torch(TreeValue)
  /home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/treevalue/tree/integration/torch.py:22: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
    register_for_torch(FastTreeValue)
  A.L.E: Arcade Learning Environment (version 0.7.5+db37282)
  [Powered by Stella]
  /home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/gym/core.py:329: DeprecationWarning: WARN: Initializing wrapper in old step API which returns one bool instead of two. It is recommended to set `new_step_api=True` to use new step API. This will be the default behaviour in future.
    deprecation(
  /home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/gym/wrappers/step_api_compatibility.py:39: DeprecationWarning: WARN: Initializing environment in old step API which returns one bool instead of two. It is recommended to set `new_step_api=True` to use new step API. This will be the default behaviour in future.
    deprecation(
  /home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/gym/utils/passive_env_checker.py:227: DeprecationWarning: WARN: Core environment is written in old step API which returns one bool instead of two. It is recommended to rewrite the environment with new step API. 
    logger.deprecation(
  /home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/gym/utils/passive_env_checker.py:233: DeprecationWarning: `np.bool8` is a deprecated alias for `np.bool_`.  (Deprecated NumPy 1.24)
    if not isinstance(done, (bool, np.bool8)):
  Traceback (most recent call last):
    File "/home/kenny/repo/DI-engine/dizoo/atari/config/serial/pong/pong_r2d2_gtrxl_config.py", line 89, in <module>
      serial_pipeline([main_config, create_config], seed=0)
    File "/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/ding/entry/serial_entry.py", line 61, in serial_pipeline
      policy = create_policy(cfg.policy, model=model, enable_field=['learn', 'collect', 'eval', 'command'])
    File "/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/ding/policy/base_policy.py", line 844, in create_policy
      return POLICY_REGISTRY.build(cfg.type, cfg=cfg, **kwargs)
    File "/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/ding/utils/registry.py", line 96, in build
      raise e
    File "/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/ding/utils/registry.py", line 82, in build
      return build_fn(*obj_args, **obj_kwargs)
    File "/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/ding/policy/base_policy.py", line 161, in __init__
      getattr(self, '_init_' + field)()
    File "/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/ding/policy/r2d2_gtrxl.py", line 355, in _init_collect
      assert 'unroll_len' not in self._cfg.collect, "Use default unroll_len"
  AssertionError: Use default unroll_len

[PaParaZz1]

We have fixed this problem in the above commit.

Here is a screenshot example of the final converged agent: