New Update breaks RLLib compatibility

[Phoenix-surgere]

Hello,

I 've been using RLLib with PettingZoo and SuperSuit in Kaggle notebooks for a while now without an issue, but today I got some substantial, unexpected errors while doing my usual pip installs (since the notebooks don't have long term memory), and eventually I traced the problem back to SuperSuit, and specifically version 3.3.4. The traceback is the following:

AttributeError                            Traceback (most recent call last)
/tmp/ipykernel_34/3725812994.py in <module>
----> 1 from ray.rllib.agents.ppo import ppo, appo, ddppo
      2 from ray.rllib.agents.a3c import a2c, a3c
      3 from ray.rllib.agents.ddpg import ddpg, td3, apex
      4 from ray.rllib.agents.dqn import apex as apex_dqn, dqn, distributional_q_tf_model
      5 from ray.rllib.agents.impala import impala

/opt/conda/lib/python3.7/site-packages/ray/rllib/__init__.py in <module>
      3 # Note: do not introduce unnecessary library dependencies here, e.g. gym.
      4 # This file is imported from the tune module in order to register RLlib agents.
----> 5 from ray.rllib.env.base_env import BaseEnv
      6 from ray.rllib.env.external_env import ExternalEnv
      7 from ray.rllib.env.multi_agent_env import MultiAgentEnv

/opt/conda/lib/python3.7/site-packages/ray/rllib/env/__init__.py in <module>
      4 from ray.rllib.env.external_multi_agent_env import ExternalMultiAgentEnv
      5 from ray.rllib.env.multi_agent_env import MultiAgentEnv
----> 6 from ray.rllib.env.policy_client import PolicyClient
      7 from ray.rllib.env.policy_server_input import PolicyServerInput
      8 from ray.rllib.env.remote_base_env import RemoteBaseEnv

/opt/conda/lib/python3.7/site-packages/ray/rllib/env/policy_client.py in <module>
     14 from ray.rllib.env.external_multi_agent_env import ExternalMultiAgentEnv
     15 from ray.rllib.env.multi_agent_env import MultiAgentEnv
---> 16 from ray.rllib.policy.sample_batch import MultiAgentBatch
     17 from ray.rllib.utils.annotations import PublicAPI
     18 from ray.rllib.utils.multi_agent import check_multi_agent

/opt/conda/lib/python3.7/site-packages/ray/rllib/policy/__init__.py in <module>
      4 from ray.rllib.policy.policy_template import build_policy_class
      5 from ray.rllib.policy.torch_policy_template import build_torch_policy
----> 6 from ray.rllib.policy.tf_policy_template import build_tf_policy
      7 
      8 __all__ = [

/opt/conda/lib/python3.7/site-packages/ray/rllib/policy/tf_policy_template.py in <module>
      6 from ray.rllib.models.modelv2 import ModelV2
      7 from ray.rllib.policy.dynamic_tf_policy import DynamicTFPolicy
----> 8 from ray.rllib.policy import eager_tf_policy
      9 from ray.rllib.policy.policy import Policy
     10 from ray.rllib.policy.sample_batch import SampleBatch

/opt/conda/lib/python3.7/site-packages/ray/rllib/policy/eager_tf_policy.py in <module>
     10 
     11 from ray.util.debug import log_once
---> 12 from ray.rllib.evaluation.episode import Episode
     13 from ray.rllib.models.catalog import ModelCatalog
     14 from ray.rllib.models.repeated_values import RepeatedValues

/opt/conda/lib/python3.7/site-packages/ray/rllib/evaluation/__init__.py in <module>
      1 from ray.rllib.evaluation.episode import Episode, MultiAgentEpisode
----> 2 from ray.rllib.evaluation.rollout_worker import RolloutWorker
      3 from ray.rllib.evaluation.sample_batch_builder import (
      4     SampleBatchBuilder, MultiAgentSampleBatchBuilder)
      5 from ray.rllib.evaluation.sampler import SyncSampler, AsyncSampler

/opt/conda/lib/python3.7/site-packages/ray/rllib/evaluation/rollout_worker.py in <module>
     17 from ray.rllib.env.multi_agent_env import MultiAgentEnv
     18 from ray.rllib.env.external_multi_agent_env import ExternalMultiAgentEnv
---> 19 from ray.rllib.env.utils import record_env_wrapper
     20 from ray.rllib.env.vector_env import VectorEnv
     21 from ray.rllib.env.wrappers.atari_wrappers import wrap_deepmind, is_atari

/opt/conda/lib/python3.7/site-packages/ray/rllib/env/utils.py in <module>
     55 
     56 
---> 57 class VideoMonitor(wrappers.Monitor):
     58     # Same as original method, but doesn't use the StatsRecorder as it will
     59     # try to add up multi-agent rewards dicts, which throws errors.

AttributeError: module 'gym.wrappers' has no attribute 'Monitor'

After pip-installing the previous SuperSuit version, everything went back to normal, thankfully. You should check this thing out because it effectively cancels compatibility with ray == '1.10.0' at least. The code I used to produce the problem is the following:

!pip install pettingzoo[all]

!pip install supersuit
!pip install autorom

!pip install lz4

! AutoROM -y

and afterward, my imports which fail:

from ray.rllib.agents.ppo import ppo, appo, ddppo
from ray.rllib.agents.a3c import a2c, a3c
from ray.rllib.agents.ddpg import ddpg, td3, apex
from ray.rllib.agents.dqn import apex as apex_dqn, dqn, distributional_q_tf_model
from ray.rllib.agents.impala import impala
from ray.rllib.agents.sac import sac, rnnsac

Note that it is not just the PPO that fails, as I've tested it with a2c and other algos as well, hence why I state that it is probably an overarching bug.

[jkterry1]

Hey, I really apologize for not replying to this sooner. @dsctt will be creating a PR to RLlib that will fix this in the near future. I'm going to close this, but please let us know if you need any additional help :)