DepRL expects gym but MyoSuite uses the newer gymnasium

[MittelmanDaniel]

This is the depRL example

from myosuite.utils import gym
import deprl

# we can pass arguments to the environments here
env = gym.make('myoLegWalk-v0', reset_type='random')
policy = deprl.load_baseline(env)
obs = env.reset()
for i in range(1000):
    env.mj_render()
    action = policy(obs)
    obs, *_ = env.step(action)
env.close()

When running it I get

python test.py 
/home/daniel/anaconda3/envs/MyoSuite/lib/python3.8/site-packages/pygame/pkgdata.py:25: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  from pkg_resources import resource_stream, resource_exists
/home/daniel/anaconda3/envs/MyoSuite/lib/python3.8/site-packages/pkg_resources/__init__.py:3154: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
pygame 2.6.0 (SDL 2.28.4, Python 3.8.19)
Hello from the pygame community. https://www.pygame.org/contribute.html
MyoSuite:> Registering Myo Envs
Traceback (most recent call last):
  File "test.py", line 2, in <module>
    import deprl
  File "/home/daniel/anaconda3/envs/MyoSuite/lib/python3.8/site-packages/deprl/__init__.py", line 1, in <module>
    from . import custom_agents, custom_mpo_torch, custom_trainer
  File "/home/daniel/anaconda3/envs/MyoSuite/lib/python3.8/site-packages/deprl/custom_agents.py", line 3, in <module>
    from .dep_controller import DEP
  File "/home/daniel/anaconda3/envs/MyoSuite/lib/python3.8/site-packages/deprl/dep_controller.py", line 5, in <module>
    import gym
ModuleNotFoundError: No module named 'gym'

because it expects gym, when pip installing gym I get another error about shapes not matching so I think there's a lot more broken

[Vittorio-Caggiano]

Hi @MittelmanDaniel , thanks for raising this issue. Can you please shre more info on which version of MyoSuite, OS and gym you are testing?

[P-Schumacher]

Hi Daniel, Deprl should work with gymnasium. Can you share all versions that Vittorio mentioned?

[MittelmanDaniel]

@Vittorio-Caggiano @P-Schumacher Hi!

I am on ubuntu 22.04, I have myosuite 2.7.0 because I just did pip install -U myosuite roughly one hour ago. I have deprl 0.3.2 I just did pip install deprl

Edit: I have gymnasium 0.29.1 (important to note not gym because I uninstalled it after it didnt work even with it)

[MittelmanDaniel]

To add on when running the myochallenge example

from myosuite.utils import gym
# Include the locomotion track environment, uncomment to select the manipulation challenge
env = gym.make('myoChallengeRunTrackP1-v0')
#env = gym.make('myoChallengeBimanual-v0')

env.reset()

# Repeat 1000 time steps
for _ in range(1000):

    # Activate mujoco rendering window
    env.mj_render()

    # Get observation from the envrionment, details are described in the above docs
    obs = env.get_obs()
    print(type(obs))
    current_time = obs['time']
    #print(current_time)

    # Take random actions
    action = env.action_space.sample()

    # Environment provides feedback on action
    next_obs, reward, terminated, truncated, info = env.step(action)

    # Reset training if env is terminated
    if terminated:
        next_obs, info = env.reset()

I get

MyoSuite:> Registering Myo Envs
    MyoSuite: A contact-rich simulation suite for musculoskeletal motor control
        Vittorio Caggiano, Huawei Wang, Guillaume Durandau, Massimo Sartori, Vikash Kumar
        L4DC-2019 | https://sites.google.com/view/myosuite

/home/daniel/anaconda3/envs/MyoSuite/lib/python3.8/site-packages/gymnasium/utils/passive_env_checker.py:159: UserWarning: WARN: The obs returned by the `reset()` method is not within the observation space.
  logger.warn(f"{pre} is not within the observation space.")
/home/daniel/anaconda3/envs/MyoSuite/lib/python3.8/site-packages/gymnasium/core.py:311: UserWarning: WARN: env.mj_render to get variables from other wrappers is deprecated and will be removed in v1.0, to get this variable you can do `env.unwrapped.mj_render` for environment variables or `env.get_wrapper_attr('mj_render')` that will search the reminding wrappers.
  logger.warn(
/home/daniel/anaconda3/envs/MyoSuite/lib/python3.8/site-packages/gymnasium/core.py:311: UserWarning: WARN: env.get_obs to get variables from other wrappers is deprecated and will be removed in v1.0, to get this variable you can do `env.unwrapped.get_obs` for environment variables or `env.get_wrapper_attr('get_obs')` that will search the reminding wrappers.
  logger.warn(
Traceback (most recent call last):
  File "test.py", line 18, in <module>
    current_time = obs['time']
IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices
Exception ignored in: <function Renderer.__del__ at 0x7d42c5cc18b0>
Traceback (most recent call last):
  File "/home/daniel/anaconda3/envs/MyoSuite/lib/python3.8/site-packages/myosuite/renderer/renderer.py", line 142, in __del__
  File "/home/daniel/anaconda3/envs/MyoSuite/lib/python3.8/site-packages/myosuite/renderer/mj_renderer.py", line 158, in close
NameError: name 'quit' is not defined

Because env is returned as an np array but is being accessed like a dict. Maybe there's something extremely wrong with my environment but it works with just stable baselines and gymnasium envs

[P-Schumacher]

Hey! can you try again with python==3.10 ? I suspect that an older python version forces an older deprl version which is not compatible with the newest myosuite version.

After installing deprl with python 3.10, please run pip list | grep deprl and tell me the deprl version number.

Best, Pierre

[MittelmanDaniel]

Ohh that may be the case I was on python 3.8 because that's the one the installation docs suggest https://myosuite.readthedocs.io/en/latest/install.html

I'll check when I get back to my laptop

Thanks so much!

[MittelmanDaniel]

@P-Schumacher Good news it finds dep! Bad news it throws a shapes dont match error

(MyoSuite) daniel@daniel-XPS-17-9720:~/myo_challenge$ python test.py 
MyoSuite:> Registering Myo Envs
    MyoSuite: A contact-rich simulation suite for musculoskeletal motor control
        Vittorio Caggiano, Huawei Wang, Guillaume Durandau, Massimo Sartori, Vikash Kumar
        L4DC-2019 | https://sites.google.com/view/myosuite

/home/daniel/anaconda3/envs/MyoSuite/lib/python3.10/site-packages/gymnasium/core.py:311: UserWarning: WARN: env.env_name to get variables from other wrappers is deprecated and will be removed in v1.0, to get this variable you can do `env.unwrapped.env_name` for environment variables or `env.get_wrapper_attr('env_name')` that will search the reminding wrappers.
  logger.warn(
Load LegWalk Baseline
Loading experiment from ./baselines_DEPRL/myoLegWalk_20230514/myoLeg/checkpoints
/home/daniel/anaconda3/envs/MyoSuite/lib/python3.10/site-packages/deprl/utils/load_utils.py:33: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  return torch.load(os.path.join(checkpoint_path, "time.pt"))
Found only the policy checkpoint, the previous run was likely only run with  <'full_save': False>
Only loading policy checkpoint.
Stochastic Switch-DEP. Paper version.

Loading weights from ./baselines_DEPRL/myoLegWalk_20230514/myoLeg/checkpoints/step_150000000.pt
/home/daniel/anaconda3/envs/MyoSuite/lib/python3.10/site-packages/deprl/vendor/tonic/torch/agents/agent.py:170: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  state_dict = torch.load(path, map_location=self.device)
/home/daniel/anaconda3/envs/MyoSuite/lib/python3.10/site-packages/gymnasium/core.py:311: UserWarning: WARN: env.mj_render to get variables from other wrappers is deprecated and will be removed in v1.0, to get this variable you can do `env.unwrapped.mj_render` for environment variables or `env.get_wrapper_attr('mj_render')` that will search the reminding wrappers.
  logger.warn(
/home/daniel/anaconda3/envs/MyoSuite/lib/python3.10/site-packages/deprl/vendor/tonic/torch/agents/mpo.py:90: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at ../torch/csrc/utils/tensor_new.cpp:278.)
  observations = torch.as_tensor(observations, dtype=torch.float32)
Traceback (most recent call last):
  File "/home/daniel/myo_challenge/test.py", line 10, in <module>
    action = policy(obs)
  File "/home/daniel/anaconda3/envs/MyoSuite/lib/python3.10/site-packages/deprl/vendor/tonic/agents/agent.py", line 42, in __call__
    return self.test_step(observation, steps=1e6)
  File "/home/daniel/anaconda3/envs/MyoSuite/lib/python3.10/site-packages/deprl/custom_agents.py", line 57, in test_step
    return super().test_step(observations, steps)
  File "/home/daniel/anaconda3/envs/MyoSuite/lib/python3.10/site-packages/deprl/vendor/tonic/torch/agents/mpo.py", line 61, in test_step
    return self._test_step(observations).detach().cpu().numpy()
  File "/home/daniel/anaconda3/envs/MyoSuite/lib/python3.10/site-packages/deprl/vendor/tonic/torch/agents/mpo.py", line 90, in _test_step
    observations = torch.as_tensor(observations, dtype=torch.float32)
ValueError: expected sequence of length 403 at dim 1 (got 0)

This is the depRL version

deprl                                0.5.0

This is the script I ran

from myosuite.utils import gym
import deprl

# we can pass arguments to the environments here
env = gym.make('myoLegWalk-v0', reset_type='random')
policy = deprl.load_baseline(env)
obs = env.reset()
for i in range(1000):
    env.mj_render()
    action = policy(obs)
    obs, *_ = env.step(action)
env.close()

[P-Schumacher]

okay, it seems the documentation is outdated on this

Run it like this:

from myosuite.utils import gym
import deprl
from deprl import env_wrappers
import time

# we can pass arguments to the environments here
env = gym.make('myoLegWalk-v0', reset_type='random')
env = env_wrappers.GymWrapper(env)
policy = deprl.load_baseline(env)
obs = env.reset()
for i in range(1000):
    env.mj_render()
    action = policy(obs)
    obs, *_ = env.step(action)
    time.sleep(0.01)
env.close()

The deprl gymwrapper had to be added.