[Bug Report] FetchPickAndPlace-v2 does not yield reproducible results

[amacati]

Describe the bug The gymnasium API allows users to seed the environment on each reset to yield reproducible results. Running the environment with the same seed should always give the exact same results. While the documentation recommends that users should seed reset only once, it does not forbid seeding multiple times.

FetchPickAndPlace-v2 does not yield reproducible results under these conditions. The reset observation is identical, but the observations start deviating at the first environment step using identical actions.

Code example

import gymnasium
import numpy as np

def test_reproducibility(env: gymnasium.Env, seed: int = 42):
    env.action_space.seed(seed)  # Reproducible actions
    action = env.action_space.sample()  # Same random action for both runs
    env.reset(seed=seed)
    obs_1, _, _, _, _ = env.step(action)
    env.reset(seed=seed)  # Same seed should produce the same observations
    obs_2, _, _, _, _ = env.step(action)  # Identical action
    if isinstance(obs_1, dict):
        for key in obs_1:
            assert np.all(obs_1[key] == obs_2[key])  # Assertion error: different observations
    else:
        assert np.all(obs_1 == obs_2)
    print(f"Reproducibility test passed for {env.unwrapped.spec.id}")

def main():
    test_reproducibility(gymnasium.make('CartPole-v1'))  # Works
    test_reproducibility(gymnasium.make("FetchPickAndPlace-v2"))  # Fails

if __name__ == '__main__':
    main()

Stack Trace:

Reproducibility test passed for CartPole-v1
Traceback (most recent call last):
  File "/home/amacati/repos/Gymnasium-Robotics/bug_report.py", line 26, in <module>
    main()
  File "/home/amacati/repos/Gymnasium-Robotics/bug_report.py", line 22, in main
    test_reproducibility(gymnasium.make("FetchPickAndPlace-v2"))  # Fails
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/amacati/repos/Gymnasium-Robotics/bug_report.py", line 14, in test_reproducibility
    assert np.all(obs_1[key] == obs_2[key])  # Assertion error: different observations
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

System Info

• Clone the latest Gymnasium-Robotics commit (commit 50c801922273331c50956909f5bf3377a012233a)
• New mamba environment with Python 3.11
• Install Gymnasium-Robotics with pip install -e .
• Ubuntu 22.04 Jammy
• Python 3.11.7

Additional context The differences are small, i.e. they sometimes pass a np.allclose assert. In the example above, the object rotation in observation 1 is [-5.18150577e-08 7.97154734e-08 -1.37921664e-16] and [-5.18150577e-08 7.97154734e-08 -1.37780312e-16] in observation 2. Note the difference in z rotation. In fact, all three rotations are not equal, but the differences are too small to be printed without additional precision.

The inconsistencies arise from the FetchPickAndPlace environment's use of mocap bodies. The position and quaternions of the mocap bodies are currently not reset properly.

Furthermore, the Mujoco integrator uses warmstarts and caches the last controls in mjData. In the current implementation, these are also not reset. Only if these four mjData fields are properly restored to their initial states, env.reset(seed=seed) yields reproducible results.

I will open up a pull request that fixes this.

Checklist

• [x] I have checked that there is no similar issue in the repo (required)

[amacati]

I have opened up the PR https://github.com/Farama-Foundation/Gymnasium-Robotics/pull/208 that fixes this behaviour.

[Kallinteris-Andreas]

I can replicate the results on my machine, note: during testing we use atol=0.00001 https://github.com/Farama-Foundation/Gymnasium/blob/72cfbc204beca309579681b1201990a3d706e070/gymnasium/utils/env_checker.py#LL57

I expanded the test to cover all robotics environments

import gymnasium
import numpy as np
import pytest
from gymnasium.utils.env_checker import data_equivalence

robotics_full_env_list = []
for env_id, spec in gymnasium.envs.registration.registry.items():
    if spec.entry_point.startswith("gymnasium_robotics"):
        robotics_full_env_list.append(env_id)

@pytest.mark.parametrize("env_id", robotics_full_env_list)
def test_reproducibility(env_id: str, seed: int = 42):
    env = gymnasium.make(env_id)
    env.action_space.seed(seed)  # Reproducible actions
    action = env.action_space.sample()  # Same random action for both runs
    env.reset(seed=seed)
    obs_1, _, _, _, _ = env.step(action)
    env.reset(seed=seed)  # Same seed should produce the same observations
    obs_2, _, _, _, _ = env.step(action)  # Identical action
    if isinstance(obs_1, dict):
        for key in obs_1:
            assert np.all(obs_1[key] == obs_2[key])  # Assertion error: different observations
    else:
        assert np.all(obs_1 == obs_2)
    assert data_equivalence(obs_1, obs_2)
    print(f"Reproducibility test passed for {env.unwrapped.spec.id}")

and other ones seem to fail

[amacati]

Yes, I already suspected that any environment that uses Mujoco and allows for solver warm starts might have this issue.

By the way, I am not sure about the performance impact of removing warm starts. If that is something you are worried about, it might be a solution to just reset the buffers if a seed is passed to env.reset.

[Kallinteris-Andreas]

Seems to affect all RobotEnv environments

FetchSlide-v1
FetchSlide-v2
FetchPickAndPlace-v1
FetchPickAndPlace-v2
FetchReach-v1
FetchReach-v2
FetchPush-v1
FetchPush-v2
HandReach-v0
HandReach-v1
FetchSlideDense-v1
FetchSlideDense-v2
FetchPickAndPlaceDense-v1
FetchPickAndPlaceDense-v2
FetchReachDense-v1
FetchReachDense-v2
FetchPushDense-v1
FetchPushDense-v2
HandReachDense-v0
HandReachDense-v1

[amacati]

Should I update my PR to fix all of them?

[Kallinteris-Andreas]

updating robotEnv should in theory fix all of them (all the mujoco based ones at least)

[amacati]

I pushed a new fix (https://github.com/Farama-Foundation/Gymnasium-Robotics/pull/208/commits/5e14ea14d1ac46c60aaed7a00edc4f997510e705) that includes both the mujoco and mujoco_py envs. It passes all tests on my side including the new reproducibility tests, feel free to have a look