TypeError: reset() got an unexpected keyword argument 'seed'

[deepboltzer]

There is an issue with the seeds after the Seeding update (#2422).

File "/home/joe/reinforce/rlenv/lib/python3.8/site-packages/gym/wrappers/time_limit.py", line 29, in reset
    return self.env.reset(seed=seed, **kwargs)
TypeError: reset() got an unexpected keyword argument 'seed'

Code for Imitation:

import gym
import ale_py
import time

print('gym:', gym.__version__)
print('ale_py:', ale_py.__version__)

env = gym.make("BreakoutNoFrameskip-v4")

print("Observation Space: ", env.observation_space)
print("Action Space       ", env.action_space)

obs = env.reset()

for i in range(1000):
    action = env.action_space.sample()
    obs, reward, done, info = env.step(action)
    env.render()
    time.sleep(0.01)
env.close()

I am using gym: 0.21.0 and ale_py: 0.7.3.

[vwxyzjn]

In pybullet I just hit the same issue. It's likely going to cause most of the third-party envs to fail.

[vwxyzjn]

@RedTachyon would you mind looking into this?

[RedTachyon]

I'm confused, are you sure it's gym release 0.21? This was released long before any seeding changes made their way into master.

[RedTachyon]

Ah shit, I managed to replicate it with pybullet, I think I know what's up. Basically wrappers forward the arguments to the inside environment, and while "new style" environments can accept anything in reset, old environments can't. So even if you don't do anything, it's trying to pass the default None onward to the environment.

Thanks for the catch, I think I have an idea on how to fix it, which will be possible thanks to the other change to the reset signature.

[RedTachyon]

The latest commit in #2515 fixes it for me with pybullet, can someone check it with atari? (for some reason a simple install doesn't work when I try to use it, and I never used atari, so probably some of the ROM stuff is lost on me)

[deepboltzer]

Thank you so much for fixing this issue @RedTachyon! The latest commit in #2515 also fixes the problem for atari.

[vwxyzjn]

Hey, I am still having this issue with the latest gym release gym=0.22.0. @jkterry1 could u re-open the issue? See the following snippet:

import gym
import numpy as np
import pybullet_envs

env = gym.make("HopperBulletEnv-v0")
env.reset()

env = gym.make("HopperBulletEnv-v0")
env = gym.wrappers.ClipAction(env)
env = gym.wrappers.NormalizeObservation(env)
env = gym.wrappers.TransformObservation(env, lambda obs: np.clip(obs, -10, 10))
env = gym.wrappers.NormalizeReward(env)
env = gym.wrappers.TransformReward(env, lambda reward: np.clip(reward, -10, 10))
env.reset()

which yields. So looks like without the wrappers the reset was ok, but with the wrappers, the reset was not ok.

pybullet build time: Sep  3 2021 23:59:09
/home/costa/.cache/pypoetry/virtualenvs/cleanrl-ghSZGHE3-py3.9/lib/python3.9/site-packages/gym/spaces/box.py:78: UserWarning: WARN: Box bound precision lowered by casting to float32
  logger.warn(f"Box bound precision lowered by casting to {self.dtype}")
argv[0]=
argv[0]=
Traceback (most recent call last):
  File "/home/costa/Documents/go/src/github.com/cleanrl/cleanrl/test.py", line 14, in <module>
    env.reset()
  File "/home/costa/.cache/pypoetry/virtualenvs/cleanrl-ghSZGHE3-py3.9/lib/python3.9/site-packages/gym/core.py", line 324, in reset
    return self.env.reset(**kwargs)
  File "/home/costa/.cache/pypoetry/virtualenvs/cleanrl-ghSZGHE3-py3.9/lib/python3.9/site-packages/gym/core.py", line 283, in reset
    return self.env.reset(**kwargs)
  File "/home/costa/.cache/pypoetry/virtualenvs/cleanrl-ghSZGHE3-py3.9/lib/python3.9/site-packages/gym/core.py", line 311, in reset
    return self.observation(self.env.reset(**kwargs))
  File "/home/costa/.cache/pypoetry/virtualenvs/cleanrl-ghSZGHE3-py3.9/lib/python3.9/site-packages/gym/wrappers/normalize.py", line 75, in reset
    obs = self.env.reset(seed=seed, options=options)
  File "/home/costa/.cache/pypoetry/virtualenvs/cleanrl-ghSZGHE3-py3.9/lib/python3.9/site-packages/gym/core.py", line 337, in reset
    return self.env.reset(**kwargs)
  File "/home/costa/.cache/pypoetry/virtualenvs/cleanrl-ghSZGHE3-py3.9/lib/python3.9/site-packages/gym/wrappers/time_limit.py", line 26, in reset
    return self.env.reset(**kwargs)
  File "/home/costa/.cache/pypoetry/virtualenvs/cleanrl-ghSZGHE3-py3.9/lib/python3.9/site-packages/gym/wrappers/order_enforcing.py", line 18, in reset
    return self.env.reset(**kwargs)
TypeError: reset() got an unexpected keyword argument 'seed'

[vwxyzjn]

Looks like this is an issue with https://github.com/openai/gym/blob/bdde1ede6c52ea5fa95f54404fa3e4f4420d7afc/gym/wrappers/normalize.py#L77 that forces to use the seed during reset

[RedTachyon]

Good catch, this will need a patch, should be fairly simple

[karlengblom]

Hello! Thanks to everyone for all the work on this repo. I get the same error message as above when running the bipedal_walker.py heuristic:

(stable-baselines3) karl@omen:~$ python reps/gym/gym/envs/box2d/bipedal_walker.py pygame 2.1.0 (SDL 2.0.16, Python 3.8.10) Hello from the pygame community. https://www.pygame.org/contribute.html Traceback (most recent call last): File "reps/gym/gym/envs/box2d/bipedal_walker.py", line 687, in env.reset() File "reps/gym/gym/envs/box2d/bipedal_walker.py", line 361, in reset super().reset(seed=seed) TypeError: reset() got an unexpected keyword argument 'seed'

[vwxyzjn]

@karlengblom What is your gym version? Do you have a demo code to replicate this?

[karlengblom]

The version is 0.23.0. I was trying to look at the bipedal walker heuristic, so the code is just python bipedal_walker.py

[RedTachyon]

@karlengblom Can you please double check that you have a fresh installation of gym? I can't replicate the issue, and it seems pretty impossible in terms of the code. If you were updating gym from an older version, perhaps there are some __pycache__ and *.pyc with the old code?

[vwxyzjn]

It works on my end.

(cleanrl-ghSZGHE3-py3.9) ➜  cleanrl git:(fix-dqn-seed) ✗ pip install gym==0.23.0 box2d-py
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: gym==0.23.0 in /home/costa/.cache/pypoetry/virtualenvs/cleanrl-ghSZGHE3-py3.9/lib/python3.9/site-packages (0.23.0)
Requirement already satisfied: box2d-py in /home/costa/.cache/pypoetry/virtualenvs/cleanrl-ghSZGHE3-py3.9/lib/python3.9/site-packages (2.3.8)
Requirement already satisfied: cloudpickle>=1.2.0 in /home/costa/.cache/pypoetry/virtualenvs/cleanrl-ghSZGHE3-py3.9/lib/python3.9/site-packages (from gym==0.23.0) (2.0.0)
Requirement already satisfied: numpy>=1.18.0 in /home/costa/.cache/pypoetry/virtualenvs/cleanrl-ghSZGHE3-py3.9/lib/python3.9/site-packages (from gym==0.23.0) (1.21.5)
Requirement already satisfied: importlib-metadata>=4.10.0 in /home/costa/.cache/pypoetry/virtualenvs/cleanrl-ghSZGHE3-py3.9/lib/python3.9/site-packages (from gym==0.23.0) (4.11.2)
Requirement already satisfied: gym-notices>=0.0.4 in /home/costa/.cache/pypoetry/virtualenvs/cleanrl-ghSZGHE3-py3.9/lib/python3.9/site-packages (from gym==0.23.0) (0.0.5)
Requirement already satisfied: zipp>=0.5 in /home/costa/.cache/pypoetry/virtualenvs/cleanrl-ghSZGHE3-py3.9/lib/python3.9/site-packages (from importlib-metadata>=4.10.0->gym==0.23.0) (3.7.0)
WARNING: You are using pip version 22.0.3; however, version 22.0.4 is available.
You should consider upgrading via the '/home/costa/.cache/pypoetry/virtualenvs/cleanrl-ghSZGHE3-py3.9/bin/python -m pip install --upgrade pip' command.
(cleanrl-ghSZGHE3-py3.9) ➜  cleanrl git:(fix-dqn-seed) ✗ python /home/costa/.cache/pypoetry/virtualenvs/cleanrl-ghSZGHE3-py3.9/lib/python3.9/site-packages/gym/envs/box2d/bipedal_walker.py
pygame 2.1.0 (SDL 2.0.16, Python 3.9.5)
Hello from the pygame community. https://www.pygame.org/contribute.html

action ['+0.00', '+0.00', '+0.00', '+0.00']
step 0 total_reward -0.08
hull ['-0.01', '-0.00', '-0.00', '-0.00']
leg0 ['+0.44', '+0.06', '+0.12', '-0.08', '+1.00']
leg1 ['+0.34', '+0.07', '+0.12', '-0.09', '+1.00']

action ['+0.03', '-0.01', '-0.02', '+0.42']
step 20 total_reward -0.73
hull ['-0.06', '+0.01', '+0.01', '-0.01']
leg0 ['+1.01', '-0.13', '-0.57', '-0.02', '+0.00']
leg1 ['+0.51', '-0.06', '-0.08', '-0.03', '+0.00']

action ['+0.03', '-0.01', '-0.01', '+0.43']
step 40 total_reward -0.62
hull ['-0.03', '+0.00', '+0.02', '-0.00']
leg0 ['+1.00', '+0.02', '-0.59', '-0.00', '+0.00']
leg1 ['+0.46', '-0.03', '-0.11', '-0.00', '+0.00']

action ['+0.04', '-0.01', '-0.01', '+0.42']
step 60 total_reward -0.29
hull ['-0.03', '-0.00', '+0.06', '-0.00']
leg0 ['+0.99', '-0.01', '-0.59', '+0.00', '+0.00']
leg1 ['+0.39', '-0.07', '-0.10', '+0.01', '+0.00']
^CTraceback (most recent call last):
  File "/home/costa/.cache/pypoetry/virtualenvs/cleanrl-ghSZGHE3-py3.9/lib/python3.9/site-packages/gym/envs/box2d/bipedal_walker.py", line 764, in <module>
    env.render()
  File "/home/costa/.cache/pypoetry/virtualenvs/cleanrl-ghSZGHE3-py3.9/lib/python3.9/site-packages/gym/envs/box2d/bipedal_walker.py", line 658, in render
    self.clock.tick(self.metadata["render_fps"])
KeyboardInterrupt

[karlengblom]

It works. I mixed up the gym version containing the bipedal_walker.py file itself with the gym version imported by it. It was imported from a python env made for SB3, so version 0.19. Thanks for responses, sorry for the inconvenience.

[bdytx5]

For me, the solution is to upgrade gym to the newest version

[JonaSi754]

Hi, stable baseline3 had been working well in my project, but when using gymnasium, I encountered this error. What should I do to fix it?

[pseudo-rnd-thoughts]

@JonaSi754 Could you provide some code to replicate this

[JonaSi754]

@JonaSi754 Could you provide some code to replicate this

Sure, I use a custom env. Here is the env I create:

import gymnasium
from gymnasium import spaces

class env(gymnasium.Env):
    def __init__(self) -> str:
        super().__init__()
        self.action_shape = (5,)
        self.action_space = spaces.box.Box(low=-1, high=1, shape=self.action_shape)
        self.observation_shape = (9,)
        self.observation_space = spaces.box.Box(low=0, high=1, shape=self.observation_shape)

And here is the traceback:

Traceback (most recent call last):
  File "/home/sssjh/Workspace/MyRL/code/algo/PPO_task.py", line 72, in <module>
    model.learn(total_timesteps=cfg.n_episodes * cfg.n_steps)
  File "/home/sssjh/miniconda3/envs/SB3/lib/python3.9/site-packages/stable_baselines3/ppo/ppo.py", line 308, in learn
    return super().learn(
  File "/home/sssjh/miniconda3/envs/SB3/lib/python3.9/site-packages/stable_baselines3/common/on_policy_algorithm.py", line 246, in learn
    total_timesteps, callback = self._setup_learn(
  File "/home/sssjh/miniconda3/envs/SB3/lib/python3.9/site-packages/stable_baselines3/common/base_class.py", line 424, in _setup_learn
    self._last_obs = self.env.reset()  # type: ignore[assignment]
  File "/home/sssjh/miniconda3/envs/SB3/lib/python3.9/site-packages/stable_baselines3/common/vec_env/dummy_vec_env.py", line 76, in reset
    obs, self.reset_infos[env_idx] = self.envs[env_idx].reset(seed=self._seeds[env_idx])
  File "/home/sssjh/miniconda3/envs/SB3/lib/python3.9/site-packages/stable_baselines3/common/monitor.py", line 83, in reset
    return self.env.reset(**kwargs)
TypeError: reset() got an unexpected keyword argument 'seed'

[pseudo-rnd-thoughts]

@JonaSi754 You need to provide the reset function (and step function), also what gym and sb3 do you use?

[JonaSi754]

@pseudo-rnd-thoughts sry for confusion, but my original code could be a little bit long. I remove some irrelevant functions. Here is the env.py

class env(gymnasium.Env):
    metadata = {'render.modes': ['human']}

def __init__(self, instance) -> str:
    super().__init__()
    self.instance = instance
    self.Jobs = read_JSP(self.instance)
    self.action_shape = (5,)
    self.action_space = spaces.box.Box(low=-1, high=1, shape=self.action_shape)
    self.observation_shape = (9,)
    self.observation_space = spaces.box.Box(low=0, high=1, shape=self.observation_shape)
    self.baseline = schedule_by_rules(instance, 5) # MOPNR

def reset(self):
    for job in self.Jobs:
        job.reset()
    self.states = states(self.Jobs)
    self.done = False
    self.state = np.zeros(self.observation_shape, dtype=np.float32)
    self.next_machines = []
    self.timer = 0
    self.offload = [False for _ in range(len(self.Jobs))]
    self.fillMachines()
    return self.state

def step(self, action):

    # Deal with action
    # Get the job and machine that we choose
    selector = action_v6(self.action_shape)
    buffer = self.next_machines[0].buffer
    _job_state = [bool(job.endTime) and job.endTime[-1] > self.timer for job in buffer]
    mask = [-int(n) + 1.1 for n in _job_state]
    this_job = selector.select_job(buffer, action, mask)

    # search forward for a position to insert
    # Or append on the tail
    self.forward_insert(this_job)

    # Record last makespan
    last_makespan = self.states.makespan

    # Update state space
    self.states.update_state(this_job, self.next_machines[0].No)

    # Give a reward
    reward = last_makespan - self.states.makespan

    # Judge if this batch of task has been done
    if sum([job.done for job in self.Jobs]) == len(self.Jobs):
        self.done = True

    # skip to the next schedule point
    self.next_machines.remove(self.next_machines[0])
    if not self.done:
        # Calculate observation for next machine
        self.fillMachines()
        self.state = self.states.observe(self.next_machines[0].No)

    # Record extra information of this step
    info = {}

    return self.state, reward, self.done, info

''' And I use gym0.21.0, gymnasium0.28.1 and stable-baselines3 2.0.0

[JonaSi754]

Oh, I just find the pattern code also meets the same problem, can you check it https://imitation.readthedocs.io/en/latest/algorithms/bc.html my versions are: gym-0.21.0, gymnasium-0.28.1, SB3-2.0.0

[pseudo-rnd-thoughts]

You either need to downgrade sb3 to 1.X or use the Gym compatibility environment (https://gymnasium.farama.org/content/gym_compatibility/#gym-v0-21-environment-compatibility) as your environment is implemented in gym's v0.21 API not v0.26's api

[JonaSi754]

thank you very much, I have resolved this!

[Hugh-Kane]

Hi, I was also having similar issues with gym_super_mario_bros, where I get the following error: TypeError: reset() got an unexpected keyword argument 'seed'

I am currently using: Python ==3.8 gym == 0.26.2 gym-super-mario-bros == 7.4.0
stable-baselines3 == 2.0.0

I would appreciate any help! Code is as follows:

# Import game
import gym_super_mario_bros
# Import joypad

from nes_py.wrappers import JoypadSpace
# Import simplified controls
from gym_super_mario_bros.actions import SIMPLE_MOVEMENT

"""
#Preprocessing step
"""
#grayscale cuts down the processing power by 66% since we don't need to process all RGB channels
from gym.wrappers import GrayScaleObservation
#import vectorization wrappers 
from stable_baselines3.common.vec_env import VecFrameStack, DummyVecEnv
from matplotlib import pyplot as plt
from stable_baselines3.common.env_checker import check_env

# 1.Create the base environment.
env = gym_super_mario_bros.make('SuperMarioBros-v3',apply_api_compatibility=True,render_mode="human" )
# 2.Simplify controls
env = JoypadSpace(env,SIMPLE_MOVEMENT)
# 3.Grayscale
#Without keep_dim, shape corresponds to (240, 256)
#With keep_dim, shape corresponds to (240, 256, 1)
env = GrayScaleObservation(env, keep_dim=True)
# 4.Wrap inside the Dummy Environment
env = DummyVecEnv([lambda: env])

state= env.reset()

#print(gym_super_mario_bros.list_envs())

[jcianci12]

For me, the solution is to upgrade gym to the newest version

Do you mean the latest version of gym or gymnasium. Thanks! 🙏

[phuongboi]

Install gym 0.18.0 (pip install gym==0.18.0) instead of gymnasium solved my problem

[jeffheaton]

Agree, using gym instead solves the issue. So we have to use the old deprecated library?

[RedTachyon]

If you want to use another old deprecated library that's been written for the old deprecated library, then there's a chance you'll have to use an old deprecated library, yes

[rohitsaikrishnan]

Hi, I was also having similar issues with gym_super_mario_bros, where I get the following error: TypeError: reset() got an unexpected keyword argument 'seed'

I am currently using: Python ==3.8 gym == 0.26.2 gym-super-mario-bros == 7.4.0 stable-baselines3 == 2.0.0

I would appreciate any help! Code is as follows:

# Import game
import gym_super_mario_bros
# Import joypad

from nes_py.wrappers import JoypadSpace
# Import simplified controls
from gym_super_mario_bros.actions import SIMPLE_MOVEMENT

"""
#Preprocessing step
"""
#grayscale cuts down the processing power by 66% since we don't need to process all RGB channels
from gym.wrappers import GrayScaleObservation
#import vectorization wrappers 
from stable_baselines3.common.vec_env import VecFrameStack, DummyVecEnv
from matplotlib import pyplot as plt
from stable_baselines3.common.env_checker import check_env

# 1.Create the base environment.
env = gym_super_mario_bros.make('SuperMarioBros-v3',apply_api_compatibility=True,render_mode="human" )
# 2.Simplify controls
env = JoypadSpace(env,SIMPLE_MOVEMENT)
# 3.Grayscale
#Without keep_dim, shape corresponds to (240, 256)
#With keep_dim, shape corresponds to (240, 256, 1)
env = GrayScaleObservation(env, keep_dim=True)
# 4.Wrap inside the Dummy Environment
env = DummyVecEnv([lambda: env])

state= env.reset()

#print(gym_super_mario_bros.list_envs())

I have the similar issue. Any help here please.

[pseudo-rnd-thoughts]

https://github.com/Kautenja/nes-py/blob/b6f4e26a96cf1dc2329b3b45b9f785dd7dbb30c6/nes_py/wrappers/joypad_space.py#L76

The NES-py wrapper reset function doesn't have a seed parameter therefore it fails

[AbhayGoyal]

@JonaSi754 Can you tell me how me how you fixed this issue? What are the exact Stable baselines and Gynasium and Gym package version used?

Thanks

[AbhayGoyal]

@pseudo-rnd-thoughts, I am using Gym 0.26.0 and stable 2.3.2 and still get the same issue. I thought with v26 this issue should not be there right?