Closed wilhem closed 2 weeks ago
Yes,
from stable_baselines3.common.env_util import make_vec_env
vec_env = make_vec_env(CustomEnv, n_envs=4)
is the way to go.
If you want to use SubprocVecEnv
, use this instead:
from stable_baselines3.common.vec_env import SubprocVecEnv
vec_env = make_vec_env(CustomEnv, n_envs=4, vec_env_cls=SubprocVecEnv)
answer in the doc: Multiprocessing Unleashing the Power of Vectorized Environments
EDIT: pass the env class instead of the env.
Well, there is a small problem in the above code since 'env' is not callable.
However, even following the tutorial it does not work for me and I traced the problem to 'isinstance(env, gymnasium.Env)' in the function _patch_env. Although my CustomEnv inherits from gymnasium.Env the isinstance call returns False. If I look (with the debugger at that code location) at the inheritance structure with 'for cls in env.class.mro: print(cls)' I see that the inheritance information is gone: <class 'main.CustomEnv'> <class 'typing.Generic'> <class 'object'>
However, if I do the same in my main code it looks as expected: <class 'main.CustomGridEnv'> <class 'gymnasium.core.Env'> <class 'typing.Generic'> <class 'object'>
I do this under Windows using stable-baseline3 v2.3.2 (which uses 'spawn' as start_method). Has anyone an idea what I'm doing wrong?
Thanx
Wait, it is callable because if you look in the class definition I implemented the call method:
def __call__(self):
return self
I'm not sure to understand your issue.
code since 'env' is not callable.
Why is it a problem?
Wait, it is callable because if you look in the class definition I implemented the call method:
Indeed, you are right. I didn't spot that in your code :-(
But does the following line of code work for you ?
And are you doing this under Windows?
I always get the error "The environment is of type <class 'main.CustomEnv'>, not a Gymnasium environment"
vec_env = make_vec_env(env, n_envs=4, vec_env_cls=SubprocVecEnv)
With a full example it may be clearer (above comment edited):
import gymnasium as gym
from gymnasium import spaces
from stable_baselines3.common.env_util import make_vec_env
from stable_baselines3.common.vec_env import SubprocVecEnv
class CustomEnv(gym.Env):
"""Custom Environment that follows gym interface."""
def __init__(self):
super().__init__()
self.action_space = spaces.Discrete(2)
self.observation_space = spaces.Box(low=-1, high=1, shape=(2,))
def step(self, action):
return self.observation_space.sample(), 0.0, False, False, {}
def reset(self, seed=None, options=None):
return self.observation_space.sample(), {}
if __name__ == "__main__":
vec_env = make_vec_env(CustomEnv, n_envs=4, vec_env_cls=SubprocVecEnv)
When I run your code I get the following error:
Process SpawnProcess-1: Traceback (most recent call last): File "C:\Entwicklung\Anaconda3\lib\multiprocessing\process.py", line 314, in _bootstrap self.run() File "C:\Entwicklung\Anaconda3\lib\multiprocessing\process.py", line 108, in run self._target(*self._args, **self._kwargs) File "C:\Entwicklung\Anaconda3\lib\site-packages\stable_baselines3\common\vec_env\subproc_vec_env.py", line 29, in _worker env = _patch_env(env_fn_wrapper.var()) File "C:\Entwicklung\Anaconda3\lib\site-packages\stable_baselines3\common\env_util.py", line 100, in _init env = _patch_env(env) File "C:\Entwicklung\Anaconda3\lib\site-packages\stable_baselines3\common\vec_env\patch_gym.py", line 33, in _patch_env raise ValueError( ValueError: The environment is of type <class 'main.CustomEnv'>, not a Gymnasium environment. In this case, we expect OpenAI Gym to be installed and the environment to be an OpenAI Gym environment. Traceback (most recent call last): File "C:\Entwicklung\Anaconda3\lib\multiprocessing\connection.py", line 312, in _recv_bytes nread, err = ov.GetOverlappedResult(True) BrokenPipeError: [WinError 109] The pipe has been ended
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "C:\Entwicklung\Anaconda3\lib\multiprocessing\connection.py", line 321, in _recv_bytes raise EOFError EOFError
Process finished with exit code 1
Please provide your system info (see bug issue template)
sb3.get_system_info() gives:
With a full example it may be clearer (above comment edited):
import gymnasium as gym from gymnasium import spaces from stable_baselines3.common.env_util import make_vec_env from stable_baselines3.common.vec_env import SubprocVecEnv class CustomEnv(gym.Env): """Custom Environment that follows gym interface.""" def __init__(self): super().__init__() self.action_space = spaces.Discrete(2) self.observation_space = spaces.Box(low=-1, high=1, shape=(2,)) def step(self, action): return self.observation_space.sample(), 0.0, False, False, {} def reset(self, seed=None, options=None): return self.observation_space.sample(), {} if __name__ == "__main__": vec_env = make_vec_env(CustomEnv, n_envs=4, vec_env_cls=SubprocVecEnv)
This example does not work. You need a call method in the class, otherwise I get this error:
Traceback (most recent call last):
File "/home/ubuntu/workspace/src/simulator/webots/controllers/supervisor_controller/supervisor_controller.py", line 470, in <module>
main()
File "/home/ubuntu/workspace/src/simulator/webots/controllers/supervisor_controller/supervisor_controller.py", line 464, in main
train(model_class, env, model_path, logdir_path, n_max)
File "/home/ubuntu/workspace/src/simulator/webots/controllers/supervisor_controller/supervisor_controller.py", line 368, in train
vec_env = make_vec_env(env, n_envs = 4, vec_env_cls = SubprocVecEnv)
File "/opt/anaconda/anaconda3/envs/dl-gym/lib/python3.10/site-packages/stable_baselines3/common/env_util.py", line 125, in make_vec_env
vec_env = vec_env_cls([make_env(i + start_index) for i in range(n_envs)], **vec_env_kwargs)
File "/opt/anaconda/anaconda3/envs/dl-gym/lib/python3.10/site-packages/stable_baselines3/common/vec_env/subproc_vec_env.py", line 114, in __init__
process.start()
File "/opt/anaconda/anaconda3/envs/dl-gym/lib/python3.10/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/opt/anaconda/anaconda3/envs/dl-gym/lib/python3.10/multiprocessing/context.py", line 300, in _Popen
return Popen(process_obj)
File "/opt/anaconda/anaconda3/envs/dl-gym/lib/python3.10/multiprocessing/popen_forkserver.py", line 35, in __init__
super().__init__(process_obj)
File "/opt/anaconda/anaconda3/envs/dl-gym/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/opt/anaconda/anaconda3/envs/dl-gym/lib/python3.10/multiprocessing/popen_forkserver.py", line 47, in _launch
reduction.dump(process_obj, buf)
File "/opt/anaconda/anaconda3/envs/dl-gym/lib/python3.10/multiprocessing/reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
File "/opt/anaconda/anaconda3/envs/dl-gym/lib/python3.10/site-packages/stable_baselines3/common/vec_env/base_vec_env.py", line 479, in __getstate__
return cloudpickle.dumps(self.var)
File "/opt/anaconda/anaconda3/envs/dl-gym/lib/python3.10/site-packages/cloudpickle/cloudpickle.py", line 1479, in dumps
cp.dump(obj)
File "/opt/anaconda/anaconda3/envs/dl-gym/lib/python3.10/site-packages/cloudpickle/cloudpickle.py", line 1245, in dump
return super().dump(obj)
ValueError: ctypes objects containing pointers cannot be pickled
You need a call method in the class, otherwise I get this error:
No we don't. gym env aren't meant to be callable
When I run your code I get the following error:
Can you double-check that you've actually ran the example that I gave?
What the following outputs?:
import gymnasium as gym
from gymnasium import spaces
from stable_baselines3.common.env_util import make_vec_env
from stable_baselines3.common.vec_env import SubprocVecEnv
class CustomEnv(gym.Env):
"""Custom Environment that follows gym interface."""
def __init__(self):
super().__init__()
self.action_space = spaces.Discrete(2)
self.observation_space = spaces.Box(low=-1, high=1, shape=(2,))
def step(self, action):
return self.observation_space.sample(), 0.0, False, False, {}
def reset(self, seed=None, options=None):
return self.observation_space.sample(), {}
if __name__ == "__main__":
env = CustomEnv()
print(isinstance(env, gym.Env))
Can you double-check that you've actually ran the example that I gave?
That was surely your example! And now "print(isinstance(env, gym.Env))" gives True, as it should be.
However, as I mentioned in my first post "isinstance(env, gym.Env)" inside _patch_env() is then False. Your code behaves exactly as my own code.
To summarize, this is what you're encountering?:
import gymnasium as gym
from gymnasium import spaces
from stable_baselines3.common.env_util import _patch_env
class CustomEnv(gym.Env):
def __init__(self):
super().__init__()
self.action_space = spaces.Discrete(2)
self.observation_space = spaces.Box(low=-1, high=1, shape=(2,))
def step(self, action):
return self.observation_space.sample(), 0.0, False, False, {}
def reset(self, seed=None, options=None):
return self.observation_space.sample(), {}
if __name__ == "__main__":
env = CustomEnv()
print(isinstance(env, gym.Env)) # True
_patch_env(env) # Fails The environment is of type <class 'main.CustomEnv'>, not a Gymnasium ...
Do you also have the issue with DummyVecEnv
? (default vec_env_cls
)
make_vec_env(CustomEnv, n_envs=4)
The direct call to _patch_env(env) does NOT give an error, only when it happens inside make_vec_env(). I guess it has something to do with the fact that it is then called from a spawned process!?
make_vec_env() with DummyVecEnv works fine. And as I understand that also does NOT involve creating a new process.
P.S. I have to go now. Will check back tomorrow.
From my side:
without the __call__
method, no matter what I do, I get the following error:
TypeError: 'CustomEnv' object is not callable
By the way the function:
env = CustomEnv()
print(isinstance(env, gym.Env))
returns True
@wilhem @AxKo It's getting pretty complicated to figure out who's getting what error. So to sum up, the problem is that under Windows, you get:
from gymnasium import spaces
import gymnasium as gym
from stable_baselines3.common.env_util import _patch_env, make_vec_env, SubprocVecEnv
class CustomEnv(gym.Env):
def __init__(self):
super().__init__()
self.action_space = spaces.Discrete(2)
self.observation_space = spaces.Box(low=-1, high=1, shape=(2,))
def step(self, action):
return self.observation_space.sample(), 0.0, False, False, {}
def reset(self, seed=None, options=None):
return self.observation_space.sample(), {}
if __name__ == "__main__":
print(isinstance(CustomEnv(), gym.Env)) # True
_patch_env(CustomEnv()) # OK
make_vec_env(CustomEnv, n_envs=4) # OK
make_vec_env(CustomEnv, n_envs=4, vec_env_cls=SubprocVecEnv) # Fails
Traceback:
ValueError: The environment is of type <class 'main.CustomEnv'>, not a Gymnasium environment. In this case, we expect
System info:
OS: Windows-10-10.0.22631-SP0 10.0.22631 Python: 3.10.9 Stable-Baselines3: 2.3.2 PyTorch: 2.3.0+cpu GPU Enabled: False Numpy: 1.23.5 Cloudpickle: 2.0.0 Gymnasium: 0.28.1 OpenAI Gym: 0.23.1
Can you confirm?
From now on, if your comment doesn't relate to this issue precisely, please open a new issue.
For some unknown reasons it works, if the environment is registered, like here:
from gymnasium.envs.registration import register
# Example for the CartPole environment
register(
# unique identifier for the env `name-version`
id="CartPole-v1",
# path to the class for creating the env
# Note: entry_point also accept a class as input (and not only a string)
entry_point="gym.envs.classic_control:CartPoleEnv",
# Max number of steps per episode, using a `TimeLimitWrapper`
max_episode_steps=500,
)
Please try that and remove the __call__
method.
Pay attention to link the class of the enviroment in the entry_point, e.g.:
from gymnasium.envs.registration import register
register(
id="Robot-v1",
entry_point=CustomEnv, # <= Not a string, but the class name!
max_episode_steps=500,
)
Source: link
About my system: I'm using Ubuntu 22.04
So it's not Windows only but Ubuntu as well? I can't reproduce the error on Ubuntu.
@qgallouedec: Yes, your above summary is correct for me.
In addition, using a registered using a registered environment also works for me: make_vec_env("CartPole-v1", n_envs=4, vec_env_cls=SubprocVecEnv) # ok
However, registration does NOT work for me: register(id='CustomEnv-v0', entry_point=CustomEnv) print(gym.envs.registry.keys()) => contains 'CustomEnv-v0' make_vec_env('CustomEnv-v0', n_envs=4, vec_env_cls=SubprocVecEnv) => Error: 'CustomEnv' doesn't exist
P.S. can you actually reproduce my problem under Windows??
Stupid question: is the class CustomEnv
in another file or in the same file, where you register the environment?
The same file! I just added my 3 lines of code to the example code of Quentin.
Can you post your code entirely removing the unnecessary parts?
What happens if you call the class and not only the name of the class?
register(id='CustomEnv-v0', entry_point=CustomEnv()) # <= CustomEnv() instead of CustomEnv
It makes no difference if I use CustomEnv or CustomEnv(). Here the complete code:
import gymnasium as gym
from gymnasium import spaces
from stable_baselines3.common.env_util import make_vec_env
from stable_baselines3.common.vec_env import SubprocVecEnv
from stable_baselines3.common.env_util import _patch_env
from gymnasium.envs.registration import register
class CustomEnv(gym.Env):
"""Custom Environment that follows gym interface."""
def __init__(self):
super().__init__()
self.action_space = spaces.Discrete(2)
self.observation_space = spaces.Box(low=-1, high=1, shape=(2,))
def step(self, action):
return self.observation_space.sample(), 0.0, False, False, {}
def reset(self, seed=None, options=None):
return self.observation_space.sample(), {}
if __name__ == "__main__":
print(isinstance(CustomEnv(), gym.Env)) # True
_patch_env(CustomEnv()) # ok
make_vec_env(CustomEnv, n_envs=4) # ok
make_vec_env(CustomEnv, n_envs=4, vec_env_cls=SubprocVecEnv) # fails
make_vec_env("CartPole-v1", n_envs=4, vec_env_cls=SubprocVecEnv) # ok
register(id='CustomEnv-v0', entry_point=CustomEnv)
print(gym.envs.registry.keys()) # => contains 'CustomEnv-v0'
make_vec_env('CustomEnv-v0', n_envs=4, vec_env_cls=SubprocVecEnv) # Error: 'CustomEnv' doesn't exist
Can you put the registration
of the environment outside the main?
Then please post the error.
import gymnasium as gym
from gymnasium import spaces
from stable_baselines3.common.env_util import make_vec_env
from stable_baselines3.common.vec_env import SubprocVecEnv
from stable_baselines3.common.env_util import _patch_env
from gymnasium.envs.registration import register
class CustomEnv(gym.Env):
"""Custom Environment that follows gym interface."""
def __init__(self):
super().__init__()
self.action_space = spaces.Discrete(2)
self.observation_space = spaces.Box(low=-1, high=1, shape=(2,))
def step(self, action):
return self.observation_space.sample(), 0.0, False, False, {}
def reset(self, seed=None, options=None):
return self.observation_space.sample(), {}
register(id='CustomEnv-v0', entry_point=CustomEnv)
if __name__ == "__main__":
print(isinstance(CustomEnv(), gym.Env)) # True
_patch_env(CustomEnv()) # ok
make_vec_env(CustomEnv, n_envs=4) # ok
make_vec_env(CustomEnv, n_envs=4, vec_env_cls=SubprocVecEnv) # fails
make_vec_env("CartPole-v1", n_envs=4, vec_env_cls=SubprocVecEnv) # ok
print(gym.envs.registry.keys()) # => contains 'CustomEnv-v0'
make_vec_env('CustomEnv-v0', n_envs=4, vec_env_cls=SubprocVecEnv) # Error: 'CustomEnv' doesn't exist
That WORKED !! I guess it is because now the registration is performed by each new process.
I only get this warning about render_mode (which is an unrelated and minor issue): UserWarning: WARN: The environment is being initialised with render_mode='rgb_array' that is not in the possible render_modes ([]).
Thanks !
I don't know... but I think that SB3 is extremely picky compared to other frameworks
Closing as the original question has been answered here: https://github.com/DLR-RM/stable-baselines3/issues/1968#issuecomment-2232592563
That WORKED !! I guess it is because now the registration is performed by each new process.
I only get this warning about render_mode (which is an unrelated and minor issue): UserWarning: WARN: The environment is being initialised with render_mode='rgb_array' that is not in the possible render_modes ([]).
Thanks !
I also have problems about render_mode
UserWarning: WARN: The environment is being initialised with render_mode='rgb_array' that is not in the possible render_modes ([]).
Did you solve it?
this comes from Gymnasium not sb3, please have a look at their doc
That WORKED !! I guess it is because now the registration is performed by each new process. I only get this warning about render_mode (which is an unrelated and minor issue): UserWarning: WARN: The environment is being initialised with render_mode='rgb_array' that is not in the possible render_modes ([]). Thanks !
I also have problems about render_mode
UserWarning: WARN: The environment is being initialised with render_mode='rgb_array' that is not in the possible render_modes ([]).
Did you solve it?
In your custom environment put the following line:
class CustomEnv(gym.Env):
metadata = {"render_modes": ["human", "rgb_array"], "render_fps": 30}
❓ Question
I have a custom environment (inherited from Gymnasium and yes
check_env
runs without any errors or warnings) and now I'm trying to migrate it to a vectorized environment.My question is: since the official documentation shows an example using a standard CartPole-v1 environment and not really a custom class or function, how do I adapt my environment class to be vectorized?
A minimal example is:
The question now is: if I do a
print(type(vec_env))
the output is aDummyVecEnv
Is it correct or I'm doing something wrong?
Checklist