Question about training in parallel

[LainWeigh]

Hi @eleurent ! Thank you for your great work!

I have a question. When I set up an environment in highway-env and use Stable-Baselines3 for training, I find that it could only run one environment training at a time. Is there a way to set up multiple environments (for example, 16) for parallel training so that the new number of training steps is 1/16 of the original number of training steps? It would be better if you could give the code using the example follow.

import highway_env
from stable_baselines3 import DQN

env = gym.make("highway-fast-v0")
model = DQN('MlpPolicy', env,
              policy_kwargs=dict(net_arch=[256, 256]),
              learning_rate=5e-4,
              buffer_size=15000,
              learning_starts=200,
              batch_size=32,
              gamma=0.8,
              train_freq=1,
              gradient_steps=1,
              target_update_interval=50,
              verbose=1,
              tensorboard_log="highway_dqn/")
model.learn(int(2e4))
model.save("highway_dqn/model")

# Load and test saved model
model = DQN.load("highway_dqn/model")
while True:
  done = truncated = False
  obs, info = env.reset()
  while not (done or truncated):
    action, _states = model.predict(obs, deterministic=True)
    obs, reward, done, truncated, info = env.step(action)
    env.render()

Thanks!

[eleurent]

Hi!

It's definitely possible, you should look into SB3's VecEnv see documentation, they have some examples. Essentially you have to use make_vec_env, and make sure you are using the SubprocVecEnv (uses multiprocessing), not DummyVecEnv (doesnt).

See also some example HighwayEnv scripts that use multiprocessing. e.g.: https://github.com/Farama-Foundation/HighwayEnv/blob/81c11d6ff50a4fecf05a8341edaecfc245924e0b/scripts/sb3_highway_ppo.py#L17

[LainWeigh]

Thank you for your kindness reply! With the methods and examples you gave, I have completed parallelization training. But I still have a question to ask. Can I first set the config in the main function like using DummyVecEnv, and then update the environment using a method similar to env.configure(config)? I tried to use the following code

if __name__ == '__main__':
    config = {
    # my config
        }

    num_cpu = 4  
    env = make_vec_env('customizedparking-v0', n_envs=num_cpu, vec_env_cls=SubprocVecEnv)
    env.configure(config)

but got an error.

AttributeError: 'SubprocVecEnv' object has no attribute 'configure'

I would be grateful if I could get another reply from you.