Failed to parallelize learning in mujoco envs but succeeded in classical control envs

[fywu85]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04
• Ray installed from (source or binary): pip install ray==0.5.3
• Ray version: 0.5.3
• Python version: 3.6.6
• Exact command to reproduce: Failed command: python experiments/mutiagent_hrl/tune.py HalfCheetah-v2 LSTMPolicy -n 1 -s 3 Succeeded command: python experiments/mutiagent_hrl/tune.py Pendulum-v0 LSTMPolicy -n 1 -s 3

Describe the problem

We try to use ray to parallelize our runner script. Specifically, we used something below

ray.init(num_cpus=3)
ray.get([run_exp.remote(env, discrete, hp, args.steps, dir_name, i)
             for i in range(args.n_training)])

This works for simple envs in classical control, such as Pendulum-v0, but does not work for all mujoco envs, e.g., HalfCheetah-v2. Specifically, in case of Half-Cheetah-v2, it starts with using ~3 CPUs, but quickly collapses into 1 CPU after 5 seconds or so. On the other hand, tests with Pendulum-v0 use all 3 CPUs, which is what we intend to do.

Source code / logs

The source code is at https://github.com/AboudyKreidieh/h-baselines/. Please use the following steps for installation:

conda env create -f environment.yml
python setup.py develop
pip install mujoco_py  # This may require additional dependencies.

To run the tests, please run:

python experiments/mutiagent_hrl/tune.py HalfCheetah-v2 LSTMPolicy -n 1 -s 3  # Not working
python experiments/mutiagent_hrl/tune.py Pendulum-v0 LSTMPolicy -n 1 -s 3  # Working

To see how we use ray to implement the parallelization, please check:

vim experiments/mutiagent_hrl/lstm_baseline.py

[ericl]

Have you checked it didn't crash or something? Use ray stack to see what is going on in the remote workers.

[zhe-thoughts]

Closing this since it's most likely no longer an issue. But @fywu85 please reopen if still needed. Thanks!