Issues with running mujoco walker 2d

[ruiiu]

Hi, when I run the code python sheeprl.py exp=dreamer_v3 env=mujoco env.id=Walker2d-v4 algo.cnn_keys.encoder=[rgb], the following is the output:

Rank-0: policy_step=788, reward_env_2=0.31653469800949097 Rank-0: policy_step=828, reward_env_2=-1.8613801002502441 Rank-0: policy_step=828, reward_env_3=-0.6631340384483337 Rank-0: policy_step=840, reward_env_0=-3.4890027046203613 Rank-0: policy_step=876, reward_env_3=-4.6154303550720215 Rank-0: policy_step=880, reward_env_1=10.097464561462402 Rank-0: policy_step=888, reward_env_2=-6.006372928619385 Rank-0: policy_step=916, reward_env_0=2.8062071800231934 Rank-0: policy_step=928, reward_env_3=2.518906831741333 Rank-0: policy_step=944, reward_env_1=0.48591500520706177 Rank-0: policy_step=952, reward_env_2=0.014924541115760803 Rank-0: policy_step=964, reward_env_0=2.63313364982605 Rank-0: policy_step=996, reward_env_1=1.226755142211914 Rank-0: policy_step=1020, reward_env_2=1.3471245765686035 Rank-0: policy_step=1024, reward_env_0=-1.6578210592269897 Rank-0: policy_step=1024, reward_env_3=-6.501708507537842

It stuck at policy_step=1024, although no errors. But also no videos saved, no trained checkpoint. I didn't change anything, just clone the repo and ran it.

Btw, what is the difference between the policy step and the environment step in the original dreamer v3 paper? How to convert them?

Thanks very much.

[michele-milesi]

Hi @ruiiu, by default the selected accelerator is the cpu, if you did not change it and you have a GPU on which train your agent, I suggest you to run the following command:

python sheeprl.py exp=dreamer_v3 env=mujoco env.id=Walker2d-v4 algo.cnn_keys.encoder=[rgb] fabric.accelerator=cuda

The difference between policy and environment steps is that the policy step is the number of times the actor selects actions during the environment interaction: at each iteration, the policy steps are incremented by num_envs * world_size. Where num_envs is the number of environments and world_size is the number of devices you are using for training (you can define them with the fabric.devices=<world_size> parameter. Instead, the environment steps are the steps performed by the environments, they can be different from the policy steps because you can set, for example, the action_repeat. The action repeat parameter specifies that every time the actor selects an action, that action is repeated action_repeat times in the environment. For example, let us suppose you are using a device, a single environment for training and the action_repeat = 2. After running 500 policy steps, the number of environment steps will be 1000.

I hope you are now clearer on the difference between policy and environment steps