[Custom Env] Robot flying

[araffin]

See https://twitter.com/araffin2/status/1500779708748869634

Environment: https://github.com/francescovezzi/quadruped_spring (note: the behavior was observed without adding springs, using Cartesian PD), the code is in quadruped_spring/env. It comes from an EPFL course on locomotion.

My student @francescovezzi is working on it and I asked him to check nothing was off with the a1 sim from https://github.com/google-research/motion_imitation

To reproduce:

1. Install env:

   git clone https://github.com/francescovezzi/quadruped_spring && cd quadruped_spring
   git checkout d0682b18e41a592484
   pip install -e .

   (snapshot of the code: https://github.com/francescovezzi/quadruped_spring/tree/d0682b18e41a5924848e1965b62b37f133bb1e67)

2. Configure RL Zoo

   git clone git@github.com:DLR-RM/rl-baselines3-zoo.git && cd rl-baselines3-zoo.git
   git checkout quadruped-spring
   pip install -r requirements.txt  # installs pybullet, SB3, SB3 Contrib (with ARS), gym, ...

3. Download pretrained ARS agent (linear controller with normalization active) https://b2drop.eudat.eu/s/jBfgKfyXePsFBik and place the folder QuadrupedSpring-v0_6 in logs/ars/

4. Run the pre-trained agent, you should be able to see it flying

   python enjoy.py --algo ars --env QuadrupedSpring-v0 -f logs/ --load-best --exp-id 6 -n 10000 --env-kwargs render:True

@francescovezzi as @erwincoumans wrote, could you first check that the commanded torque is not too high? (if that's what you meant by "It can happen when the forces (action) is too large.")

[francescovezzi]

The attached folder contains the code to reproduce the issue without having to install the StableBaseline3 or RL-zoo library. They have been used to get the controller contained in the folder. quadruped_flying.zip

Actually Torque commands are as expected clipped into [-33.5, 33.5] [Nm].

[erwincoumans]

here action is joint angles

self.robot.ApplyAction(init_motor_angles)

and now it became torques:

self.robot.ApplyAction(torques)

is that intentional?

It is best to make sure torques and motor angles don't change dramatically between simulation steps (which are 0.001), apply some filter.

[erwincoumans]

Unfortunately, we cannot help debugging custom environments, so closing the issue.