Hold Cassie in the air

[farzadab]

Fix cassie in air and run:

• [x] run PD and see if there's anything wrong (motion, rewrads, etc)
• [x] train and see how much it improves if at all
  • [x] how does the motion look like?

[farzadab]

Running PD: Observations

• seems like joint speeds are highly noisy. Using a moving average for the PD control.
• realized we are not using a joint velocity reward: using stable joint velocities for that too
• PD already gets 75% on the reward function at this stage (will be changed)

[farzadab]

Running Training: Observations

• strange:
  • it didn't get any better (might be due to small sample size; my fault)

  • but it does a crazy motion and still gets ~74% reward which is bad. Need to make the reward more sensitive

    Update:

• I increased max_episode_steps and decreased decay_gamma and at least the reward is getting better rather than worse.
• As for the crazy motion I made the reward function more strict (higher multiplier before applying exp) and it seems a bit better.
• It gets 360/400 reward in the deterministic case which is pretty good. Still it doesn't follow JVel too well.

Issue: The final motion seems a bit jerky: Cassie in-air3 video.

---

Update2: Fixed the jerkiness issue by using a lower gain (divide by 1.9)

• Commit ID: 4ec08444