Pusher environment with Spring pipeline

[wladekpal]

Hi,

I've been trying to experiment with Pusher env using Spring as a pipeline, and had significant trouble getting anything to train. I investigated a bit and found that there seems to be something wrong with Spring-Pusher combination. For example in the training notebook training goes well for Positional and Generalized pipelines, but for Spring it fails. The graphs below are generated using said notebook:
Spring	Positional	Generalized

I tried to find out what the problem may be and after looking at renders it seems to me that in Spring pipeline there is no friction (or it is very small) between the object and the surface. I also include the video generated with Spring pipeline, that shows this behavior.

https://github.com/google/brax/assets/56890546/a610a239-c97d-4f85-93bd-0cfba70be975

[btaba]

Hi @wladekpal

The pusher model doesn't have contact friction. The object rests on a plane using joint constraints

https://github.com/google/brax/blob/b45760cfa6ba0c3567ae36f988766a8299853c49/brax/envs/assets/pusher.xml#L84-L85

So I'm a bit surprised that the positional pipeline works, but spring doesn't due to "damping" of the velocity.

With that being said, the spring pipeline dampens velocity in the integrator here:

https://github.com/google/brax/blob/b45760cfa6ba0c3567ae36f988766a8299853c49/brax/spring/integrator.py#L48-L49

So maybe you can add vel_damping here to see if that helps?

https://github.com/google/brax/blob/b45760cfa6ba0c3567ae36f988766a8299853c49/brax/envs/assets/pusher.xml#L16

[wladekpal]

Alright, thanks for the response. Generalized backend works fine for my purpose, I just wanted to know if that discrapency in backends is expected. Since it is I will close the issue.

[AwesomeLemon]

The issue should be reopened, as it is a real bug that breaks pusher when the spring backend is used. I encountered exactly the same behaviour as @wladekpal

Adding vel_damping as suggested by @btaba solved the problem, though I had to set its value to -1, despite it being set to either 0 or 1 in the Brax codebase.

Looking at the integrator.py code shows that having vel_damping > 0 leads to anti-damping, as xd_i.vel is multiplied by exp(positive number) > 1. Someone should probably check if vel_damping should indeed be set to 1 (and not -1) in walker2d.xml and hopper.xml

P.S. Hilariously, the hyperparameter optimization procedure that I ran was able to work around the infinite sliding by nudging the object very slightly, so that it would spend the most of the episode next to the goal