Is cube state perception used from real world?

[animesh-garg]

HI team Great job with this effort.

• How is the cube state inferred or estimated. is the policy running with perception in the loop?
• Is the policy running with simulaiton in the loop using sim state of the cube but applying the same actions on the real object?

[anag004]

Hi Animesh,

Thanks for the questions!

How is the cube state inferred or estimated. is the policy running with perception in the loop?

The cube state is implicitly estimated through proprioception (there are no perception / touch sensors). In particular, the recurrent policy uses the discrepancy between commanded and observed joint angles to infer the presence of an object which allows it to adapt behavior automatically. A simple way to test that adaptation is happening is to disable collisions in sim or remove the cube in real and observe that the fingers collapse on themselves.

Is the policy running with simulaiton in the loop using sim state of the cube but applying the same actions on the real object?

No, there is no simulation in the loop at deployment time. At deployment time, observations to the policy are computed using joint angles on the real hand. These are fed through the policy to obtain actions which are then directly commanded to the motors.

Hope this helps :)