Difference bettwenn VectorOptimizer and DexPilotOptimizer

[Smilels]

Hi, thank you for your astonishing work and sharing. I am confused about the difference between VectorOptimizer and DexPilotOptimizer. Which optimizer works better from your experience?

[yzqin]

hi @Smilels

The DexPilotOptimizer is a variant of the VectorOptimizer, inspired by the DexPilot paper. It incorporates a fingertip prior to the original vector retargeting objective. For grasping tasks, the DexPilotOptimizer is recommended. For other tasks, the VectorOptimizer remains a suitable choice.

By the way, thank you for your excellent work on TeachNet, which has been a valuable inspiration for our teleoperation research.

[Smilels]

Hi, Yuzhe, thanks for your time and great explanation. Yes, I see the weight variable in the Optimizer.py.

I still have a question about the rendering in Sapien. Through visualize_hand_object.py, the retargeting results are shown in Sapien. Sometimes, I find that the robot's fingertips overlap with the object, and sometimes, the object is not even touched by the human hand or the robot hand. However, the object is still moving stably with the hands. In the robot hand and the object overlapping cases, how do you keep a stable grasp?

In the untouching cases, how do you prevent the object from falling?

Thanks a lot.

[yzqin]

When performing offline retargeting with an object in hand, position-based retargeting is preferable to vector or DexPilot methods.

The visualize_hand_object.py script disables SAPIEN's physical simulation, functioning solely as a visualizer. Consequently, objects move regardless of contact accuracy.

Physically, directly replaying a retargeted trajectory doesn't produce a valid result. Instead, researchers often use retargeted results as references for motion imitation or as training data for reinforcement learning combined with imitation learning.

[Smilels]

hi there, thanks for your explanation.

With your code, I don't experience an obvious frequency difference between position-based retargeting and vector-based retargeting. Why do you suggest using vector or DexPilot to perform real-time retargeting?

[yzqin]

Hi @Smilels

Great question I refer vector retargeting as real-time retargeting not because it is faster. Compared with position retargeting, which target at handling offline hand-object dataset, vector retargeting is used for online tasks like teleoperation.

The difference between them are not running speed, but target applications.

[yzqin]

The objective of position retargeting and vector retargeting is different. Vector retargeting try to be more intuitive for close-loop control while position retargeting try to keep the finger tip position. It is better to use different retargeting optimizer for different applications.

[Smilels]

thanks a lot. That's clear. Thanks again for your sharing