using `warm_start` with position optimizer

[giuliano-97]

Description

I am trying to to retarget the hand motions in DexYCB to a jaw gripper, similar to the panda gripper in dex-urdf. I followed more or less the logic of the position_retargeting example, but because the rest (or "flat") position of the robot gripper is with the gripper pointing upward, the result looks like this:

https://github.com/user-attachments/assets/bad99300-54aa-4fa4-99c9-e10b535e77e7

The "fingertips" of the gripper are more or less tracking the thumb and index finger of the human hand, however the wrist position is completely different, but I'd like it to be more or less aligned with the human hand (if that makes sense?). I figured this might be an issue with the optimizer converging to a solution where source and target wrist are not aligned so I looked at the code in SeqRetargeting in seq_retarget.py and noticed the method warm_start:

    def warm_start(self, wrist_pos: np.ndarray, wrist_orientation: np.ndarray, global_rot: np.array):
        """
        Initialize the wrist joint pose using analytical computation instead of retargeting optimization.
        This function is specifically for position retargeting with the flying robot hand, i.e. has 6D free joint
        You are not expected to use this function for vector retargeting, e.g. when you are working on teleoperation
        Args:
            wrist_pos: position of the hand wrist, typically from human hand pose
            wrist_orientation: orientation of the hand orientation, typically from human hand pose in MANO convention
            global_rot:

        """
       ...

which sounds like what I need. So I tried to use it by passing the position of the first joint of MANO for a given frame and set the other two rotations to identity. like this:

        vertex, joint = compute_hand_geometry([hand pose stuff])
        retargeting.warm_start(
            wrist_pos=joint[0, :],
            wrist_orientation=np.zeros(3),
            global_rot=np.eye(3),
        )

Unfortunately this results in the following error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)

File ~/dev/dex-retargeting/dex_retargeting/seq_retarget.py:74, in SeqRetargeting.warm_start(self, wrist_pos, wrist_orientation, global_rot)
     [71](~/dev/dex-retargeting/dex_retargeting/seq_retarget.py:71) target_wrist_pose[:3, :3] = mat
     [72](~/dev/dex-retargeting/dex_retargeting/seq_retarget.py:72) target_wrist_pose[:3, 3] = wrist_pos
---> [74](~/dev/dex-retargeting/dex_retargeting/seq_retarget.py:74) wrist_link_name = self.optimizer.wrist_link_name
     [75](~/dev/dex-retargeting/dex_retargeting/seq_retarget.py:75) wrist_link_id = self.optimizer.robot.get_link_index(wrist_link_name)
     [76](~/dev/dex-retargeting/dex_retargeting/seq_retarget.py:76) name_list = [
     [77](~/dev/dex-retargeting/dex_retargeting/seq_retarget.py:77)     "dummy_x_translation_joint",
     [78](~/dev/dex-retargeting/dex_retargeting/seq_retarget.py:78)     "dummy_y_translation_joint",
   (...)
     [82](~/dev/dex-retargeting/dex_retargeting/seq_retarget.py:82)     "dummy_z_rotation_joint",
     [83](~/dev/dex-retargeting/dex_retargeting/seq_retarget.py:83) ]

AttributeError: 'PositionOptimizer' object has no attribute 'wrist_link_name'

Looking at the available optimizers in optimizer.py the only one which defined that member variable is the DexPilot optimizer. But the docstring says that method is supposed to be used with position retargeting, not teleop, so at this point I'm a bit confused. Could you give some tips on how to use this method and / or provide an example?

Thank you in advance :)

[giuliano-97]

I looked at it again today with a fresh mind and it doesn't work even with DexPilotOptimizer since the wrist_link_name member variable is not set in the constructor. However, it can be made to work by just explicitly setting in it for the robot in the optimizer inside the retargeting e.g. like this:

config_path = get_default_config_path(ROBOT_NAME, RetargetingType.position, HAND_TYPE)
override = dict(add_dummy_free_joint=True)
config = RetargetingConfig.load_from_file(config_path, override=override)
retargeting = config.build()
retargeting.optimizer.wrist_link_name = "panda_hand"

and then in the position retargeting loop:

    for i in trange(start_frame, num_frame):
        hand_pose_frame = hand_pose[i]
        vertex, joint = compute_hand_geometry(...)

        # Update poses for robot hand
        indices = retargeting.optimizer.target_link_human_indices
        ref_value = joint[indices, :]

        # if warm_start is enable, do the warm start to align the wrist pose
        if warm_start:
            wrist_position = joint[0, :]
            wrist_orientation = rotations.compact_axis_angle_from_matrix(estimate_wrist_orientation(joint))
            retargeting.warm_start(
                wrist_pos=wrist_position,
                wrist_orientation=wrist_orientation,
                global_rot=np.eye(3), # not sure about this one but it works
            )

        qpos = retargeting.retarget(ref_value)[retarget2sapien]
        robot.set_qpos(qpos)

        scene.update_render()

and the result changes to this:

https://github.com/user-attachments/assets/be5539ba-5992-4fa5-921c-46d0ec50fc5a

[yzqin]

Hi @giuliano-97 thanks for the detailed report for the bug in position retargeting and warm start.

The warm start function has been fixed in a80caf59c0e7578a7823cc7ce11b2da8472f0494 with new signatures and this function is used in the examples of position retargeting.

-- Yuzhe

[giuliano-97]

Hi @giuliano-97 thanks for the detailed report for the bug in position retargeting and warm start.

The warm start function has been fixed in a80caf5 with new signatures and this function is used in the examples of position retargeting.

-- Yuzhe

great, just retested master and it seems to work as expected, thanks :)