How to convert mano 3d keypoints into canonical pose?

[Qingcsai]

Hi! Thanks for your great work! I want to know how to convert my 3d keypoints( from mano) to canonical pose? Or could you tell me which function in this repo is the implementation of B ^{-1} in the paper of equation 1 as the figure below?

I tried transform_to_canonical() in halo/converter.py but that didn't work. I am not sure if I was wrong. Could you help me ?

Thanks for your awesome job again!

[korrawe]

Hi Qingcsai,

B^(-1) is implemented in the forward pass of the PoseConverter here. The function will give you matrices that, when applied to the key points, will convert it to the canonical pose. Each matrix that composes B^(-1) is one function call. The documentation is quite messy at the moment, which I will try to improve when I have more time.

The transform_to_canonical() only undo the global translation and rotation of the input before the whole process begins.

One thing I want to point out is that our canonical pose is defined differently than that of MANO (the bone is straight in our definition). Our hand surface is learned with this canonical pose, but if you want to convert to MANO canonical pose you might need to do another conversion at the end.

[Qingcsai]

Thanks for your reply! I will try it soon.

[Qingcsai]

Hi Korrawe! I use transform_to_canonical() before the whole process begins, and then apply trans_mat which is returned from class PoseConverter to get canonical pose , but the result is not ideal like below:

The origin mano pose:

After canonicalization pose:

The processing codes are presented as below.

In fact, I also useconvert_joints(target_js, source='mano', target='biomech') before and convert_joints(target_js_after_vis_tras, source='biomech', target='mano') after canonicalization, could you tell me is this step neccessary? Or is there any step I did wrong?

Thank you!

# vis joint before process
plot_skeleton_single_view(target_js[0], joint_order='mano', color='r', show=True)

target_js = convert_joints(target_js, source='mano', target='biomech')

target_js_after, normalization_mat = transform_to_canonical(target_js, torch.zeros(target_js.shape[0], device=device))

pose_normalizer = PoseConverter(straight_hand=False)
trans_mat, _ = pose_normalizer(target_js_after, torch.zeros(target_js.shape[0], device=device), return_rot_only=False)

trans_mat = trans_mat.numpy()

target_js_after_vis = np.concatenate((target_js_after, np.ones((target_js_after.shape[0],21,1))), axis =-1)
target_js_after_vis = target_js_after_vis[..., None]
# print('target_js_after_vis', target_js_after_vis.shape) # (1, 21, 4, 1)

target_js_after_vis_tras = np.matmul(trans_mat[:,:,:3,:], target_js_after_vis)
target_js_after_vis_tras = np.squeeze(target_js_after_vis_tras, axis=3)
# print('target_js_after_vis_tras', target_js_after_vis_tras.shape) #  (1, 21, 3)

target_js_after_vis_tras = convert_joints(target_js_after_vis_tras, source='biomech', target='mano')

plot_skeleton_single_view(target_js_after_vis_tras[0], joint_order='mano', color='r', show=True)

[Qingcsai]

Hi @korrawe ! I have some changes in the code, and apply canonicalization in a single right hand pose, here are the results:

• the original mano pose

• some visualize result during process which is uncommented from your source code in PoseConverter

• the result from local_coords_after_unpose

• this is the final canonicalized pose from diffrent view:

It seems better than the left hand one which I tried before, but why this happens?

And the result from local_coords_after_unpose seems like the final result I want to get, but the final result is different, did I miss something important?

Sorry for bothering you again~ Could you help me?

[korrawe]

Hi, The code only works for the right hand because of the assumption about indexing. For the left hand, you can flip the skeleton along any axis, then use it as if it is a right hand, then flip the final result again.

If you don't care about the transformation matrices then you can just extract the result from local_coords_after_unpose. I might not have time to look into the final canonicalized pose at the moment but the demo_kps_to_hand.py includes these steps inside so it might help.

[Qingcsai]

Thank you! I will close this issue.