Cannot recover hand keypoints from mano parameters.

[deyingk]

Hi Zhixuan,

Thanks for your great work and sharing the codes. I'm having some trouble recovering 3D hand keypoints from the given mano parameters. I could recover the hand mesh (778 vertices) perfectly using the mano layer implemented by hassony2. However, the recovered hand keypoints don't match with the provided annotated hand keypoints. Do you have some insights on this? Or any instructions on correctly recovering hand keypoints from given mano parameters? Thanks in advance!

The following is the code snippet I used to reconstruct the hand keypoints and hand mesh.

import torch
from manopth.manolayer import ManoLayer
from manopth import demo
import os

import numpy

anno_dir = "Sample_hand/subject_1/hand/00000004/reconstruction"
r_params = numpy.loadtxt(os.path.join(anno_dir, "mano_params_r.txt"))
r_params = torch.Tensor(r_params).unsqueeze(0)

translation = r_params[:,:3]
pose_params = r_params[:,3:26]
shape_params = r_params[:, 26:]

# MANO layer
ncomps = 20
mano_layer = ManoLayer(
    mano_root='mano/models', use_pca=True, ncomps=ncomps, flat_hand_mean=False)
# Forward pass through MANO layer
mano_hand_verts, mano_hand_joints = mano_layer(pose_params, shape_params)
mano_hand_verts = mano_hand_verts/1000 + translation
mano_hand_joints = mano_hand_joints/1000 + translation

# load annotation
hand_verts = numpy.loadtxt(os.path.join(anno_dir,"vertices_r.txt"))
hand_joints = numpy.loadtxt(os.path.join(anno_dir,"keypoints_r.txt"))

hand_verts = torch.Tensor(hand_verts).unsqueeze(0)
hand_joints = torch.Tensor(hand_joints).unsqueeze(0)

# printing
print(hand_joints - mano_hand_joints)
print(hand_verts - mano_hand_verts)

[zhixuany]

Are you asking how to get the hand keypoints given a MANO hand mesh? If so, the MANO model contains a matrix called J_regressor (16 x 778) which maps 778 MANO vertices to 16 hand keypoints (finger tips excluded). And we use MANO vertices with indices 744, 320, 443, 555, 672 as tip keypoints of thumb, index, middle, ring and little finger respectively. The resulting 21 hand keypoints are organized in an order consistent with that in OpenPose.