nimble_to_mano function can not generate hand pose in pca

[delaprada]

Hi!

I try to use nimble as a differentiable layer to generate 3D hand model with Freihand dataset. I use the nimble_to_mano function as you kindly suggested in another issue to generate mano vertices from nimble vertices and use the mano joint regressor to obtain the joints from vertices in order to calculate 3D joint loss.

But I found that when I transform nimble to mano using nimble_to_mano function, the generated hand is not in PCA, whose pose is quite weird:

3D joints:

Project 3D joints to 2D:

Does the use_pose_pca=True still work when I transform nimble to mano using nimble_to_mano?

Or maybe it is because my code is wrong?

Here is my code to transform nimble to mano:

nlayer = NIMBLELayer(pm_dict, tex_dict, device, use_pose_pca=True, pose_ncomp=30, shape_ncomp=20, nimble_mano_vreg=nimble_mano_vreg)

skin_v, muscle_v, bone_v, bone_joints, tex_img = nlayer.forward(new_theta, new_beta, new_texture, handle_collision=False)

nimble_to_mano = nlayer.nimble_to_mano

nimble_mano_v = torch.div(nimble_to_mano(verts=skin_v), 1000).to(device) # scale to meter

th_J_regressor = torch.Tensor(np.array(self.smpl_data['J_regressor'].toarray())).to(device) # mano joint regressor
th_j = torch.matmul(th_J_regressor, nimble_mano_v) # get 16 joints
tips = nimble_mano_v[:, [745, 317, 444, 556, 673]] # get tips from mesh vertices

th_j = torch.cat([th_j, tips], 1)
th_j = th_j[:, [0, 13, 14, 15, 16, 1, 2, 3, 17, 4, 5, 6, 18, 10, 11, 12, 19, 7, 8, 9, 20]]  # reorder joints

# th_j is the final joint representation with shape [batch_size, 21, 3]

Thank you very much!

[reyuwei]

Maybe you can try setting is_surface to True, like this line.

nimble_to_mano computes mano vertices from nimble surface points, use_pose_pca=True shouldn't affect it.

[delaprada]

Maybe you can try setting is_surface to True, like this line.

nimble_to_mano computes mano vertices from nimble surface points, use_pose_pca=True shouldn't affect it.

Thanks a lot!

I tried your suggestion and I found that maybe it is because of my training strategy. Using 3D joint normalized loss to train nimble_layer is better than using 3D joint loss.