About the position of pelvis joint

[linjing7]

Hi, thanks for your excellent work. I find that in the function merge_joint(), you set the position of the pelvis joint to be the middle of left hip and right hip joints: pelvis = (joint_img[lhip_idx, :] + joint_img[rhip_idx, :]) * 0.5. But as shown in SMPL-made-simple file, the pelvis joint is not exactly in the middle of hip joints and is a little higher than them. So do we need to manually change the position of pelvis joint?

[mks0601]

Hi, yes you're correct. As I'm using L1 loss, the training is robust to some noisy targets. I'm not sure changing pelvis will have large impacts, but maybe you should try it?

[linjing7]

Hi, the impact is minor when training hand4whole. But when you fit the 3d dataset in NeuralAnnot, I wonder whether the 3d keypoints is root-relative? If yes, all of the 3D keypoints would be wrong since the root (pelvis joint) is not correct and I think it would have a large impact.

[mks0601]

When I fit SMPL/MANO/SMPL-X to 3D datasets, I considered such small deviations when making the joint regression matrix. Also, after the fitting, I align translation using all 3D GT joints (not only with root joint), so I guess NeuralAnnot's fits are safe from this issue.

[linjing7]

Hi, thanks for your timely reply. Do you mean changing the joint regression matrix in SMPLX model? Could you please tell me which file you use?

[mks0601]

No, I made joint regression matrix by adjusting the default joint regression matrix, defined in the pkl files. Which dataset do you want?

[linjing7]

Hi, my 3d keypoints in the format of coco-wholebody. So I need the joint regression matrix corresponding to coco-wholebody.

[mks0601]

I thought you're talking about MoCap datasets like H36M as you said like 3D keypoints. MSCOCO does not provide 3D keypoints and therefore, the fitting from NeuralAnnot are not done in a root-relative way.

[mks0601]

So when fitting NeuralAnnot to MSCOCO, I followed the same pelvis definition of this repo. As performance of Hand4Whole is not affected much by the definition of pelvis, NeuralAnnot's fits are not impacted largely by the definition of pelvis of COCO.

[linjing7]

Hi, sorry for the confusion. I'm fitting a multi-view dataset by:

1. annotating the 134(133+pelvis) 2D keypoints in coco-wholebdody format,
2. performing triangulation and get the 134 3D keypoints,
3. fitting SMPLX to the 3D root-relative keypoints. However, the definition of pelvis (pelvis= lhip_joint*0.5+rhip_joint*0.5) of my 3d keypoints is different from SMPLX (figure below) and leads to an unsatisfying result. Do you have any advice?

[linjing7]

This is the visualization of my body and hand keypoints.

[mks0601]

You can load default joint regression matrix from pkl files and adjust the definition of pelvis by the mean of hips? It seems pelvis in your dataset is an average of two hips.

[linjing7]

Okay, thanks for your advice and I'll have a try.