Incorrect Results from SMPL Estimations with Monocular Data

[Daaohame]

Hello,

Thanks for your excellent work. I encountered some issues when using my customized dataset, specifically related to the SMPL estimations. I compared SMPL vertices visualizations as shown in the figure above. They are for monocap/lan_images620_1300/images/00/0000.jpg and from various sources:

1. Original MonoCap dataset
2. Multiple-view estimations from EasyMocap (used by MonoCap for SMPL parameter estimation)
3. Monocular estimations from EasyMocap
4. Our customized monocular SMPL estimations

When processed through GauHuman, only 1 and 2 yield correct results, while 3 and 4 consistently produce incorrect outcomes (predicted images are all black). One observed issue is the generation of erroneous 2D bound masks from 3 and 4. My speculation is that this is rooted in the differences in SMPL estimations. As shown in the figure, 1 and 2 are closely aligned, as are 3 and 4. However, the differences between these two groups are significant.

Given these observations, I’m seeking your insights –

• Could the discrepancies between multiple-view and monocular estimations, perhaps in their coordinate systems, be causing these issues?
• I’ve also noticed GauHuman computes SMPL vertices by (1) obtaining the original vertices $V$ from the SMPL models, and (2) $V=VR^T+T$, instead of directly reading the dataset’s vertices. Interestingly, they seem to be identical. Could you explain the rationale?

I am also looking forward to any additional insights that could assist in generating accurate results from monocular SMPL estimations. Thank you for your time and assistance.

[rorrewang]

Hi, @Daaohame @skhu101 I also encountered the same problem. I tried to use easymocap to process monocular video and get the default camera parameters and smpl parameters, but in the training stage, boundmask returned an abnormal shape. Have you found a solution to this problem?Thaks!

x, y, w, h = cv2.boundingRect(bound_mask[0].cpu().numpy().astype(np.uint8))  #x:0 y:0 w:1 h:1

I think this error is probably caused by the wrong camera parameters. My camera parameters come from the default camera parameters generated by easymocap, but I don't know how to solve this error.I want to find a stable way to process monocular custom data with Gauhuman. Can you give me some good suggestions?

[skhu101]

Hi @Daaohame @rorrewang, thanks for your questions. This codebase should work with SMPLs and camera parameters estimated from EasyMocap. Could you check projecting SMPL vertices to 2D images and see whether the transformation relationship is correct? Attached is a file for EasyMocap annotations. in_the_wild.py.zip

[rorrewang]

Hi @Daaohame @rorrewang,感谢您的提问。该代码库应适用于从 EasyMocap 估计的 SMPL 和相机参数。您能否检查将 SMPL 顶点投影到 2D 图像并查看转换关系是否正确？附件是EasyMocap注释的文件。 in_the_wild.py.zip

Amazing! it works! Thank you for your help!

[Daaohame]

Hi @skhu101 @rorrewang, thanks for your help! The issue is resolved by adjusting the camera parameters, as you suggested and similarly discussed in [https://github.com/zju3dv/EasyMocap/issues/365]. The key is to use blank camera parameters that align with your solution.