SMPL overlay can't align well.

[IsshikiHugh]

Hi, thank you for your great dataset and the toolkit. I have some problem similar to https://github.com/sha2nkt/moyo_toolkit/issues/5#issue-1952364873, but since the issue is closed without any follow-up, I create a new issue.

I have go through your toolkits and try to render SMPL mesh over the image.

In order to get a relatively good results, I do these things:

1. For camera R, t, it seems like t is defined in mm while SMPL is defined in m. So I need do t /= 1000 before apply the T_w2c transformation.
2. The image is downsampled, so I tried both resize image to 2w, 2h and make f, cx, cy to their half.
3. In order to align the frame id, I get the smallest image name id. (e.g., the first frame is 4 annotated in the image file name). Then I let offset = 4. Than I get SMPL g.t. in 60fps, like. Than the k-th image in one folder is aligned to gt[::2][k+offset].

Then, I get results like this:

220926_yogi_body_hands_03596_Supta_Baddha_Konasana_-a/YOGI_Cam_02

My questions are:

1. Is this overlay shown in figure above is expected?
2. I haven't try to use SMPL-X g.t., because it's not convenient in my situation. But if this is caused by SMPL-x and SMPL difference, do you have some SMPL-X render results that have higher quality than this?
3. I think the problem might happens on transl, since mesh results in the figure is both off the position and smaller. I simply use the transl provided in the g.t. pkl file. If that's something wrong with me, do you have any suggestion on how to locate the problem?

Thank you very much!

[sha2nkt]

Hi,

Thanks a lot for your interest in our work. Unfortunately, I never tried rendering the SMPL mesh on the image. What I did try was projecting the Mocap markers on the image as described here. As the SMPLX mesh is fit to the same markers and resides in the same coordinate frame, I would expect the mesh to align better than the results you show. To resolve this, I would suggest you to

1. Try projecting the markers on the image as described in the readme.
2. Visualize the markers and the SMPL-X mesh for the same frame in 3D to make sure they align
3. If they align in 3D, you can follow the same projection function for the mesh vertices as I provide for the markers. Doing this should result in perfect alignment.

Let me know if you face any trouble in the above.

[IsshikiHugh]

Hi, Thank you for your reply. I have tried the mask and they aligns well except they are define in mm which is different from SMPL/SMPLX's space. Now I tried to render the SMPLX results, and it seems like it's better:

I also compare the 3D mesh and I found that:

The translation and the 'scale' is different. So my conclusion is that maybe the SMPL g.t. does have a low quality.

My additional question is:

• I found that there is only one v_template provided in my downloaded dataset, does that means there is only one v_template and every sequence use that template?

Thank you!

[sha2nkt]

Hi,

Thanks for the update! The SMPL-X fit indeed looks better aligned with the image.

As we noted in the README, our SMPL fits are not technically ground-truth. They were obtained by converting our ground-truth SMPL-X fits to SMPL using the conversion script here. Since the conversion is an optimisation, it is prone to fitting errors and may not be as good as the original SMPL-X fits.

For your last question, yes, since there is only one subject in the dataset, so we provide only one v_template. You can use the same v_template for every sequence.

[sha2nkt]

We are working on getting better SMPL fits and will release it soon. I will update here when the release happens :)