search

microsoft / MeshTransformer

Research code for CVPR 2021 paper "End-to-End Human Pose and Mesh Reconstruction with Transformers"
https://arxiv.org/abs/2012.09760
MIT License
614 stars 95 forks source link

questions on 3DPW #27

Closed siyuzou closed 2 years ago

siyuzou commented 3 years ago

Hi! Thanks for your great work. I have some questions about training on 3DPW.

  1. According to docs/EXP.md, when train on 3DPW, 3dpw/test_has_gender.yaml is used to evaluate during training:

    --val_yaml 3dpw/test_has_gender.yaml \

    When evaluate on 3DPW after training, 3dpw/test.yaml is used instead:

    --val_yaml 3dpw/test.yaml \

    I only find test_has_gender.yaml in the provided 3dpw.tar archive. Are the above 2 files the same? If not, what are the differences?

  2. According to metro/tools/tsv_demo_3dpw.py#L71 and metro/tools/run_metro_bodymesh.py#L395, neutral SMPL model is used to generate GT kp3d and GT vertices, for both 3DPW train set and testing set. I wonder if it is correct because 3DPW provides gender attributes, and only use the model corresponding to the given gender gives the correct output, as far as I know.

Thanks again and looking forward to your reply!

kevinlin311tw commented 3 years ago

A1: Sorry for the typo. We use test_has_gender.yaml for evaluation.

A2: Good question. I'm not sure about this. We can try to use gender-specific SMPL during testing. In that case, I think our MPVE might be a bit worse since we don't have gender-specific GT in training and we can't learn gender-specific 3D shape. For MPJPE and PA-MPJPE, the results should be similar to the neutral ones.

siyuzou commented 3 years ago

Thanks for your timely reply!

I have visualized some SMPL models of 3DPW (project the 3D mesh on top of the image), using both gender-specific SMPL and neutral SMPL. The results show that using gender-specific SMPL gives the best visualization results (the projected 3D mesh fits with the image). Neutral SMPL gives worse projected results, and using the opposite gender gives the worst results.

This indicate that using neutral SMPL model to train on 3DPW introduces additional bias to the network, further testing on 3DPW with neutral SMPL model may hide this error on those metrics, but the model is actually performing worse because of misusing of data.

Have you tried or considered pre-train the network (on Human3.6M and other datasets) using neutral SMPL only, and fine-tune on 3DPW with gender-specific SMPL (and of course, test with gender-specific SMPL) ? I think this might make the training and testing on 3DPW correct and consistent.

kevinlin311tw commented 3 years ago

Thanks a lot for the suggestions. This would be a reasonable solution. We will definitely try to add gender information for both training and testing.