search

kasvii / PMCE

[ICCV 2023] PyTorch Implementation of "Co-Evolution of Pose and Mesh for 3D Human Body Estimation from Video"
https://kasvii.github.io/PMCE
MIT License
142 stars 5 forks source link

Datasets for 3D pose estimation #6

Open USTCWzy opened 7 months ago

USTCWzy commented 7 months ago

Hello! I notice that you provide two scripts for 3d pose estimation networks, human3.6m only or multiple datasets (coco, mpii...), respectively. Would you mind sharing the performance difference between these two configurations (I didn't find any related ablation results in the paper) and which configuration is used in the final checkpoint in the paper?

Thanks very much for your reply!

USTCWzy commented 7 months ago

And what's the difference between the provided checkpoints~(i.e., mesh_mpii3d, mesh_3dpw, mesh_h36m), because you only provide two training scripts for mesh prediction. Which datasets are used for mesh_mpii3d.pth.tar training?

USTCWzy commented 7 months ago

I think I might understand the first question. But I still have two problems:

  1. what are the training datasets for mesh_mpii3d.pth.tar, is it the same as mesh_hm36m which used only one dataset for training, testing, and evaluation.
  2. To train the model for 3dpw, you mentioned you use 5 datasets for training, but for human3.6m you only used one dataset for training and testing. In some views, human3.6m is an in-lab dataset in which the scenes, backgrounds, and postures are limited compared to in-wild datasets. I wonder if training only on human3.6m will show overfitting on human3.6m datasets, archiving good performance but showing poor results in other datasets (e.g., 3dpw)?

Thanks very much for your reply!

USTCWzy commented 7 months ago

And what are the validation datasets for human3.6m and mpii3d training ...

kasvii commented 6 months ago
  1. The 2 configurations are because the different joint sets detected by different 2d pose detectors (CPN for human3.6m and ViTPose for 3DPW and MPII3D), where the 3dpw joint set has 19 joints and the h36m joint set has 17 joints. You can refer to Pose2Mesh we followed for the illustration.
  2. The training datasets for mpii3d are mixed datasets, the same as 3dpw. And it is tested on mpii3d testing dataset. It can directly use the checkpoints of 3dpw to test, thus we did not provide its training script. In other words, you can use the training script of 3dpw to train a mpii3d model.
  3. Human3.6m is trained and tested on its dataset, following the same setting of pose-based methods. But as you mentioned, it may cause overfitting. It's interesting to add more datasets to train, it may be useful for both accuracy and generalization ability.
  4. For human3.6m, following previous methods, such as VIBE, TCMR, our model is trained on 5 subjects (S1, S5, S6, S7, S8) and tested on the other 2 subjects (S9, S11). There is no validation set. For mpii3d, it is only tested by using the checkpoint of 3dpw. Hope this reply helps you!
USTCWzy commented 3 weeks ago

Thanks very much for your reply!