Questions about finetuning orig_cliff on 3dpw

[CptDiaos]

Thanks for sharing this great work! I've got a few questions when strating to reproduce the orig_cliff's result on 3dpw using the code and annotations you guys provided. 1)According to your paper, the cliff_3dpw_ft.ckpt was obtained via finetuning the weights trained on real dataset(h36m, coco, mpii, mpi-3dhp), is that correct? And if so , do you finetune the weights on 3dpw-only data or mixed real-data with 3dpw?

2)Is the 3dpw annotation files for finetuning orig_cliff the same as the one for finetuning bedlam? According to your code, you use train/dataset/dataset_smpl.py as the dataloader for real data training, where it reads 2d keypoints from ['part'] and ['openpose'] npz keys. However, the 3dpw annotation file you provide only has ['gtkps'] key, which is in coco format and read by train/dataset/datasetx.py . Do you use the converted 24-kp smpl format from coco formatto implement the supervision on 3dpw?

Sorry to bother! And my great thanks again!

[pixelite1201]

Hello,

1. Yes, we finetunue the weights trained on real datasets (h36m, coco, mpii, mpi-3dhp) with mixed real-data with 3DPW.
2. We use the coco format keypoint for training. The annotation file is saved in data/real_training_labels/3dpw_train.npz

We have also uploaded the config file used in finetuning in configs/orig_cliff_3dpw_ft.yaml. You could checkout the readme to see how to run finetuning.

[CptDiaos]

Thank you for your replies for my questions and the config file! As for the second question, actually what confuses me is how you convert the original 18 coco keypoints stored in ['gtkps] of 3dpw_train.npz into the 49-keypoint-supervision in the real dataset training, as the dataloader train/dataset/dataset_smpl.py doesn't seem to be able to load data stored in ['gtkps']. While in the other dataloader train/dataset/datasetx.py where ['gtkps'] is loaded, the code convert the 18 coco keypoints into 24 smpl keypoints via mapping, which to my knowledge is not an accurate subset of the 49 keypoints either. Maybe the 18->49 part of code is missing? Here is the joints num for 2D supervision for real data in BEDLAM code: https://github.com/pixelite1201/BEDLAM/blob/7e21f75e17c4140b3bc3e3ab99d0673f6f8d9120/train/losses/losses.py#L25C9-L28

[pixelite1201]

Thank you for your replies for my questions and the config file! As for the second question, actually what confuses me is how you convert the original 18 coco keypoints stored in ['gtkps] of 3dpw_train.npz into the 49-keypoint-supervision in the real dataset training, as the dataloader train/dataset/dataset_smpl.py doesn't seem to be able to load data stored in ['gtkps']. While in the other dataloader train/dataset/datasetx.py where ['gtkps'] is loaded, the code convert the 18 coco keypoints into 24 smpl keypoints via mapping, which to my knowledge is not an accurate subset of the 49 keypoints either. Maybe the 18->49 part of code is missing? Here is the joints num for 2D supervision for real data in BEDLAM code: https://github.com/pixelite1201/BEDLAM/blob/7e21f75e17c4140b3bc3e3ab99d0673f6f8d9120/train/losses/losses.py#L25C9-L28

Yes, you are correct. The release 3dpw_train.npz file is incorrect. It is the file used for training BEDLAM-CLIFF and not original CLIFF. We have updated the correct file on the server that contains openpose keypoints. Please download it again. And thanks a lot for testing it out.

[CptDiaos]

Thank you for your patience and the just-in-time check! I‘ve tested the new annotation file and my question is resolved. You guys really did a great job!