Data Used in Protocol 1, 2, and 3

Arthur151 / ROMP

Monocular, One-stage, Regression of Multiple 3D People and their 3D positions & trajectories in camera & global coordinates. ROMP[ICCV21], BEV[CVPR22], TRACE[CVPR2023]

https://www.yusun.work/

Apache License 2.0

1.36k stars 231 forks source link

Data Used in Protocol 1, 2, and 3 #417

Open mkhoshle opened 1 year ago

mkhoshle commented 1 year ago

@Arthur151 Hi, I have been looking into the Tables on comparisons across various models. There are several protocols.

Could you please help me understand the specific datasets used for training and testing in each protocol and what are the differences between them?
based on my understanding protocol2 use 3DPW during train and testing (with *) or only uses it during testing. Is this correct?
Are protocols 1 and 2 the same except that protocol 1 does not use ground truth during inference? I am a bit confused about this.

Thanks in advance,

Arthur151 commented 1 year ago

Hi Mahzad Khoshlessan,

If you are interested in the evaluation protocols on 3DPW used in ROMP, I think I can help.

The protocol 1 is borrowed from 3DPW challenge, ECCV 2020. We don’t use any data from 3DPW for training. We directly evaluate on the entire set (training, valuation, and test set) of 3DPW. Different from other single-person method that need to compute the bounding box using the ground truth 2D poses, ROMP don’t use any ground truth during inference. Especially, we use the evaluation code and ground truth provided by 3DPW challenge for evaluation, which is different from commonly used ground truth from VIBE.

The protocol 2 evaluate only on the test set. And the ground truth and evaluation code are brought from VIBE. Similarly, ROMP doesn’t use any ground truth during inference.

Best, Yu

mkhoshle commented 1 year ago

Hi @Arthur151 ,

Thanks for the feedback. I see. So the datasets that are used across protocols 1 and 2 during the training part are the same. Is this correct?

Training datasets in protocols 1 and 2 are: 3d pose [h36m, mpiinf], pseudo-label 3D dataset [UP], 2d pose [coco, lsp, mpii, AICH]. Is this correct? What is the percentage of each data used during training? In the .yml file we need to provide the percentage of each data that we want to use and I am not sure what they need to be for both protocols 1 and 2.
Where can I find the evaluation code for 3DPW? I could not find anything on the challenge page. Is this the same evaluation code available in the ROMP repository? https://competitions.codalab.org/competitions/24938#results

Thanks in advance,

mkhoshle commented 1 year ago

@Arthur151 I would really appreciate it if you could answer the above questions.

Thanks in advance,