Data format of AGORA test result

[linjing7]

Hi, thanks for your excellent work.

I test Hand4Whole on AGORA test set and run check_pred_format.py from AGORA github to verify that the prediction results are in correct format. However, an error occurs as: joints should be of shape (24,2) but you are providing (1, 137, 2). I note that the save format of joints in AGORA.py is as follow: 'joints': joint_proj.reshape(1, -1, 2). Can I modify it to'joints': joint_proj.reshape(-1, 2)[1:1+24, :] to fix the bug?

[mks0601]

That is weird. Are you testing on body-only test set?

[linjing7]

Hi, I test Hand4Whole on AGORA whole-body dataset. You can check the bug in this line. It seems that AGORA only use 24 joints for person matching. BTW, can I directly submit the test result saved in AGORA.py into official website to test the performance? Is the data format correct?

[linjing7]

I test with the pretrained model provided in github. The result is quite curious and the F1 score is pretty low. I think there maybe something wrong with the 'joint' I save in AGORA.py since it is used for person matching.

[mks0601]

I think the AGORA team changed the eval codes.. Is there any clarification which 24 joints should be used?

[linjing7]

Hi, I check the evaluation code of AGORA. The shape of gt joints is (127, 2) and the first 24 joints are used for matching.

[mks0601]

Then could you try 'joints': joint_proj.reshape(-1, 2)[:24, :]?

[linjing7]

Hi, I have tried it. But unfortunately, the result keeps the same. I check the gt joints in the evaluation code and find it is the same as gt_joints_2d from the AGORA label.

[mks0601]

In your previous answer, you used [1:1+24]. Have you tried [0:0:24]?

[linjing7]

Yes, I have tried both cases, but the result keeps the same. The evaluation code uses IoU of bbox calculated from the 24 kpts for person matching, and I think it would not be greatly influenced by [1:1+24] or [0:0:24].

Sorry for troubling you and I will check the bug by myself. If I found the solution, I'll update it here.

[mks0601]

The current codes are the very ones that I used to upload results to AGORA evaluation benchmark, so I think current codes might work. I understand that AGORA dataset is kind of complicated, and required format for the evaluation is not clear :(

[linjing7]

Okay, thanks for your reply, I'll double-check my data processing of AGORA dataset and try again.

[linjing7]

@mks0601 Hi, could you please upload the AGORA_test_bbox.json again? The google link is invalid now.

[mks0601]

Could you try after deleting caches? I recently changed url, and it works for me

[linjing7]

Hi, I have deleted caches, but it does not work. Do you mean this link?

[mks0601]

Ah I meant this link: https://drive.google.com/drive/folders/18CWsL28e8v50rqEbYMoU4yHHWoGJdpg_?usp=sharing , which is from here: https://github.com/mks0601/Hand4Whole_RELEASE#data

[mks0601]

I fixed that link, too

[linjing7]

Hi, thank you very much. The problem is solved and I can correctly test on AGORA test set with the original code :)

[mks0601]

Good for you! What was the problem?

[linjing7]

Hi, @mks0601, I think there may be something wrong during my data processing of AGORA. I recollect AGORA and its labels following the instruction again and the mistake disappears.

BTW, we don't need to run check_pred_format.py from AGORA github to check the data format. I directly submit the result saved by AGORA.py and can obtain a correct result.

Thank you again for your patient reply!

[mks0601]

Got it!