search

zc-alexfan / hold

[CVPR 2024✨Highlight] Official repository for HOLD, the first method that jointly reconstructs articulated hands and objects from monocular videos without assuming a pre-scanned object template and 3D hand-object training data.
https://zc-alexfan.github.io/hold
MIT License
303 stars 7 forks source link

Evaluation server submission #19

Closed zc-alexfan closed 2 months ago

zc-alexfan commented 3 months ago

This post will collect any issues related to the evaluation server.

VioletEvar commented 2 months ago

Hi, @zc-alexfan ! I've followed the instructions in the https://github.com/zc-alexfan/hold/blob/master/docs/arctic.md and created arctic_preds.zip after extracting predictions for all nine given sequences. However, when I tried to submit arctic_preds.zip, I noticed that there's no "Bi-CAIR" task to pick on https://arctic-leaderboard.is.tuebingen.mpg.de/algorithms/37/upload page, and when using MR-S, MR-E, IFE-S, IFE-E evaluations, it doesn't work. After the error above, I've tried to upload trial files given in https://github.com/zc-alexfan/arctic/blob/master/docs/leaderboard.md, and they work fine. Therefore, I was wondering if the server is updated for Bi-CAIR task? Looking forward to your reply.

zc-alexfan commented 2 months ago

hi,let me double check with our engineer to push this asap.

zc-alexfan commented 2 months ago

Dear HOLD users,

Our evaluation server is now online. To avoid any last-minute technical issues, please do a test submission ASAP on the server (evaluation could takes up to 10-20 minutes). Due to time zone differences, we do not guarantee on-time responses during last-minute submissions.

By default, there are 10 submissions allowed per month. However, for this competition, we allow additional submissions but please email me (zicong.fan@inf.ethz.ch) if you ran out of submission quotas.

Once you submitted the results, you can make the results public if you wish, but it is not mandatory.

Metric: We use hand-relative chamfer distance, CD_h as the main metric for this competition.

Important:

For more details about the competition, please refer to https://hands-workshop.org/challenge2024.html#challenge2

For details about submission on the server, refer to: https://github.com/zc-alexfan/hold/blob/master/docs/arctic.md

Have fun! Alex