search

Hilti-Research / hilti-slam-challenge-2022

51 stars 1 forks source link

Submission issue: exp07 to 21 are not evaluated #12

Closed brytsknguyen closed 2 years ago

brytsknguyen commented 2 years ago

Hi Hilti team,

I just submitted my result (VIRAL SLAM) and only exp01, exp02, exp04 are scored. Strangely exp04, exp05, exp06 are scored and tallied with exp01, exp02, exp03. Moreover exp07, expp09 ... exp21 are ignored. Could you please take a look?

IamPhytan commented 2 years ago

Two weeks ago, I asked the same question in #8. It looks like it takes more time than expected to process the groundtruth for the Sheldonian and the corridor

hilti-challenge commented 2 years ago

sorry for the late reply, but we decided not to give out evaluation results for the mentioned sequences to avoid too extensive algorithm tuning.

IamPhytan commented 2 years ago

Thank you for the reply. The evaluation website says that only the latest submission with the highest score will count on the leaderboard

Yet, since you said that the last exp will not be evaluated live, how will these files be included in the scoring ? For now, no matter if we include these files or not, the submissions are virtually the same and we have no real feedback on whether or not the exp07-21 files were submitted

hilti-challenge commented 2 years ago

The final score will consist of 8 datasets: exp01,exp02,exp03,exp07, exp09,exp11,exp15 and exp21.

We do not publish the results on exp07-21 to avoid too extensive tuning. If those files are in the subbmission zip they will be taken into account for the final score.

YoshuaNava commented 2 years ago

I'm aware that the final results will be made public on the 23rd of May for the ICRA workshop. Is this also the date that we can get to know individual results for exp07-21? Or can we have access to them earlier?

Thank you.

hilti-challenge commented 2 years ago

you will recieved an email hopefully today with the full results.