Information regarding automatic evaluation on the evaluation server

[KhanhTungTran]

Hi everyone, here are some information and important notes for evaluating on the server:

Result submission Follow https://votchallenge.net/howto/overview.html for how to create your submission. Do not forget to pack the results with the vot pack command. Make sure that in the tracker identifier in the manifest.yml (by default is inside the output zip file) match with the tracker short name you register through our Google Form. Then submit your zip file in Participate tab at: AIHUB.ML - Competition.

Important notes

• Successful submission will result in a FINISHED message, and you will receive an email about the results of submission shortly after.
• For each submission, the evaluation will run for roughly 45 minutes to 1 hours and 30 minutes, sometimes it can take even longer depends on server loads. To avoid bottlenecking the server, you should try to submit earlier, especially when the deadline is close.
• After 10 successful submissions (meaning necessary files are submitted, input structure of your zip file is correct, etc) you will not receive evaluation results for any of your submission after that.
• Each successful submission needs to be at least 24h apart from each other to mitigate overfitting attempts.
• Each submission that resulted in an error will not be count toward the limit of 10 successful submissions.

Feels free to ask any questions, or you can contact: aihub.admin@openfactor.org for questions specific to the evaluation server.

[KhanhTungTran]

Update: You can see something like this on the submission page:

Please note that this counts both failed and successful submissions. The number of successful submission left for each tracker is sent with the results email.