codalab / codabench

Codabench is a flexible, easy-to-use and reproducible benchmarking platform. Check our paper at Patterns Cell Press https://hubs.li/Q01fwRWB0
Apache License 2.0
77 stars 28 forks source link

Auto-migrate leaderboard submissions #1341

Closed ihsaan-ullah closed 7 months ago

ihsaan-ullah commented 9 months ago

@ mention of reviewers

@Didayolo

A brief description of the purpose of the changes contained in this PR.

Previously, all submissions of a phase were migrated to a new phase in the auto-migration.

NOW, only leaderboard submissions will be migrated

Issues this PR resolves

Test

Checklist

Didayolo commented 9 months ago

I think it makes sense that the tests are failing. I guess the tests need to be adapted to match this change.

Didayolo commented 7 months ago

@ihsaan-ullah

I encountered a problem while testing this on the test server.

Here is the test competition: https://codabench-test.lri.fr/competitions/143

Here is what I did:

When the final phase started, the two submissions got run, so the change worked (only leaderboard submissions got run).

In the logs, I can find the scores and everything, so the submissions worked well:

Capture d’écran 2024-04-19 à 13 48 59

However, on the final phase leaderboard, the scores are N/A and the detailed results lead to 404 error:

Capture d’écran 2024-04-19 à 13 49 26
ihsaan-ullah commented 7 months ago

I think this problem is not related to the change I made because we have never tested auto migration. Or maybe you have tested auto migration before?

I am not sure but I think someone has reported this before

Didayolo commented 7 months ago

OK, I had the impression it worked before but I'm not sure. I'll open another issue about it.

Didayolo commented 7 months ago

Indeed, it's here: