search

crowdAI / crowdai

Fighting for Open Science with Open Data
https://www.crowdai.org
GNU Affero General Public License v3.0
149 stars 32 forks source link

Initial score #569

Closed FlorianColombo closed 6 years ago

FlorianColombo commented 6 years ago

For new submission, the initial score (before any evaluation) should not appear in the leaderboard

scarroll32 commented 6 years ago

They can be deleted by crowdAI admins ... the reason pre-challenge submissions are accepted and viewable on the leaderboard is for a per-challenge smoke test.

Are there any entries that should be deleted? Paste them in here & I'll take care of it.

FlorianColombo commented 6 years ago

Ah. If there is no automated way, then I guess it's useless... However, we might want to prevent people submitting last minute in order to appear first in the leaderboard. Maybe, by refusing new submission n(=3?) weeks before the challenge ends?

spMohanty commented 6 years ago

Hi Florian, I understand where this issue comes from, but its actually a bit tricky to implement in a general way. I dont think it makes sense at this point to implement this as a site wide feature. But what I can do it try to deal with it at the evaluation API level. But there also there is a risk for some submissions not receiving the threshold number of evaluations and hence a divergence between the internal leaderboard of the evaluation API and the crowdai leaderboard.

Or we can also transparently deal with it by adding a rule that only submissions with N number of evaluations will be considered as a prize. We have one consistent leaderboard for both crowdai and the evaluation api. And at the end of the challenge manually ensure that all submissions have the threshold nimber of evalutions. Even if that means that we ourselves ll have to evaluate them.

Let me know what works for you.

scarroll32 commented 6 years ago

Can this be closed?

scarroll32 commented 6 years ago

Closed due to inactivity.