search

Skybox-Technologies / csgo-ai-competition

CSGO.ai competition
34 stars 16 forks source link

Submission and Leaderboard #3

Open mlpanda opened 4 years ago

mlpanda commented 4 years ago

Hi, I was wondering how you expect the submission process to be. In Kaggle competitions there is a clear format for an output file with generated predictions, it would be great to have a clear guideline in order to structure our code properly. Furthermore it would add a lot of value with a leaderboard to have an idea of the performance compared to the competition - it benefits both the individuals and the organizers as it will push the best performance throughout the competition.

MiniXC commented 4 years ago

In case this might be helpful, there seems to be a good Julia package that could do most of the heavy lifting for a public leaderboard: https://github.com/mrtzh/Ladder.jl

kristoff3r commented 4 years ago

We are definitely going to give a clear guideline for submissions well in advance, but it is not ready yet.

I like the idea of a leaderboard if it isn't too much hassle to setup. If you know of any other resources or best practices for doing that then please post it here.

MiniXC commented 4 years ago

There seems to be an open-source alternative for Kaggle for this, CodaLab. In their faq they say

Anybody can use the public instance of Codalab competitions for free, as long as he/she agrees with the Terms and conditions. However, if you run a large competition with code submission, we advise you to first contact us. To find out how to host your own competition, go to the competition help page.

I don't think this competition would need code submission (just target labels should be fine) so this could be an option. Apparently it's as easy as uploading a "competition bundle" they describe here. There also is a public instance of what seems to be a wizard for creating that competition bundle from text data: https://chalab.lri.fr

I have never used this platform (or heard of it before) but it looks like a decent alternative to Kaggle.

It also seems to be possible to run your own instance of the platform, should there be any legal concerns: https://github.com/codalab/codalab-competitions/blob/develop/README.md#quick-installation-for-linux

TeaPearce commented 4 years ago

I just submitted something - no idea if the performance was competitive or not though. Would have been great to have had a leaderboard.

What kind of accuracies were people getting? My models seemed to level out around 70%.

MiniXC commented 4 years ago

I decided to not compete because of the lack of a leaderboard and the initial problems with the dataset... Either on its own would have been ok. In combination these problems made me fearful that there might be differences between the private test set and the train dataset that would be impossible to know of by competitors until the competition is over. I hope this is not the case though of course, good luck @TeaPearce