Research Paper / Slides

[faroit]

Hi

I'd like to know more about how crowdAI works behind the scenes. I started studying the rails code. However, is there a research paper or presentation of the overall system design?

I am especially interested in

• the way the custom evaluation metrics are proposed
• how are the computations carried out? Does crowd AI run participants code?
• how is private data for scoring handled?
• is the full system open source or is there a closed part?

[marcelsalathe]

Hi faroit

There is no paper. To answer your questions, evaluation metrics are developed in collaboration with challenge organizers. How the computations are carried out also depends on the specific challenge, but yes, we can run participant's code. We use an evaluation infrastructure for that which we haven't open sourced. Private data for scoring is handled also on a case by case basis - every challenge is somewhat different.

Btw, crowdAI is not maintained anymore, we have created AIcrowd as a new entity and are developing there. The frontend will remain open source, and the backend will remain private.