search

ramp-kits / autism

Data Challenge on Autism Spectrum Disorder detection
https://paris-saclay-cds.github.io/autism_challenge/
69 stars 43 forks source link

Will these data be available after the challenge? #12

Open puolival opened 6 years ago

puolival commented 6 years ago

Hi,

I just wanted to ask what will happen to these data after the challenge? Also, is it possible to use these data for other purposes? For example, suppose one comes up with some novel idea during the challenge and would like to publish that (in collaboration with the owners of the data). Would it be possible?

Br, Tuomas

glemaitre commented 6 years ago

I would like to have @GaelVaroquaux, @kegl and @agramfort opinions.

I would think that it could be great to benchmark new submission post-challenge. However, it would require computation power on our side and we need to think about the conditions to to that.

ghost commented 6 years ago

I want to know if I could use these preprocessed data(*.csv files) to do my own research after the challenge? maybe publish some papers with it. Besides, is it possible to get another part of the preprocessed data(not raw data) in the private set after the challenge? hope to have your reply, thx. @glemaitre

kegl commented 6 years ago

I'm not the main person to decide, but my suggestion would be to keep the test data private. We could have a service where researchers could officially register their experiments, submit the code, and receive one private test score before publication, which we would also make public, independently whether the researcher decides to include it or not in the paper.

The plan for this challenge is similar: every participant can select one submission (the default is the one with the highest public score), for which we will publish the test score. This is why we have now this message at the top of each submission:

"By default, all your submissions enter the competition, and the one with the highest public leaderboard score will become your official entry. If you don't want that this submission becomes a candidate, pull it out by pressing the button."

GaelVaroquaux commented 6 years ago

I do believe that the private data cannot be released: part of it is brain imaging data for which we do not have authorisations of making public.