Investigate how to integrate the bootLadderBoot to the NLP Sandbox

[tschaffter]

@yy6linda I would like to investigate if we could integrate the bootLadderBoot to the NLP Sandbox. The bootLadderBoot implements a strategy developed by Elias Chaibub at Sage (@echaibub) to limit the over-fitting of solutions but limiting the amount of information returned to the submitter. In a nutshell, the exact score of a submission is returned to the developer IF the performance is significantly better than the last submission whose score was returned. I think that's it in a nutshell.

Googling bootLadderBoot returns this result first:

https://rdrr.io/github/Sage-Bionetworks/challengescoring/man/bootLadderBoot.html

This investigation will be useful for the NLP Sandbox as well as the new challenge platform that we develop at Sage.

Task

• Get familiar with the bootLadderBoot
• Contact Elias if you have any question
• Come up with the proposal on how the bootLadderBoot could be integrated to the NLP Sandbox
• Identifies any pain points

[tschaffter]

@yy6linda presented a review of the paper of the bootladderboot method. Next week she will provide additional information regarding how this method could be integrated to the NLP Sandbox.

I put the slide I presented this morning about ladderboot and ladder algorithms to prevent overfitting in DREAM challenges: link to slide

[tschaffter]

Done