Define reference scenarios and initiate benchmark of our different approaches

[bowni]

• [x] Definition of scenarios and schemas (Eric): link to editable schema
• [x] Select an "open science" web tool for committing benchmark results (Nathanaël)
• [x] Highlight corrupted data more clearly
• [ ] Run each scenario at least once (Eric + help?)
• [x] Describe reference scenarios in the documentation (Eric)

[bowni]

@RomainGoussault @JustineBoulant with the Flexible Splitter PR #320 ready (pending reviews 😎), and the scenarios defined, I think we are all set to run them at least once. Let me know if you try some of them! I am going to try "The Trio" on CIFAR10 :)

[codecov-io]

Codecov Report

Merging #321 (bb939a1) into master (2e28840) will not change coverage. The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #321   +/-   ##
=======================================
  Coverage   82.63%   82.63%           
=======================================
  Files          13       13           
  Lines        2637     2637           
=======================================
  Hits         2179     2179           
  Misses        458      458           

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 2e28840...bb939a1. Read the comment docs.

[bowni]

Hi everyone, as mentioned this morning in the bootcamp launch I suggest to merge this PR to include the definition of the reference scenarios. We can then open another one in draft status, to add incrementally the first results we get on the reference scenarios. So I put it in review now!

[arthurPignet]

what do you think about adding a reference to the dataset(s) we plan to use ?

[bowni]

what do you think about adding a reference to the dataset(s) we plan to use ?

I don't think this is where the focus should be. If needed, this is mentioned on the schema.