search

NeuroTechX / moabb

Mother of All BCI Benchmarks
https://neurotechx.github.io/moabb/
BSD 3-Clause "New" or "Revised" License
697 stars 177 forks source link

Data quality level #276

Open jsosulski opened 2 years ago

jsosulski commented 2 years ago

We could add a data quality level either to BaseDataset or to Evaluation. The idea would be something like:

Level 0 (All data): Load everything that is defined in the dataset. Level 1 (Sane data): Load data where the basic information corresponds to the description in the documentation/paper. E.g. only load subjects where Target/NonTarget ratios are correct and the number of events etc. is correct. Level 2 (Hand-vetted data): Probably a lot of effort, but in this setting we could omit subjects / sessions where a large number of artifacts is produced e.g. due to eye blinks or faulty electrodes. Level -1 (undefined): New dataset PRs need to be vetted, until that is done we could assign some arbitrary level to these.

Additionally, after a dataset has been checked, e.g. via a sanity check script, it could get a badge / tag on the documentation data checked and loading correctly and other datasets get either data not checked, use at your own risk or there are issues with this dataset, see Github #X

I am open to other suggestions / ideas.

jsosulski commented 2 years ago

Another idea:

Each dataset could have its own issue label, and the badges could then turn green if there are no issues with a dataset, and if there are, directly link to e.g. issues with the query is:issue is:open label:bug label:sosulski2019

sylvchev commented 2 years ago

I really liked the idea of a quality label for the dataset.

IMO the level 0 (all data) is already implemented: this is the dataset.get_data() that returns a nested dict structure with the raw MNE file. You have all events and eeg data converted in MNE format. Level 1 is what we should have when a dataset is merged into MOABB. We could add a -1 level for dataset without sanity check (or to handle the NeiryDaemon case). I understand the idea behind the level 2, but I think this is important to have artefact, faulty electrodes and so, to have some diversity in the datasets.

Il like the badge idea, I'm looking to some way to implement it, it seems even possible to make some dynamic badges. Instead of having an overall level, could we have a list of badges to specify data quality "metrics":

[Artefact]() [BadChannels]() [ChannelNoise]() [ClassImbalance]()