Statistics in MOABB

[sylvchev]

MOABB has a very good statistical methodology for experimental analysis and benchmarking algorithms. It could be a nice addition to reinforce these aspects with:

• a specific example/tutorial, that details the chosen statistics and how to use it.
• improve definition of chance level

Regarding chance level, we simply use the limit when the number of samples tends to infinity (50% for 2 classes, 25% for 4 classes, etc). This chance level is a crude approximation and could be refined, either using the class sample balance (Better than random: a closer look on BCI results) or by characterizing the class sample distribution (Exceeding chance level by chance)

[bruAristimunha]

Hi @sylvchev,

Do you have the biomodal implemented somewhere?

[sylvchev]

Yes, the paper is here, their matlab code is here and my python version is:

def chance_level(nbepoch, nbclass, alpham=0.01):
    # nbepoch = length(y);
    # nbclass =  length(unique(y));
    threhold = binom.ppf(1-alpham, nbepoch, 1/nbclass) * 100/nbepoch
    return threhold

alpham is the target p-value (for example 0.01)