Drop of accuracy for BNCI2014_001 from v0.5.0 to v1.0.0

[gustavohenriquesr]

After upgrading to moabb v1.0.0, the accuracy value dropped when using BNCI2014_001 dataset.

For example, with moabb 0.5.0, using this Braindecode tutorial (Braindecode version 0.8.1), I get about 70% of average accuracy after 100 epochs. However, when using version 1.0.0 of moabb I get about 60%, with the same number of epochs. I can't figure why this is happening, since I'm using the same model and parameters.

I have also tested other models from Braindecode and I ended up having the same problem.

The only differences in the pipeline are:

When using moabb 1.0.0 (as in the tutorial): dataset = MOABBDataset(dataset_name="BNCI2014_001", subject_ids=[subject_id])

train_set = splitted['0train'] # Session train

valid_set = splitted['1test'] # Session evaluation

When using moabb 0.5.0: dataset = MOABBDataset(dataset_name="BNCI2014001", subject_ids=[subject_id])

train_set = splitted['session_T'] # Session train

valid_set = splitted['session_E'] # Session evaluation

Here's the code:

    # Braindecode version: 0.8.1    

    from braindecode.datasets import MOABBDataset

    subject_id = 3

    # MOABB v1.0.0
    dataset = MOABBDataset(dataset_name="BNCI2014_001", subject_ids=[subject_id])

    # MOABB v0.5.0
    #dataset = MOABBDataset(dataset_name="BNCI2014001", subject_ids=[subject_id])

    from numpy import multiply

    from braindecode.preprocessing import (Preprocessor,
                                           exponential_moving_standardize,
                                           preprocess)

    low_cut_hz = 4.  # low cut frequency for filtering
    high_cut_hz = 38.  # high cut frequency for filtering
    # Parameters for exponential moving standardization
    factor_new = 1e-3
    init_block_size = 1000
    # Factor to convert from V to uV
    factor = 1e6

    preprocessors = [
        Preprocessor('pick_types', eeg=True, meg=False, stim=False),  # Keep EEG sensors
        Preprocessor(lambda data: multiply(data, factor)),  # Convert from V to uV
        Preprocessor('filter', l_freq=low_cut_hz, h_freq=high_cut_hz),  # Bandpass filter
        Preprocessor(exponential_moving_standardize,  # Exponential moving standardization
                     factor_new=factor_new, init_block_size=init_block_size)
    ]

    # Transform the data
    preprocess(dataset, preprocessors, n_jobs=-1)

    from braindecode.preprocessing import create_windows_from_events

    trial_start_offset_seconds = -0.5
    # Extract sampling frequency, check that they are same in all datasets
    sfreq = dataset.datasets[0].raw.info['sfreq']
    assert all([ds.raw.info['sfreq'] == sfreq for ds in dataset.datasets])
    # Calculate the trial start offset in samples.
    trial_start_offset_samples = int(trial_start_offset_seconds * sfreq)

    # Create windows using braindecode function for this. It needs parameters to define how
    # trials should be used.
    windows_dataset = create_windows_from_events(
        dataset,
        trial_start_offset_samples=trial_start_offset_samples,
        trial_stop_offset_samples=0,
        preload=True,
    )

    splitted = windows_dataset.split('session')

    # MOABB v1.0.0
    train_set = splitted['0train']  # Session train
    valid_set = splitted['1test']  # Session evaluation

    # MOABB v0.5.0
    #train_set = splitted['session_T']  # Session train
    #valid_set = splitted['session_E']  # Session evaluation

    import torch

    from braindecode.models import ShallowFBCSPNet
    from braindecode.util import set_random_seeds

    cuda = torch.cuda.is_available()  # check if GPU is available, if True chooses to use it
    device = 'cuda' if cuda else 'cpu'
    if cuda:
        torch.backends.cudnn.benchmark = True
    # Set random seed to be able to roughly reproduce results
    # Note that with cudnn benchmark set to True, GPU indeterminism
    # may still make results substantially different between runs.
    # To obtain more consistent results at the cost of increased computation time,
    # you can set `cudnn_benchmark=False` in `set_random_seeds`
    # or remove `torch.backends.cudnn.benchmark = True`
    seed = 20200220
    set_random_seeds(seed=seed, cuda=cuda)

    n_classes = 4
    classes = list(range(n_classes))
    # Extract number of chans and time steps from dataset
    n_chans = train_set[0][0].shape[0]
    input_window_samples = train_set[0][0].shape[1]

    model = ShallowFBCSPNet(
        n_chans,
        n_classes,
        input_window_samples=input_window_samples,
        final_conv_length='auto',
    )

    # Display torchinfo table describing the model
    print(model)

    # Send model to GPU
    if cuda:
        model = model.cuda()

    from skorch.callbacks import LRScheduler
    from skorch.helper import predefined_split

    from braindecode import EEGClassifier

    # We found these values to be good for the shallow network:
    lr = 0.0625 * 0.01
    weight_decay = 0

    batch_size = 64
    n_epochs = 100

    clf = EEGClassifier(
        model,
        criterion=torch.nn.NLLLoss,
        optimizer=torch.optim.AdamW,
        train_split=predefined_split(valid_set),  # using valid_set for validation
        optimizer__lr=lr,
        optimizer__weight_decay=weight_decay,
        batch_size=batch_size,
        callbacks=[
            "accuracy", ("lr_scheduler", LRScheduler('CosineAnnealingLR', T_max=n_epochs - 1)),
        ],
        device=device,
        classes=classes,
    )
    # Model training for the specified number of epochs. `y` is None as it is
    # already supplied in the dataset.
    _ = clf.fit(train_set, y=None, epochs=n_epochs)

[bruAristimunha]

Hey @gustavohenriquesr,

I think this got solved on braindecode size, at least with my tests, and I didn't note early that it was linked with this issue. https://github.com/braindecode/braindecode/pull/563

You can check on our size when you have some time, but not for now.

Install braindecode from scratch so that everything will works, command:

pip install -U https://api.github.com/repos/braindecode/braindecode/zipball/master#egg=braindecode

[bruAristimunha]

FYI @sylvchev