Review: Redes Neurais Profundas Triplet Aplicadas a Classificacao de Sinais em Interfaces Cérebro Computador

[Kotzly]

Review of Redes Neurais Profundas Triplet Aplicadas a Classificacao de Sinais em Interfaces Cérebro Computador.

[Kotzly]

Aprendizado Profundo e Interfaces Cérebro-Computador

Author: Willian Rampazzo Institution: Universidade Estadual de Campinas - UNICAMP

The promising results obtained by DCNNs in different areas indicate that there is potential to employ them in the task of processing brain signals in BCI systems. In this work, we explore the use of these networks in a Steady-State Visually Evoked Potentials (SSVEP) -based BCI investigating, initially, whether different input formats such as the raw signal or a signal transformation, for example, the short-time Fourier transform, influence the performance of DCNNs. For such, we propose new DCNN architectures and evaluate them in different input formats, comparing their performances to the approaches usually adopted in the BCI classification step. This work also investigates whether it is possible to use the knowledge transfer technique to adjust pre-trained DCNNs to BCI data. Experimental results indicate that DCNNs are an option to apply to brain signal processing in BCIs.

[Kotzly]

Related works

1. CECOTTI, H.; GRAESER, A. Convolutional neural network with embedded Fourier transform for EEG classification

Convolutional network with 4 layers: input layer, convolutional layer with 10 filters (size 5), using hiperbolic tangent as activation function. Also a transition class and a rejection approach.

2. CECOTTI, H. A time–frequency convolutional neural network for the offline classification of steady-state visual evoked potential responses. The dataset is composed of 2 subjects, and one network is trained for each subject.

Same author from last paper, but the network was increased, with bigger filters and more layers. Not using the transition class nor the rejection approach, greatly improved results from the last work. 10 subjects were used to create the dataset, also training one network for each subject. The activation functions were changed to sigmoid.

3. CECOTTI, H. Convolutional neural networks for event-related potential detection: impact of the architecture

In this work the authors compared 6 architectures. The networks were trained with 16 subject individually, and also with a subset of subjects. Results showed that the networks that trained in a set of subjects performed better than the average result for the respective architecture trained for each subject, showing that they can generalize. All activation functions were ReLU.

4. BCI-SSVEP based approach for control of walking in virtual environment using a convolutional neural network

Using the same architecture as in [2], but not using the superior harmonics in the input. There are 3 visual stimuli that the user can focus to guide the it's path in a virtual reality environment. Five electrodes were use to acquire the EEG signals, and are segmented in 2 second windows and normalized.

5. THOMAS, J.; MASZCZYK, T.; SINHA, N.; KLUGE, T.; DAUWELS, J. Deep learning-based classification for brain-computer interfaces.

In this work the authors compare 6 algorithms, and used a public dataset with 11 subjects

6. KWAK, N.-S.; MÜLLER, K.-R.; LEE, S.-W. A convolutional neural network for steady state visual evoked potential classification under ambulatory environment.

Em Kwak et al. (2017), os autores compararam o desempenho de três algoritmos frequentemente empregados em BCI SSVEP, a Análise de Correlação Canônica (CCA, do inglês, Canonical Correlation Analysis), o Índice de Sincronização Multivariado (MSI, do inglês, Multivariate Synchronization Index) e o CCA-kNN, com o desempenho de duas CNNs e uma ANN propostas no trabalho para controlar um exoesqueleto em ambiente ambulatorial.

Eight electrodes were used, and the algorithms were trained for every subject individually and the CNN with 3 layers was the best one, having up to 99.28% accuracy.

7. AZNAN, N. K. N.; BONNER, S.; CONNOLLY, J.; MOUBAYED, N. A.; BRECKON, T. On the Classification of SSVEP-Based Dry-EEG Signals via Convolutional Neural Networks.

Authors evaluated a CNN that is used in the raw EEG signals. When trained and evaluated in a single subject, accuracy was 96%, but accuracy greatly reduced when training and testing in different subjects. Increasing the CNN depth helped obtain a 69% accuracy.

8. AZNAN, N. K. N.; ATAPOUR-ABARGHOUEI, A.; BONNER, S.; CONNOLLY, J.; MOUBAYED, N. A.; BRECKON, T. Simulating brain signals : creating synthetic EEG data via neural-based generative models for improved SSVEP classification.

An approach for generating synthetic EEG data was used to augment the sataset used in previous work. Results showed that this synthetic data, when used for training, helped the CNN converge.

9. ATTIA, M.; HETTIARACHCHI, I.; HOSSNY, M.; NAHAVANDI, S. A time domain classification of steady-state visual evoked potentials using deep recurrent-convolutional neural networks.

A CNN is used to extract features from the raw data, then a LSTM (128 units) is used to correlate the features, and a fully connected layer is used to classify it in 4 frequencies. This network is compared to the CCA algorithm, outperforming it by 6%.

10. ATTIA, M.; HETTIARACHCHI, I.; MOHAMED, S.; HOSSNY, M.; NAHAVANDI, S. A frequency domain classifier of steady-state visual evoked potentials using deep separable convolutional neural networks.

A architecture inspired by VGG is used, having as input 2-second spectograms of the input signal. Eight electrodes are used in the input, coming from 4 subjects, and a network is trained for each one.

[Kotzly]

Dataset

Described in [11], with 10 healthy volunteers, and the protocol was the visualization of repetitive visual stimuli, at 6, 7.5, 12, 15m 20 and 30Hz. They collect 8 sessions of 12 seconds for each frequency plus the rest. EEG was acquired from 16 electrodes placed

During acquisition, a 8º order band-pass butterworth filter was used, between 5 and 60 Hz, and a notch filter from 58 to 62 was also used.

This data was used to create 4 datasets:

• STFT absolute value;
• Spectogram, without higher harmonics;
• Spectogram, with one high harmonic;
• Raw signals.

Architectures

Two architectures were proposed, one that receives raw signals:

And one that receives images (the spectograms):

Also, there were experiments performed using Transfer Learning, using the ResNet architecture with pre-trained weights on ImageNet, with 5 size variations.

Discussion and conclusion

Five experiments were performed:

• Analyze the input data format;
• Investigate different DCNN's architectures;
• Investigate the Transfer Learning technique applied to this problem;
• Investigate the Triplet loss function;
• Change the training protocol, training for a set of users and testing on users that the network have not been trained with.

[11] CARVALHO, S. N.; COSTA, T. B.; URIBE, L. F.; SORIANO, D. C.; ALMEIDA, S. R.; MIN, L. L.; CASTELLANO, G.; ATTUX, R. Effect of the combination of different numbers of flickering frequencies in an SSVEP-BCI for healthy volunteers and stroke patients.