Problems with conversion of simple CNN for time series data

[Matthijspals]

Hi, First of all, thank you very much for making this toolbox, it has been of great use to us! We are working on classifying two class EEG data using spiking networks. Currently we are however unable to get any decent accuracy with the converted model, we hope you might have any idea what is going wrong.

A typical input to our model looks like this: The actual input to the model will be these two EEG channels stacked on top of each other to create an image of size (201, 2).

Our model is defined as follows:

input_layer = Input(input_shape)
layer = Conv2D(filters=12,
               kernel_size=(20,2),
               strides=1)(input_layer)
layer = BatchNormalization()(layer)
layer = ReLU()(layer)
layer = AveragePooling2D(pool_size=(30,1), strides=(10,1))(layer)
layer = Flatten()(layer)
layer = Dropout(0.5)(layer)
layer = Dense(units=2,activation='softmax')(layer)

Note that we have tried various variants of this model, with different number of kernels / filter sizes / strides. We have also tried swapping the average pooling layer out for another convolutional layer. The model (in these various configurations) typically gets around 80 to 90 % validation accuracy, with the parsed model (see below) coming in equal (or very close) to this.

=================================================================
input_1 (InputLayer)         [(1, 201, 2, 1)]          0         
_________________________________________________________________
0Conv2D_182x1x12 (Conv2D)    (1, 182, 1, 12)           492       
_________________________________________________________________
1AveragePooling2D_16x1x12 (A (1, 16, 1, 12)            0         
_________________________________________________________________
2Flatten_192 (Flatten)       (1, 192)                  0         
_________________________________________________________________
3Dense_2 (Dense)             (1, 2)                    386       
=================================================================

For conversion we use a config file with the following settings:

[tools]
evaluate_ann = True
normalize = True
simulate = True

[simulation]
simulator = nest
duration = 150
num_to_test = 80
batch_size = 1
dt = 0.1

[input]
poisson_input = True
input_rate = 1000

[cell]
tau_refrac = 0.1
delay = 0.1
v_thresh = 0.01

We have tried directly varying the scaling of the input images and varying the poisson input rate, the simulation time and turning normalization (and exp scaling) on and off. Yet the accuracy of our spiking model is usually around 50-60% (we have two classes so basically chance). For normalization we simply use the test set. We have mostly used nest as a backend due to it running locally, our final goal is to use SpiNNaker as backend. However we have previously been succesful with first tuning a model in Nest and then storing the weights + loading them on a server with SpiNNaker.

As a final remark, we have tried looking at the generated plots, and typically we noted that the correlation between the ANN's activation and the SNN's spiking rate is really low for the input convolutional layer (around 0.2-0.6), and similarly so for the average pooling, so we are guessing something goes already wrong here. There are spikes in all layers, the output dense layer however seems to always spike a lot, but always just for one particular class.

Thank you very much for taking a look, and please let me know if there is any information missing!

[rbodo]

Hi,

Thanks for the detailed description. It's good that you looked at the correlation plots; feel free to post or send me the plots of the Conv layer, maybe I see some pattern.

One possible source of ANN-SNN mismatch is outliers the biases (either within the Conv layer itself or induced by the BatchNorm layer). That would be apparent by off-diagonal lines in the correlation plot. You can also check the effect by temporarily disabling the bias (and BatchNorm if possible) during training.

When using Poisson input, it's expected that there's always some noise in the hidden layer correlations, though that should result in accuracy at chance level. You should be able to turn off Poisson input to remove the additional spike variability. Then the toolbox uses regularly sampled input spike trains (I hope it still works, haven't used the Nest simulator in a while).

Another thing to check is at what point during the conversion the accuracy drops. What is the accuracy of the ANN right after training? What accuracy does the toolbox report for the "input model"? And for the "parsed model"? They should all be the same.

Is your dataset normalized to the range [0, 1]? That would be good (it seems like it from the plot you showed).

You might want to try the builtin INIsim once. It won't be useful for Spinnaker but it will tell you if the conversion works in principle for that particular architecture and dataset. If you get good results there, then it's likely an issue with the neuron parameters used in Nest.

[Matthijspals]

Hi,

thank you very much your response! I just retrained the model with Biases in the convolutional layer set to False. Turning of BatchNorm is not really an option as our model doesn't learn properly any more then. The validation accuracy after training is 95.24% which is exactly equal to that of the parsed model, so parsing seems to go fine. The accuracy of the spiking model with config file as above is again chance (48.81).

Here's the correlation plot of the convolutional layer, as you can see most points are not at all on the diagonal, so most likely this is where something is going wrong:

I also tried turning off the poisson input which still gives chance level accuracy (but also not sure how this is handled in Nest), and indeed our input is normalised to be in the range [0, 1].

INIsim, with config file as above actually seems to work reasonably well. With Poisson input set to False and sim time = 50, I got 89.29% accuracy on the whole val set, however with Poisson input set to True this is only 62.51%.

I have also attached the stored model, input npz files and config file in case it is of any use. ModelAndInput.zip

[rbodo]

Hmm, interesting. Given that the accuracy seems to be ok on INIsim without but not with Poisson input, I have the feeling the issue might be related to the oscillatory nature of the EEG data. But since turning off Poisson input on Nest does not improve the accuracy by much, there must be another issue too. Notice that soft reset (reset by subtracting the threshold) is not implemented for Nest backend (not even sure it would be supported on Spinnaker). So we only have hard reset (to zero) available, which can cause a large drop in accuracy. There is not much you can do about it except try to increase the simulation duration or decreasing the time resolution further. But to test whether the input has anything to do with it I would replace your data with constant input (e.g. all set to 0.5), or uniformly distributed random input in range [0, 1]. The model weights could also be left randomly initialized (skip training). Then instead of looking at the accuracy, just check the correlation plot of the conv layer. You should get decent correlation with your config, even with reset-to-zero. If the correlation is still bad, it might be something with the architecture. If the correlation is good on random input and random weights, add your trained weights back in and check the correlation again. If it's bad, there's likely outliers in the weight / bias distribution. If it's good, then the input data might have to be preprocessed...

Sorry for not being able to help more, don't have access to a Nest installation atm.

[Matthijspals]

Thanks again for your input! We have managed to gain quite decent results by first transforming the EEG to a time-frequency representation (spectrograms) and then classifying that with a spiking CNN.

Of course being able to classify the raw signal has quite some advantages over first doing a time-frequency transformation, so we are aiming to explore more in detail what goes wrong at a later stage (using your pointers). I'm closing the issue now, but in case we get to being able to succesfully classifying the raw EEG signal I will post the solution here (or in a new issue?) in case anyone else is running into it.

[rbodo]

Nice, glad to hear it! Would be great if you could post your findings here, will be helpful to the community for sure.