about the accuracy loss of converted to snn

[jackyfaster]

Hi rbodo, I trained a binarynet model with keras(2.2.4). when i use snntoolbox , i get the following result: /////////////////////////////////////////////// Evaluating input model on 100 samples... Top-1 accuracy: 98.00% Top-5 accuracy: 100.00% //////////////////////////////////////////// Evaluating parsed model on 100 samples... Top-1 accuracy: 98.00% Top-5 accuracy: 100.00% //////////////////////////////////////////// Moving accuracy of SNN (top-1, top-5): 15.00%, 53.00%. Moving accuracy of ANN (top-1, top-5): 98.00%, 100.00%. Simulation finished. Total accuracy: 15.00% on 100 test samples. Accuracy averaged over classes: 15.06% ///////////////////////////////////////////////////

and the config file as the following: ////////////////////////////////////// [paths] dataset_path = D:\Pycharm_workspace\snn_toolbox-master-binary\examples\datasets\mnist filename_ann = binarynet_mnist_all_bias_95.87

[input] model_lib = keras

[tools] evaluate_ann = True normalize = False

[simulation] simulator = INI duration = 30 num_to_test = 100 batch_size = 50 keras_backend = tensorflow top_k = 5

[cell] binarize_weights = True reset = Reset to zero //////////////////////////////////////

if i use a float model, the snntoolbox works well, but binarynet model, it's not good , i really don't know why, please give me some suggestions, thanks a lot.

[rbodo]

@jackyfaster Sorry for the delay, I'm travelling at the moment, will get the chance to take a look some time next week.

[jackyfaster]

@rbodo Nice, I'm waiting for your message.

[rbodo]

When you say "BinaryNet", are you referring to binarizing both weights and activations, using the framework of Mathieu Courbariaux? If so, #33 might be related.

But I see you are using keras and tensorflow in your config file, so it seems you binarized the network yourself? What kind of binarization function are you using for the activations? binary_tanh for [-1, 1], or binary_sigmoid for [0, 1], or something else?

Some things to check / try:

• Inspect the _INI.h5 file which the toolbox saves and make sure the weights are indeed binarized as desired.
• Increase the simulation duration
• Use "Reset by subtraction" instead of "Reset to zero"
• Enable output plots by setting plot_vars = {'all'} in section [output] of your config file. Then compare the ANN activations and the SNN spikerates. This should give you an idea where the SNN diverges from the ANN.
• Does your model use a MaxPooling layer? They might cause some drop in accuracy when converted to spiking, but not that much.
• You are probably using softmax as output activation function. Try setting softmax_to_relu = True in the [conversion] section.

[jackyfaster]

@rbodo Hi, I binarized both weights and activations myself, and i used binary_tanh[-1, 1] binarization function. thanks for your suggestions much, i will try.

[jackyfaster]

@rbodo hi,

i am sorry to bother you again, i changed the binarization function from binary_tanh to binary_sigmoid, the box works well. thanks for your suggestions again! and i want to ask a question more, i run a same ANN model in the box, not only one time, i got the different results as the following:

one result

Batch 2 of 2 completed (100.0%) Moving accuracy of SNN (top-1, top-5): 93.00%, 100.00%. Moving accuracy of ANN (top-1, top-5): 91.00%, 100.00%. Simulation finished.

another result

Batch 2 of 2 completed (100.0%) Moving accuracy of SNN (top-1, top-5): 91.00%, 100.00%. Moving accuracy of ANN (top-1, top-5): 91.00%, 100.00%. Simulation finished.

I would be grateful if you could give me some advice.

[rbodo]

Just to clarify: The network in both cases is the same?

Also, I assume you made sure that the samples in the 2 batches are the same in both cases.

Then you need to make sure that the binarization functions you use are deterministic. Go to the script snntoolbox/utils/utils.py and have a look at the various binarization functions. They expose the argument deterministic, which is True by default. If you use any of them, check that this parameter is not maybe set to False.

Finally, you could find out which two samples were misclassified in the second case. Then plot the spikerates, correlations etc for these two samples, and see whether / how they change when you run it again.