Unexpected Simulation Accuracy Results on Loihi

[jimzhou112]

Hello,

I am using SNN Toolbox to convert a Keras/TF ANN model to SNN to run on Intel's Loihi chip. The resulting simulation accuracy numbers for both ANN and SNN are unexpected. My pretrained Keras/TF model has a Top-1 accuracy of around 35% on the test set, but the simulated ANN on Loihi peaks at 87.5% Top-1 accuracy with an 80% Top-1 accuracy on the SNN at the end of batch 80. As the number of batches increases, both of the simulated accuracy figures continually decline from that point onward until the last batch.

What is causing this discrepancy between my Keras/TF ANN model's 35% accuracy and the much higher numbers reported during simulation for both ANN/SNN? How can I get the simulated ANN model to match the true accuracy of my Keras/TF model?

Also, why does the accuracy increase over time, peaking at batch 80, then decline continuously for every batch afterwards?

I've attached my config file as well as the SNN/ANN accuracy data on the simulation. Thank you for any help.

accuracy.txt config.txt

[rbodo]

That should simply be an effect of small sample size. I've noticed the same for other datasets. In your first 80 samples there happen to be many that the network recognizes correclty; since the sample size is still small it has a large effect on the mean. You could just shuffle your data once; the peak will probably be smoothed out.

[jimzhou112]

Spot on answer, a good shuffle resolved it very nicely. Thank you.