search

NeuromorphicProcessorProject / snn_toolbox

Toolbox for converting analog to spiking neural networks (ANN to SNN), and running them in a spiking neuron simulator.
MIT License
360 stars 104 forks source link

Batch size in examples/mnist_keras_loihi.py #102

Closed mantier closed 2 years ago

mantier commented 3 years ago

I tried to run the loihi example on a KB stick and ran into trouble.

The first problem was that the intel loihi adapter set a SLURM environment. I reported this as an issue here: https://github.com/intel-nrc-ecosystem/models/issues/13 I solved this issue by removing the line os.environ['SLURM'] = '1'.

There is a second problem. In some cases, a few batches were calculated but usually execution gets stuck before the first batch. Only twice, execution went through completely. tutorials from the nxsdk_app run without any problem. Behavior is erratic. In this case the program ran without a problem: delme In this case it stopped after a complete first batch: delme

I found out that only batch size 1 should be supported image Indeed, in this case the program does not stall. However, I want to ask whether this is to be expected. I once ran the example on the superhost and it worked well. If it should work with batch size > 1, this looks like a bug. If it should not work with batch size >1 (maybe in case the program is not executed on the superhost), it would be nice if execution stops with an error message.

rbodo commented 3 years ago

batch_size > 1 is implemented but still experimental (softmax support with batch size > 1 not fully validated).

The stalling issue has been reported before and has occured for batch size of 1 as well (larger batch sizes may just expose the issue faster). Afaik there has not been an identification / solution of the root cause. One suggestion was to use the Nahuku32 board (see last post in this thread), which of course is of no use if you are on KB). Guess it's best to stay with a batch size of 1 for now.