Comparing SpiNNaker and NEST

[felix-schneider]

In order to convince people here (at FZI) to use SpiNNaker, I tried to run a comparison between NEST and SpiNNaker. Essentially, our argument was going to be that SpiNNaker can do what NEST does now, in the same quality, but faster. Unfortunately, I was not able to produce the same output from the same network across SpiNNaker and NEST.

For reference, this is the simulation I was running. It is an example that ships with pyNN: https://gist.github.com/Scaatis/6acb5d023e16a5ea64be Invoke with python va_benchmark.py <spiNNaker/nest> CUBA I tested with the current master branch on a SpiNNaker 3 board versus pyNN 0.7.5 using nest 2.2.2

Note: To run this yourself, you will have to modify sPyNNaker to accept passing an rng to FixedProbabilityConnector (see #151). I hacked this feature in, you can find the changed files here: https://gist.github.com/Scaatis/7492f59415aeb1e9ef14

This is necessary so the networks run on NEST and SpiNNaker are actually the same.

I graphed the activations of the neurons form both simulators (pictured are the first 300 neurons over the first 20ms/200 simulation steps) As you can see, for about 10ms, SpiNNaker is only slightly ahead of NEST, after that, the activations are completely different.

I also tried simply running SpiNNaker twice with the same script and comparing activations: The output diverges as the simulation time increases.

We would very much like to use SpiNNaker for the speed advantage, but for our applications (HBP SP10) it's important that we get the same results despite switching simulators, and at the very least get the same results for successive runs.

Did I choose a poor benchmark? Are the differences intentional? Where does the non-determinism come from (I suspect synchronization issues between the cores)? Could there be a slower, but more synchronized, deterministic simulation mode?

[apdavison]

The va_benchmark network is very sensitive to small perturbations. It is essentially impossible to get identical spike outputs on different simulators, but the spike statistics are reproducible - see the section "Limitations on Reproducibility" in http://dx.doi.org/10.3389/neuro.11.011.2008

I don't know enough about the non-determinism with SpiNNaker to comment.

[rowleya]

There are precision differences between NEST and SpiNNaker (due to the use of fixed-point in the computations on SpiNNaker), so a particularly sensitive network will certainly diverge. These networks are chaotic in nature as I understand it.

In terms of the different outputs from different runs, although the network is the same by fixing the rng parameter, are the poisson sources also seeded? Otherwise, you will always get differences here. Dropped packets are worth investigating, depending on the network you are running.

[felix-schneider]

The Poisson sources are only present if COBA is passed as the benchmark name. As I am running CUBA, there are no Poisson Spike Sources.

I'm not sure if you mean dropped packets between SpiNNaker and the host, but I have SpiNNaker directly connected to the host machine by cable, no devices in between.

[rowleya]

OK, we will have to examine this in more detail to see what is going on.

On 23/10/2015 17:40, Felix Schneider wrote:

The Poisson sources are only present if |COBA| is passed as the benchmark name. As I am running |CUBA|, there are no Poisson Spike Sources.

— Reply to this email directly or view it on GitHub https://github.com/SpiNNakerManchester/sPyNNaker/issues/152#issuecomment-150628604.

[felix-schneider]

We built a second example, with a hopefully less volatile network. The simulation is intended to be 100% deterministic, no randomness involved in creation or running. It's 10 layers of 10 neurons each, with spikes from two sources at different frequencies being fed into the first layer. https://gist.github.com/Scaatis/253680fe1f1bd2ce2885

First of all, we did not have a problem with SpiNNaker being non-deterministic in this example.

Here is the voltage plot for a neuron from the first layer (the one immediately after the inject) at 1ms timestep. Except for an odd hiccup at the end, they look quite similar. This is zoomed in: There is a difference, but not unlike the difference between nest and other simulators.

This is a plot of the recorded spikes: There are several things to note here: The spikes are recorded in different layers, they all have index 1 because the pictured spikes are neuron 1 of each layer firing in succession. The synaptic delay is 1ms, all parameters are the same between spiNNaker and nest. There are two issues here: The first spike is recorded one timestep before the first spike of nest, despite the voltage plot showing both spiking simultaneously. This is likely just a difference in how spikes are recorded. The second, more pressing, problem is that the spikes propagate through the network twice (!) as fast in SpiNNaker as in nest.

This is the voltage plot form before, at 0.2ms timestep. Again, it's quite similar, though the spike time is now actually different:

These are the spikes at 0.2ms resolution: The spikes for spiNNaker are 1.6ms apart, the ones from nest 2.0ms. Previously, the timestep exaggerated this difference, but further decreasing it past 0.2ms doesn't change it any further.

This is a voltage plot for a neuron from the second layer: The excitation starts at the same time for both spiNNaker and nest, but spiNNaker spikes 3 timesteps earlier, despite not having reached the threshold voltage (-50mV)

The small differences in the voltage I understand, they could become a problem in more complex networks as the simulation goes longer, but they are to be expected.

I'm wondering whether I'm hitting on odd edge cases here, or if there are functional differences between the impmentations of the neuron model (IF_cond_exp in this case). I'm particularly interested in a remark in neuron_model_lif_cond_impl.c:

// simple Leaky I&F ODE - discrete changes elsewhere -  assumes 1ms timestep?
static inline void _lif_neuron_closed_form(
        neuron_pointer_t neuron, REAL V_prev, input_t input_this_timestep) {

    REAL alpha = input_this_timestep * neuron->R_membrane + neuron->V_rest;

    // update membrane voltage
    neuron->V_membrane = alpha - (neuron->exp_TC * (alpha - V_prev));
}

Could this be the reason the neuron spikes prematurely at a timestep other than 1ms? Though that does not explain the different behaviour at 1ms timestep. I'm interested in your opinions.

[hopper333]

Hi Felix

I am partly responsible for the neuron solving code at the moment and so will have a look at this as soon as I return to Manchester and can have a discussion with Andrew. This is likely to be early next week (unless Andrew is away himself?).

One thing I would say is that you cannot expect SpiNNaker and NEST to be absolutely identical in all cases as we are working with fixed-point arithmetic and synapses with limited precision. Together with the guys at Juelich, we are just about to carry out a very large and thorough comparison with NEST, but using statistical measures of differences. NEST itself has at least two flavours: 'precise' and 'discrete', and these can produce clearly different responses themselves on quite simple neuron models and networks. I think if you compare any two simulators on exact voltage traces you will find some differences - there is not a 'reference' as far as I know, but in the case of simple networks I would trust the ODE solver in Mathematica with high numerical precision to get as close to a reference result as possible (NEST 'precise' has also matched Mathematica very accurately on some LIF models).

M

On 27 Oct 2015, at 14:04, Felix Schneider wrote:

We built a second example, with a hopefully less volatile network. The simulation is intended to be 100% deterministic, no randomness involved in creation or running. It's 10 layers of 10 neurons each, with spikes from two sources at different frequencies being fed into the first layer. https://gist.github.com/Scaatis/253680fe1f1bd2ce2885

First of all, we did not have a problem with SpiNNaker being non-deterministic in this example.

Here is the voltage plot for a neuron from the first layer (the one immediately after the inject) at 1ms timestep. [volt1ms]https://cloud.githubusercontent.com/assets/208336/10759439/9333ee52-7cb7-11e5-83ca-291da588b1ed.png Except for an odd hiccup at the end, they look quite similar. This is zoomed in: [volt1mszoom]https://cloud.githubusercontent.com/assets/208336/10759455/a8aba0ae-7cb7-11e5-9a5a-2a249b59b19c.png There is a difference, but not unlike the difference between nest and other simulators.

This is a plot of the recorded spikes: [spikes1ms]https://cloud.githubusercontent.com/assets/208336/10759518/ff49dd22-7cb7-11e5-8f36-e177d01499ab.png There are several things to note here: The spikes are recorded in different layers, they all have index 1 because the pictured spikes are neuron 1 of each layer firing in succession. The synaptic delay is 1ms, all parameters are the same between spiNNaker and nest. There are two issues here: The first spike is recorded one timestep before the first spike of nest, despite the voltage plot showing both spiking simultaneously. This is likely just a difference in how spikes are recorded. The second, more pressing, problem is that the spikes propagate through the network twice (!) as fast in SpiNNaker as in nest.

This is the voltage plot form before, at 0.2ms timestep. [volt02ms]https://cloud.githubusercontent.com/assets/208336/10759781/a7e7eec8-7cb9-11e5-84a7-ed77954331f8.png Again, it's quite similar, though the spike time is now actually different: [volt02mszoom]https://cloud.githubusercontent.com/assets/208336/10759816/d902a052-7cb9-11e5-8c12-8758e5e33596.png

These are the spikes at 0.2ms resolution: [spikes02ms]https://cloud.githubusercontent.com/assets/208336/10759863/1f397c44-7cba-11e5-806e-2c7f570272a8.png The spikes for spiNNaker are 1.6ms apart, the ones from nest 2.0ms. Previously, the timestep exaggerated this difference, but further decreasing it past 0.2ms doesn't change it any further.

This is a voltage plot for a neuron from the second layer: [secondlayer02ms]https://cloud.githubusercontent.com/assets/208336/10760100/26da1dc2-7cbb-11e5-8788-4cb59f8bce8a.png The excitation starts at the same time for both spiNNaker and nest, but spiNNaker spikes 3 timesteps earlier, despite not having reached the threshold voltage (-50mV)

The small differences in the voltage I understand, they could become a problem in more complex networks as the simulation goes longer, but they are to be expected.

I'm wondering whether I'm hitting on odd edge cases here, or if there are functional differences between the impmentations of the neuron model (IF_cond_exp in this case). I'm particularly interested in a remark in neuron_model_lif_cond_impl.c:

// simple Leaky I&F ODE - discrete changes elsewhere - assumes 1ms timestep? static inline void _lif_neuron_closed_form( neuron_pointer_t neuron, REAL V_prev, input_t input_this_timestep) {

REAL alpha = input_this_timestep * neuron->R_membrane + neuron->V_rest;

// update membrane voltage
neuron->V_membrane = alpha - (neuron->exp_TC * (alpha - V_prev));

}

Could this be the reason the neuron spikes prematurely at a timestep other than 1ms? Though that does not explain the different behaviour at 1ms timestep. I'm interested in your opinions.

— Reply to this email directly or view it on GitHubhttps://github.com/SpiNNakerManchester/sPyNNaker/issues/152#issuecomment-151505633.

---

Michael Hopkins, SpiNNaker project, APT group, School of Computer Science, University of Manchester, Manchester M13 9PL michael.hopkins@manchester.ac.ukmailto:simon.davidson@manchester.ac.uk

[alan-stokes]

Just a curious question. Should this be moved over to the google group? as its more of a general issue to be rectified by the field. Its unlikely to be rectified within a issue tracked bug, as theres no solid bug to report.

[felix-schneider]

Just to be clear: We don't expect the output to be exactly the same as NEST. However, the more similar the simulators are, the smoother the transition to SpiNNaker will be.

[alan-stokes]

closed as requests.