search

SpiNNakerManchester / sPyNNaker

The SpiNNaker implementation of the PyNN neural networking language
Apache License 2.0
100 stars 42 forks source link

Too much delay causes delay packets to be be dropped. #1045

Open Christian-B opened 3 years ago

Christian-B commented 3 years ago

Script: import spynnaker8 as sim

n_neurons = 76 # 75 works 76 fails sim.setup(timestep=1.0)

pop_1 = sim.Population(n_neurons, sim.IF_curr_exp(), label="pop_1") input1 = sim.Population( n_neurons, sim.SpikeSourceArray(spike_times=range(0, 1000, 100)), label="input")

input2 = sim.Population( n_neurons, sim.SpikeSourceArray(spike_times=range(0, 1000, 100)), label="input") input3 = sim.Population( n_neurons, sim.SpikeSourceArray(spike_times=range(0, 1000, 100)), label="input") input4 = sim.Population( n_neurons, sim.SpikeSourceArray(spike_times=range(0, 1000, 100)), label="input") input5 = sim.Population( n_neurons, sim.SpikeSourceArray(spike_times=range(0, 1000, 100)), label="input") sim.Projection( input1, pop_1, sim.OneToOneConnector(), synapse_type=sim.StaticSynapse(weight=5, delay=1)) sim.Projection( input1, pop_1, sim.OneToOneConnector(), synapse_type=sim.StaticSynapse(weight=5, delay=20)) sim.Projection( input1, pop_1, sim.OneToOneConnector(), synapse_type=sim.StaticSynapse(weight=5, delay=40)) sim.Projection( input1, pop_1, sim.OneToOneConnector(), synapse_type=sim.StaticSynapse(weight=5, delay=60)) sim.Projection( input1, pop_1, sim.OneToOneConnector(), synapse_type=sim.StaticSynapse(weight=5, delay=80)) pop_1.record(["spikes"]) sim.run(1200)

spikes = pop_1.spinnaker_get_data("spikes") sim.end() print(len(spikes)) print(10 5 n_neurons)

Christian-B commented 3 years ago

Found when moving https://github.com/SpiNNakerManchester/sPyNNaker8/pull/429

rowleya commented 3 years ago

If too many packets arrive at the core at the same time, and they can't all be processed within a single timestep, it will throw those away. This is by design. Having different delays will help a bit because the packets are then captured in delay extensions which spread these packets out more.

Christian-B commented 3 years ago

The issue is NOT the just the number of packets that arrive at the core at the same time.

Changing it so that each projection comes from a different but identically firing SpikeSourceArray avoid the issue.

So it is an overload on the one DelayVertex that is causing the packets to be lost.

rowleya commented 3 years ago

Do you have an example of the error being generated? This might indicate which bit is losing the packets to make it more clear...

Christian-B commented 3 years ago

It reports:

20 packets from pop_1:0:75 on 1, 1, 4 were dropped from the input buffer, because they arrived too late to be processed in a given time step. Try increasing the time_scale_factor located within the .spynnaker.cfg file or in the pynn.setup() method.

A maximum of 2 background tasks were queued on pop_1:0:75 on 1, 1, 4. Try increasing the time_scale_factor located within the .spynnaker.cfg file or in the pynn.setup() method.

Then it reports only 3769 spikes instead of the expected 3800

Christian-B commented 3 years ago

Changing it so that each project comes from a different but identical input population and instead it reports:

The input buffer for pop_1:0:75 on 1, 1, 8 lost packets on 759 occasions. This is often a sign that the system is running too quickly for the number of neurons per core. Please increase the timer_tic or time_scale_factor or decrease the number of neurons per core.

BUt it has all 3800 spikes.

rowleya commented 3 years ago

The first case where it loses packets is expected if the receiver core is overloaded, and from the messages, it is. It doesn't seem related to delay extension overload here either.

The second case where it loses packets and then creates all spikes is quite odd! That may need more investigation...

andrewgait commented 3 years ago

The first case where it loses packets is expected if the receiver core is overloaded, and from the messages, it is. It doesn't seem related to delay extension overload here either.

The second case where it loses packets and then creates all spikes is quite odd! That may need more investigation...

After some discussion on Skype this morning I believe we have concluded that both these cases are behaving as we would expect them to with the current code, and that the drops in the input buffer in the case where there are separate input populations are of packets that don't matter with regard to the correct functioning of the code.

It's worth noting here that in this particular example (with separate input populations), increasing the default value of incoming_spike_buffer_size (from 256 to 512) in the config file stops the warnings / overflows etc.

Christian-B commented 2 years ago

Assigning to @rowleya as may be an interesting test script for his no longer dropping packets idea.