Closed Christian-B closed 5 years ago
Christian-B
commented
5 years ago Note same script also dies later:
2018-12-19 13:02:15 ERROR: Error when calling pacman.operations.fixed_route_router.fixed_route_router.FixedRouteRouter.call with inputs dict_keys(['board_version', 'placements', 'destination_class', 'machine'])
Traceback (most recent call last):
File "/home/brenninc/spinnaker/IntroLab/synfire/synfire.py", line 39, in
Christian-B
commented
5 years ago Note there appears to be an growing cost in the partitioner as the exact same script reduced by a factor of 10 n_populations = 20
takes: 2018-12-19 13:07:44 INFO: Time 0:00:24.960147 taken by PartitionAndPlacePartitioner
rowleya
commented
5 years ago Doing some tests - the partitioning of edges seems to be taking a while (population partitioning was reasonable)...
rowleya
commented
5 years ago It is worth noting that this example results in over 40,000 cores being used. Each application vertex has 10,000 atoms and gets split in to 100 neurons each, so there are 100 machine vertices for every application vertex. Additionally, each of these has a delay extension. During partitioning, we are not allowed to remove edges, so a single edge going from 10,000 atoms to 10,000 atoms becomes 100 edges from every pre-vertex to every post-vertex, so 10,000 edges per pair of populations.
I think that this is therefore not that surprising overall. This is one of the issues we have to deal with however; a very simple network description can explode into a very large SpiNNaker network with very little effort.
rowleya
commented
5 years ago Note also that your key error could be a sign of something that needs blacklisting in the machine. It is worth keeping an eye on which board you got when you got this error, and seeing if it reoccurs in future simulations at the same place.
Christian-B
commented
5 years ago Yes I full agree that this is a LARGE job with 40001 cores uses excluding special ones.
The interesting factor here is that an increase of times 10 cores results in an increase of times 40 in time.
rowleya
commented
5 years ago This is true, though I would need to think about it a bit more. It may be that the connectivity is more than 10 times increased with 10 times as many cores... It isn't clear that this is the case here though...
rowleya
commented
5 years ago In my own test, I found that with 200 populations, it took 2 minutes 52 seconds to partition, or 172 seconds. With 20 populations, it took 22 seconds. So that looks like a less than 10 times increase to me.
rowleya
commented
5 years ago Partitioning does result in 5.5GB of RAM being used by the python process once I run it however. I think your system has at least 16GB RAM so I would have expected it to cope with this without caching, but it could be some sort of system thrashing that causes your run to take 16 minutes.
Christian-B
commented
5 years ago Rowley does not consider this an issue so closing.
May be related to: https://github.com/SpiNNakerManchester/PACMAN/issues/190
Using a very big 40,001 core but simple network: Time 0:16:01.598417 taken by PartitionAndPlacePartitioner
Script:
import spynnaker8 as sim from spynnaker8.utilities import neo_convertor
n_neurons = 10000 n_populations = 200 weights = 5 delays = 17.0 simtime = 50000
sim.setup(timestep=1.0, min_delay=1.0, max_delay=144.0) sim.set_number_of_neurons_per_core(sim.IF_curr_exp, 100)
spikeArray = {'spike_times': [[0]]} stimulus = sim.Population(1, sim.SpikeSourceArray, spikeArray, label='stimulus')
chain_pops = [ sim.Population(n_neurons, sim.IF_currexp, {}, label='chain{}'.format(i)) for i in range(n_populations) ] for pop in chain_pops: pop.record("all")
connector = sim.OneToOneConnector() for i in range(n_populations): sim.Projection(chain_pops[i], chain_pops[(i + 1) % n_populations], connector, synapse_type=sim.StaticSynapse(weight=weights, delay=delays))
sim.Projection(stimulus, chain_pops[0], sim.AllToAllConnector(), synapse_type=sim.StaticSynapse(weight=5.0))
sim.run(simtime)