MemoryError due to synaptic parameters

[eneftci]

Some feature of pyNCS need to be changed to cope with large address spaces due to synaptic parameters (weights, delays etc.). I'm currently dealing with such a system, and address spaces of size 28 bits requires about a gigabyte of memory, which is clearly too much.

One way to address this problem is to remove synaptic parameters (e.g. weights delays) from the input address space, and add them only when they are needed: e.g. when a connection is created. This might involve having a third address Specification (in addition to AER in and AER out), that a new Connection class, say PWConnection, uses to bulid physical addresses corresponding to the right weight,delay, in addition to neuron, synapse, probability, etc.

But this means that even in the absence of a mapper, one requires Connection to stimulate the neurons. We could create a virtual (software) mapper that maps events from the virtual chip to the real neurons.

Any thoughts?

[fabioedoardoluigialberto]

I encountered the same troubles when dealing with SRAM synapses, where each weight corresponds to a different address. However, the use of structured arrays for AddrGroups and the "online" generation of addresses (only when needed) should have partially made this problem more tractable. Isn't this the case now? From what you say I understand that the whole address space is populated when you create a new Population. Anyway, in my case the numbers are not that big (like 58x32x16x2 if I'm not mistaken) and we managed to manipulate them pretty easily.

I don't think it's a good idea to branch classes for specific implementation choices (i.e., parameters as different AER events) but I agree that the system should be able to deal with large address spaces using a more memory efficient solution. I also agree that one possibility is to use Connections to "trigger" address translations and generate addresses on the fly, instead of leaving this to population. In summary, Populations should only contain the information needed to generate the addresses in the proper manner (this info taken from Setup, neuronblocks, etc.), so that whenever addresses are needed (for example to create mapping using Connection or to visualize data) Population can do it.

I guess the above is basically what you suggested but I don't see the need for a new PWConnection or a new Address Specification. The AddrSpec by definition should contain all the needed information already, and I think it's fine if it's only 'in' or 'out'. Am I missing something?

I also don't see why one requires Connection to stimulate the neurons without the mapper but first things first :)

[eneftci]

Unfortunately, NeuronBlock tries to reserve memory for the entire address space in NeuronBlock.expand_dims (in _buildGrid). I haven't looked at how crucial this is, but I expect that some features depend on the availability of the entire address space in the block.

My solution with the PWConnection didn't need to modify the core features of pyNCS so we are guaranteed to have old scripts to work. If we're ready to change neuronblock and take the risk of breaking compatibility, then your solution is best.

[sheiksadique]

• I agree with @eneftci that generation of any kind of addresses should be put off till the creation of Connection objects because when you have a chip with say 10 bit weight resolution on its synapses, even a single neuron is going to need 2^10 addresses in its synapse AddrGroup, which is kind of point less.

[fabioedoardoluigialberto]

(Also the "dummy chip" idea is linked to this problem. At some point we were considering to have a dummy chip where the "out" addresses are all the possible "in" addresses in the setup .. but of course this is not really feasible.)

You are right we shouldn't break compatibility. But I still think an elegant solution is possible. The idea would be that anything that needs addresses should be able to craete them online, instead of having them somewhere. Or at least there should be the possibility to do that if needed so both methods should be seamlessly interchangeable. My mind goes to sparse matrices, which are stored in an efficient way, while any element of groups of elements can still be retrieved easily. Following the same lines we could have a "sparse population", or actually "large" population, where addresses are created only upon request. This idea doesn't apply if the general trend of AER systems is to go towards a situation where parameters are sent with each spike. (Note: personally I think AER should only be used for addresses, i.e., which neuron fired. But I know pyNCS should be general enough to handle other strategies).

Alright, at this point while I still don't see the need for a PWConnection or PWDConnection etc. I kinda see the idea of having a new AddrSpec, but I'm still not convinced... In the end, AddrSpecs should resamble what's in the chip and so if the internal decoder considers each weight (or any other parameter) as a different input AER, that's what it should go in AddrSpec in pyNCS.

Perhaps for the moment, in order to run experiment without the "memory error", you could run a loop where you:

1. create and populate two small populations
2. create the mapping between these populations and append it to the whole mapping table
3. delete the populations (or re-populate) In this way memory occupation should be maintained some orders of magnitude smaller, depending on the number of neurons you have in the hardware. (nobody uses 10 bit resolution on synapses, as far as I can tell, never happens but I agree that even 4 bit, as is the case for some systems out there, might be a problem, and if you have 2 parameters with 4 bit resolution...)

In conclusion in my opinion addresses should just be created when needed and if also NeuronBlock needs them it should be able to create them as well, like Connections.

@sheiksadique , you seem to agree with us but what do you propose to solve this problem? :P

[eneftci]

@fabioedoardoluigialberto The problem is that NeuroChip builds the entire address space regardless of the size of the populations. So your workaround would not work.

Technically, the list of connection is a sparse representation of the connection matrix already.

After careful consideration, the easiest way to overcome this problem is to omit the dimensions related to the synaptic parameters in NHML under , so the neuron blocks do not treat each possible synapse parameter as a separate input, and let the connection fill in the missing bits related to the weights when required, i.e. online (e.g. via an additional addrSpec). These parameters are provided when connection by adding the missing parameters as arguments to the Connection class. The parameter can simply be the field id of the addrSpec.

This means rewriting the base class Connection, and its easy to make it backward compatible: no parameters means old functionality. It's elegant and simple. If no one complains, I will implement this asap.