Potential issue with initialisation of recurrent weight matrix

[Matthijspals]

Hi Robert Kim,

first of all, being able to create functioning spiking networks like this is a really cool method!

I have a question about the initialisation of the network. It seems when you initialise with Dale's law, you do not balance the excitatory and inhibitory input to neurons - the expected input to neurons is a lot larger than 0, as excitation dominates. This throws of the eigenspectrum of the recurrent weight matrix (example from a network initialised with your code): There is one large outlier in the eigenspectrum, leading to the activation of the neurons exploding off in this direction - instead of the activity being in the chaotic regime as desired.

I was wondering why this initialisation was chosen and how it affects the results. E.g. in the paper you state: "By varying the gain term, we determined if highly chaotic initial dynamics were required for successful conversion", are the networks actually chaotic?

[rkim35]

Hi Matthijs,

Thank you for the comment. You are correct in that enforcing the Dale's principle does alter the initial chaotic dynamics. I looked into the effect of the gain term without enforcing the Dale's principle and it didn't affect the learning and conversion. It will be interesting to somehow remove the outlier and see how that affects RNNs with Dale's principle.

Thank you!

Robert

[Matthijspals]

Hi Kim,

thanks so much for the response! To get rid of the outlier one can simply scale the inhibitory neurons' weights to balance out the excitatory output (such that the expected input to neurons is 0). This is also done in Song, Yang and Wang, 2016. That still leaves a few small outliers, which can largely be removed by separately balancing each row of the recurrent matrix, see Rajan and Abbott, 2006

Here are some quick simulations I ran:

Would be interesting to see how this affects results!