Open arvoelke opened 5 years ago
FYI, I've now tried the L2-optimization directly on the filtered spike data (as I've been doing for my Reservoir Computing experiments), but that doesn't seem to improve things. I've tried this in conjunction with lowering the regularization, and using the VirtualEnsemble
to scale up to 3000
neurons, but still can't break the floor. This means loihi_rates
should not be the main issue here, but rather most of the neurons must be doing nearly the same thing. I think decreasing the time-step helps a little bit, but doesn't overcome the limited heterogeneity.
Any examples of using random sampling to break the synchrony without increasing noise (as suggested in https://www.nengo.ai/nengo-loihi/examples/communication_channel.html)? I tried to do this by starting the simulation with a brief pulse of random current to each neuron, but this ended up raising an error for blowing through the memory. Could each post-neuron on the the host -> chip stimulus connection be assigned dedicated heterogeneous spike-generators?
I've been trying to dig into quantifying and separating out potential sources of error. @celiasmith mentioned that @hunse has a way of improving the emulator accuracy by more accurately modelling the voltage overshoot? We could try this on the above code?
I've since tried a number of other things:
from nengo.utils.neurons import rates_isi
), within each neuron, and across all neuronsIn all cases things look really similar between nengo
and nengo_loihi
on a sinusoidal encode-decode with set_defaults
, and so I've been unable to tease apart any differences here.
By the way, dropping dt
further to 1e-5
raises an exception:
~/CTN/nengo-loihi/nengo_loihi/loihi_cx.py in discretize(self)
263 break
264 else:
--> 265 raise BuildError("Could not find appropriate wgtExp")
266 elif b_max > 1e-8:
267 b_scale = BIAS_MAX / b_max
BuildError: Could not find appropriate wgtExp
After some more investigation, the dominant source of error was found to be coming from the on/off NIF
generators that encode the input signal into spikes. One way around this is to use Principle 3, configure the inter_tau
accordingly, and set the input connection synapse to None
(see https://github.com/nengo/nengo-loihi/issues/115#issuecomment-447015272) -- but this is not a good solution at the moment because all of the interneurons and spike generators share the same tau, and special care is needed to handle the pre-synaptic and post-synaptic radii. Another way around this is to use a 2*k
-bit binary code to transmit the input signal at each time-step with 2^k
precision. A proof-of-concept is provided in #158. This reveals a new noise floor, and weirdness at 600-800 neurons, that both require further investigation.
This is what the picture looks like, comparing the two simulators, on branch binary-encoding
(#158), using neuron_type=nengo_loihi.neurons.LoihiSpikingRectifiedLinear()
, the stimulus np.sin(2*np.pi*t/sim_t)
, and reporting the nengolib.signal.nrmse
across 10 trials. The two are getting closer, but nengo_loihi
still bottoms out at around 300 neurons. Edit: It should be even better, but I think there are issues with the branch I need to sort out.
For Principle 3, the tricks from https://github.com/nengo/nengo-loihi/issues/115#issuecomment-447015272 work with the same ReLU model but on integrator-accuracy
(#124) and without any need for binary spike encoding. However this also plateaus at around 300 neurons.
(Code available on request.)
Since we quantize everything when putting it on Loihi, it's not unlikely for some neurons to end up with exactly the same parameters, and thus be redundant. I've seen this happen with at least one pair even in the case of a 100-neuron population. So this could be another effect adding to this issue. One way around it could be to add a little bit of noise to all neuron voltages, just to help them do different things. The first step, though, would be to quantify how big of an issue this really is.
Some progress was made in the investigation that I don't believe was sync'd back to this issue. In particular, all of the spike-times tend to be synchronized with one another until a burst of noise is injected into the network (at 20 seconds in the plot below). In general, the dependent spike-variability has an enormous effect on the error for large numbers of neurons.
The first step, though, would be to quantify how big of an issue this really is.
What kind of quantification are you looking for? I've done some pretty thorough experimentation above and with the emulator for my thesis. In particular, there is a noise floor of 5% that Nengo-Loihi can never scale below, even with continuous binary encoding with 8 bits of precision (0.4% error; #158), and no decode neurons. That's just a normal NEF encode-decode and spiking ReLU neurons and no extra filtering. With 1000 neurons the error is around 300% worse relative to vanilla Nengo. Furthermore, 100 Nengo neurons are the same as 1000 Nengo-Loihi neurons (see plot from previous post).
To me this is quite alarming. It should in theory continue to scale to 0% as more neurons are added, but instead is bottoms-out quite rapidly. As an engineer I would like to be able to add neurons until my function falls within spec, and right now I can barely meet spec for 100 Nengo neurons. Theorem 3.2.1 in my thesis proves rigorous criteria for ensuring the error continues to scale away even with arbitrarily high input frequencies and arbitrarily small taus, regardless of firing rates. The natural next step IMO is to see why this criteria is being violated.
The quantification I'm looking for is regarding neurons with the same parameters. If by 200-300 neurons, most new neurons generated have the exact same parameters as an existing neuron in the population, then it's unsurprising that adding more doesn't help. But if new neurons still have new parameters, then the cause could be something else.
I made a basic script to try this, and it's not finding much redundancy between neurons. However, this will largely depend on the input connections to an ensemble, and thus how the weights/biases get quantized. If you ever see a warning about "lost bits" in the weights, be very wary. Even without such a warning, there is quantization in the weights and biases that could be making a lot of neurons redundant. But based on this basic exploration, this doesn't seem to be the main factor driving what you see in your scripts above. It seems that even neurons with slightly different parameters are still having a lot of redundancy.
The other thing is that you'll likely see a lot more redunancy for higher firing rates (because of the aliasing discussed above), which means for values near +/- 1. Of course, though, if you just lower the rates, then that adds its own "noise" since the spikes are more sparse. So just lowering the max firing rates of the population doesn't seem to help much.
I tried my trick of subtracting the voltage threshold on spike, to get rid of the aliasing problem, but it doesn't appear to help (code is here). So it seems that the aliasing isn't the main cause of neuron synchrony. In general, it's not terribly surprising that synchrony is more of a problem on Loihi then in core nengo, since with discretized values there's fewer ways for neurons to become desynchronized.
I just realized, what we really need to be looking at is the diversity in the firing rates at your particular input value of 1. Since you jump straight to that input value, neurons with the same max firing rate but different gain/bias don't have time to desynchronize. I played around a bit with ramping up to an input of 1 over e.g. 0.1 seconds, and it looks like it might help a bit, but I don't have any conclusive results since changing the input signal changes the RMSE of it as well (we should maybe be using relative RMSE---dividing by the reference signal RMS---rather than absolute).
One other thing to be aware of is that the maximum firing rate of a neuron is different on Loihi, because we can't discount the overshoot from the refractory period like we do in core nengo. So for example, if the refractory period is two steps, then that means the voltage is zero for two steps, and then can fire the third step (on Loihi). So if your tau_ref=0.002
, that means the maximum firing rate is 333 Hz on Loihi, whereas it's 500 Hz in core Nengo.
Anyway, this needs more research, because there's a lot of ideas here but nothing conclusive. I think your approach of adding noise is the best one, since I'm not convinced we can get adequate heterogeneity to properly desynchronize neurons through heterogeneity alone. I'd also suggest using more diverse input signals, since I'm not sure how well just using an input of 1 will generalize (though we do often have sudden changes in signals either coming into or within our networks, which will have a synchronizing effect on neurons). I think we need some more practical examples on which to test adding some noise, maybe some of the benchmarks.
There seems to be a critical noise floor (~5% error) that can't be overcome by adding more neurons to a single layer encode-decode. The floor is already hit around 200 neurons. This could have to do with synchrony in the spikes (i.e., a lack of heterogeneity, causing two neurons to be no different than one with twice the spike height)? This synchrony could be coming from the discretization of the neuron model in time, or the quantization of its variables and time-constants in space? Or systematic differences between
loihi_rates
and the actual spike rates?I've tested this (on master) against a wide range of L2-regularization constants (from
1e-5
to1e0
). With standard Nengo, you can always continue to improve performance by increasingn_neurons
and droppingreg
according to the spike-noise (O(1/sqrt(n))
), but it doesn't seem to be the case here.Is there any way to break the ~5% floor?
Of relevance to #126: uncommenting the configuration to the
max_rates
andintercepts
defaults below makes everything worse by raising the noise floor to ~17% (not shown). I believe this is because it increases the level of spike synchrony by narrowing the range of the response curves at1
. And so further investigation might be required to determine the best defaults.On the other hand, lowering
dt
does seem to make things slightly better.I've also tried partitioning up to
2000
neurons across many sub-ensembles, and solving the decoder optimization problem across all ensembles simultaneously using myVirtualEnsemble
network, and that actually shows performance degrade past1000
neurons (code omitted, for brevity).Code for
dt
plot: