Closed keegandent closed 2 weeks ago
There's additional subtlety, because you should only fire if the charge value exceeds your threshold at the end of the integration cycle. So if my threshold is 10 and my current potential is 0, then receiving three spikes with weights of 5 and one with a weight of -9 means at the end of the integration cycle, I shouldn't fire, but my potential should be 6.
Different neuroprocessors have addressed this in different ways:
DANNA: There was an explicit max potential and min potential. In the above example, if the maximum potential is 13, then the three spikes with a weight of 5 would add up to 13 and not 15, and then the negative spike would set the potential to be 4. Obviously, now the order of spikes matters and is an additional potential mismatch between simulator and neuroprocessor. In DANNA, there were 16 subcycles within each integration cycle, and when Adam wrote the simulator, he accounted for that, which removed the nondeterminism. It slowed the simulator down, of course.
RAVENS: If I'm not mistaken, the maximum fan-in of Ravens means that you can calculate the min and max potentials a-priori, and set min and max thresholds accordingly.
I'm not sure what I advocate here — we can add parameters for min_potential and max_potential and then pray it doesn't bite us with non-determinism too badly. If we set it high enough, it won't. We can also have the simulator identify non-determinism, and flag it to the user, so that they can modify the min/max potentials.
That can be done with graph analysis too (which is pretty much what RAVENS does), but graph analysis will be overly conservative.
Thoughts?
On Aug 14, 2024, at 12:33 AM, Keegan Dent @.***> wrote:
(best viewed on GitHub site/app for TeX formatting)
From my review of the RISP processor code, it seems that, in a scenario where there is no leak on a neuron and non_negative_charge is false, a neuron could, in theory, accumulate infinite negative charge. This doesn't happen on the positive end because there is a maximum threshold for neurons, and therefore they must eventually fire with continued application of positive charge.
Obviously this is not a behavior I can replicate in hardware with finite integer widths. At the moment I only bother to provide enough width to accommodate $[- 2 ^ {n - 1}, 2 ^ {n - 1} - 1]$ where $n = \left\lceil \log_{2}{ \left( \text{threshold}+\text{threshold_inclusive} \right)} \right\rceil$. For neurons with small thresholds, this will cause significant divergence from the Framework when they accumulate negative charge.
The way I see it, there are two ways to address this problem.
Thanks
— Reply to this email directly, view it on GitHubhttps://github.com/TENNLab-UTK/fpga/issues/17, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AGEAIREDD52V4WZ6OWGZEKTZRLM3JAVCNFSM6AAAAABMPQ2XB6VHI2DSMVQWIX3LMV43ASLTON2WKOZSGQ3DIOBXHE2DSOI. You are receiving this because you were assigned.Message ID: @.***>
Wouldn't it be easy to get around this nondeterminism that is based on the order in which charge is received from incoming synapses by just restricting the summed potential at the end of an integration cycle as opposed to during the integration cycle? This could be done in the same loop that checks if potential is >= threshold or that does RISP leak and non_negative_charge.
I think a RISP min_potential parameter that replaces the non_negative_charge parameter would be interesting and very hardware-friendly. On the algorithmic and training side, allowing a min_potential=-infinity neuron to accumulate infinite negative charge debt such that it is extremely difficult to ever spike again seems useful in some cases but harmful in others. In other words, I think it would be the classic optimization experiment we do where we give EONS a set of parameters to work with and get inconclusive results that are sort of all over the board. In other words, I speculate that a defined min_potential would not be significantly detrimental to RISP network performance.
As Keegan mentioned, I see no need for a max_potential parameter as the threshold does this at the end of each integration cycle already.
Let me see if I understand from a hardware point of view (ha ha). From the network, you can deduce what the maximum and minimum potential values will be during an integration cycle. You can account for that during integration, and then you can restrict the values that get stored from cycle to cycle. I was assuming that you are using the same "register" (for lack of a better word) to both store the potential at the end of the cycle, and calculate the potential during the cycle.
On Aug 14, 2024, at 10:40 AM, Bryson Gullett @.***> wrote:
Wouldn't it be easy to get around this nondeterminism that is based on the order in which charge is received from incoming synapses by just restricting the summed potential at the end of an integration cycle as opposed to during the integration cycle? This could be done in the same loop that checks if potential is >= threshold or that does RISP leak and non_negative_charge.
I think a RISP min_potential parameter that replaces the non_negative_charge parameter would be interesting and very hardware-friendly. On the algorithmic and training side, allowing a min_potential=-infinity neuron to accumulate infinite negative charge debt such that it is extremely difficult to ever spike again seems useful in some cases but harmful in others. In other words, I think it would be the classic optimization experiment we do where we give EONS a set of parameters to work with and get inconclusive results that are sort of all over the board. In other words, I speculate that a defined min_potential would not be significantly detrimental to RISP network performance.
As Keegan mentioned, I see no need for a max_potential parameter as the threshold does this at the end of each integration cycle already.
— Reply to this email directly, view it on GitHubhttps://github.com/TENNLab-UTK/fpga/issues/17#issuecomment-2288980774, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AGEAIRGUJOWRZHXKWASUSRTZRNT6NAVCNFSM6AAAAABMPQ2XB6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBYHE4DANZXGQ. You are receiving this because you were assigned.Message ID: @.***>
In hardware, the sum of weighted spikes is done using combinational logic (all happens simultaneously in digital addition circuits without the need of a register/memory). The number of bits/wires for the summed value is calculated from the network before deployment. It is ideally calculated using the max and min combination of incoming synapse weights (however, it is instead currently based on the number of bits used for each incoming synapse weight for the purpose of keeping the code modular). For a given neuron, the only register/memory is for its accumulated potential that is updated with the output of the sum circuit at the end of an integration cycle (a clock rising edge). At this rising clock edge (the end of the integration cycle), a simple check that restricts the neuron potential to min_potential could be added. Make sense?
@jimplank, @BGull00 is correct. The intermediate summation within the neuron is bounded by the number of inputs and synapse weight limits. The bigger issue is that over successive cycles, the neuron's charge can potentially drift indefinitely into the negatives, so I need to bound this for the post-integration charge memory.
I think a RISP min_potential parameter that replaces the non_negative_charge parameter would be interesting and very hardware-friendly. On the algorithmic and training side, allowing a min_potential=-infinity neuron to accumulate infinite negative charge debt such that it is extremely difficult to ever spike again seems useful in some cases but harmful in others.
I didn't even think about this, but replacing that binary flag with a min value is a great idea. The FPGA implementation can simply reject anything that won't work with integer hardware. I would say it should be restricted to $[-\infty, 0]$, much the same way spike_value_factor
can't be less than 1. Otherwise, we would need to start overriding fire and leak behavior which sounds like a mess.
(best viewed on GitHub site/app for TeX formatting)
From my review of the RISP processor code, it seems that, in a scenario where there is no leak on a neuron and
non_negative_charge
isfalse
, a neuron could, in theory, accumulate infinite negative charge. This doesn't happen on the positive end because there is a maximum threshold for neurons, and therefore they must eventually fire with continued application of positive charge.Obviously this is not a behavior I can replicate in hardware with finite integer widths. At the moment I only bother to provide enough width to accommodate $[- 2 ^ {n - 1}, 2 ^ {n - 1} - 1]$ where $n = \left\lceil \log_{2}{ \left( \text{threshold}+\text{threshold\_inclusive} \right)} \right\rceil$. For neurons with small thresholds, this will cause significant divergence from the Framework when they accumulate negative charge.
The way I see it, there are two ways to address this problem.
non_negative_charge==true
for FPGA networks. We only support discrete networks anyway, so perhaps adding another restriction isn't overly burdensome to the user.Thanks