[SP] Local inhibition tiebreaker

[Oblynx]

Local inhibition among a small synapse population presents a challenge in stabilizing output sparsity.

Currently the k-winners number isn't strictly enforced; if multiple minicolumns tie at the kth place, all of them will become active. This is ok when the population of competing neurons is large, but, if the inhibition area becomes too small or the synapse formation probability is too low, the total output sparsity is very unstable.

TODO

• [ ] Implement tiebreaker logic
• [ ] Enable it conditionally only when it's probably required (because it will have a performance impact)

Tiebreaker logic

Z(y), activate(o)= o .>= Z(o) must become smarter.

1. Add a random perturbation to each element: o .>= Z(o) -> o + rand(Float32,size(o)) .> Z(o)
2. Fractional thresholds from Z(y):

Let a: number of neurons per area. We only want to activate 1 of the columns that tie at the kth spot. Then Z(y) = overlap(k)+t, where t maximizes the probability that only 1/a samples in [0,1) be > t.

Let this probability be p(X=1) where X:= num of {a draws in [0,1)} > t. We need t that maximizes p(X=1):

p(X=1):= a * (1-t) * t^(a-1)
∂p/∂t = a(-t^(a-1) +(1-t)(a-1)t^(a-2)) = a ((a-1)(1-t) - t) t^(a-2)
∂p/∂t = 0 ⇔ 
{a>2}:  t=0 || a-1 +t-ta -t = 0 ⇔ t=0 || t=(a-1)/a

but

t:=0 ⇒ p(X=1)= 0 ⇒ t = (a-1)/a

therefore

t:= (a-1)/a ⇒ p(X=1)= ((a-1)/a)^(a-1)

[Oblynx]

For local inhibition the calculated value of t is best. For global inhibition, t ~ 0.5 works out best (keeping 1/2 of the tying minicolumns). I've probably misunderstood what ties mean.

k = ceil(Int, s*a()) gives another source of uncertainty