Implement SP with apical connections to other active columns

[breznak]

Why:

• allows SP to work as associative-memory (like SOM)
• better quality of produced SDRs, #155 , better noice resistance

Question:

• same as TM with cellsPerColumn = 1 ?

Implementation:

• after #93 , uses Connections' mechanism
• I think @fergalbyrne had similar idea and implementaion (in Lua?) on HTM Discourse

EDIT: forum discussion at https://discourse.numenta.org/t/replace-inhibition-with-proximal-horizontal-inhibitory-connections/5538

[ctrl-z-9000-times]

@breznak, Could you please post a link to @fergalbyrne 's work?

Here is my goto reference on apical dendrites: (Larkum 2013)

I have my own hypotheses about apical dendrites. I think that apical dendrites are a part of attention.

[breznak]

I think it'll be somewhere here https://leanpub.com/realsmartmachines/read First ideas were pblished and discussed at HTM discourse boards.

I can have a deeper look later for the Fergal's work.

Basically, what I want to do:

• in cortex, SP+TM mechanisim is in one layer (separated algorithms here just for implementation)
• SP should work as an associative memory (ie SOM)
• I think SP should work as TM with 1 cell-per-column -> first order memory

[Thanh-Binh]

Do you know if the current SP includes or can be extended to the idea „union temporal pooler“ in htmresearch?

[ctrl-z-9000-times]

the idea „union temporal pooler“ in htmresearch?

Im'm not certain what exactly the "Union TP" is, but I don't think that the current SP does this. It can likely be extended to do this, but that isn't part of the plan at the moment.

[Thanh-Binh]

David: i do not know in details but Union TP is subclass of SP and from Numenta points of view it is very interesting for sequence recognition. I will look at it if I have time to know how good it is

[ctrl-z-9000-times]

Ok, i think i am building something similar. Its called a ColumnPooler and it is in issue #94. It does not have apical dendrites though

[Thanh-Binh]

@ctrl-z-9000-times: if I understood you well your „ColumnPooler.cpp“ corresponds to Numenta htmresearch union_temporal_pooler.py, BUT motto ColumnPooler.py. Am I right here?

[breznak]

ColumnPooler.py.

I think it is the implementation of CP from py, right? And also, CP should be an addition to SP,TM. The logic is quite well separated on high-level

• SP: tranfsforms your data into SDR, learns spatial similarity
• TM: takes SDRs and learns temporal sequences
• CP: takes active outputs and predictions from TM and forms invariant representation of the object

So you can solve MNIST with SP, but I think it should be (image->SP->CP->Classifier)

PS: please move discussions about CP to its relevant thread (alr exists), we'll leave this topic for SP ~ SOM and apical connections

[breznak]

topic for SP ~ SOM and apical connections

I'd like to revive my work on the SP with horizontal connectivity. @ctrl-z-9000-times I'm now thinking how/if this idea is different to the ColumnPooler? both add (CP heavily) the horizontal connections among columns. Is this similar/same to your work on CP/SP that uses CP?

[ctrl-z-9000-times]

I don't entirely understand the SOM, but I believe it's the same idea as a CP.

The CP uses horizontal connections to do two competitions: first all of cells with both feed-forward-proximal input AND active distal dendrites (which get input from the horizontal connections) compete to activate. Then if the CP has not reached maximum sparsity, there is a second competition in which all cells (including unpredicted cells) compete to activate. In this way the predictive inputs can influence the results without overriding the feed forward input.

• forms invariant representation of the object

The ColumnPooler, as written by Numenta, will not form invariant representations. They do it by freezing certain cellular activities while an object is presented during training. I have a theory about how it happens in the brain with an online algorithm: based on the paper (Kropff & Treves 2008) and it relates to only the SP.

I have a reproduction of the Kropff & Treves 2008 paper at: https://github.com/ctrl-z-9000-times/KropffTreves2008_reproduction/blob/master/README.md I direct your attention figures 3 & 4, which show the untrained & trained behavior of the model, respectively. Notice that before training the cells respond to small & disjointed areas, and after training they respond to large contiguous areas of the input. This is the concept of invariance applied to a spatial location.

[breznak]

The ColumnPooler, as written by Numenta, will not form invariant representations. They do it by freezing certain cellular activities while an object is presented during training.

this is a bummer, I had not known that. Will need to have a look.

The CP uses horizontal connections to do two competitions: first all of cells with both feed-forward-proximal input AND active distal dendrites (which get input from the horizontal connections) compete to activate. Then if the CP has not reached maximum sparsity, there is a second competition in which all cells (including unpredicted cells) compete to activate. In this way the predictive inputs can influence the results without overriding the feed forward input.

What I describe in #287 .4/ might be something else.

• Cells form (from distribution pool) connections to local and further peer cells.
• Hebbian rule is applied: when both fire, connection strenghtens, otherwise decays.
• 4.B associative recall (this is same as you describe): a cell can be active from a peer cell (active distal dendrites, in addition to feed-forward proximal input)
• 4.A proximal horizontal/peer connections implement inhibition neighbourhood