Artificial General Intelligence (AGI) – General Discussion, Resources Etc.

LLM as search engines, Transformers as big hierarchical maps/dictionaries/hash tables etc.

A discussion on a Numenta's forum regarding the LLM as search engines etc.:

https://discourse.numenta.org/t/nn-centroid-clustering-search/ https://discourse.numenta.org/t/is-billions-of-parameters-or-more-the-only-solution/10384/107

The links were from a discussion with Boris Kazachenko, who is arguing that NN are doing centroid clustering. He is the creator of the CogAlg project (originally "Cognitive Algorithm"), of which I coined that shorthand title "CogAlg" in an email back in 2010 and it was adopted in mid-late 2010s as the name in Github when the project started to get developed in code. https://github.com/boris-kz/CogAlg

The popularity of ChatGPT and GPT3 used for all kinds of queries suggeted that.

A few thoughts from me from a discussion about CogAlg on 20.2:

https://github.com/boris-kz/CogAlg/blob/master/frame_2D_alg/Illustrations/generic%20graph.drawio.png

Todor: An interpretation and association, from my POV and own understanding of some general cognitive processes: that may be (building, an element for building) a graph that is tracing a chain of matches/mapping. [Regarding a CogAlg "Cognitive Node" and graphs. Well, this is a generality.]

One of the core abstract structure and process of intelligence is the mapping between different representation[s] and coordinates and the sequence of mappings in between the path, train of thought, chain of intermediate/incremental representations, like the "associative thinking" in more high level view of human cognition.

Clustering, connectivity clustering in general are also about that.

The above is present in ANN in transformers as well, where transformers look like huge and complex multilevel, multi range, multi step maps/hash tables - they even have Query/Key/Value in their definitions and terminology.

Also I'd probably prefer a term like "cognitive element" instead of "neuron", although the growing of axons and dendrites, vertices and nodes of the graph, may make a good analogy with the real neurons and possibly it is not bad a term for that reason. (...) Re transformers - ... I meant an abstract essence of their function, application; not the way they build the representation used to execute the function or that they are doing it exactly as you do.

One thing that they do though is hierarchically finding the correlation between adjacent input elements in sequences of bigger ranges, a bigger regions of adjacent tokens.

Also for now I don't know how the "tokens" in the transformers -- possibly RNN's inputs as well -- map to your scheme.

Is there a counterpart to transofmers'/RNN's "tokens"/token-dictionary in your algorithm?

I can assume the lowest level patterns could serve for such tokens, or some set of them with some "limited" properties (a complexity measure, number of patterns etc. which fit in a given memory limitation etc. - at least that's what I'd do).

The first levels build the most basic dictionary of the patterns which are encountered first and fall within a certain complexity range, however each level has dictionaries and such ranges, and possibly part of the "grammar" is inter-level, it may "spill" between the levels.

(Well, sure this is too abstract without working with precise patterns and structures.) B: Right, this is a hierarchical graph, with multiple composition levels B: In terms of results, transformers do learn graphs, through positional encoding. Key-value pairs are links/edges. But the main question is how they learn them, which is totally different. B: Nothing in NNs is similar in terms of core learning process: they do centroid-based clustering: comparison-last, I do connectivity-based: comparison-first. That's the main point. Dictionaries, grammar: these are all language-inspired distractions. B: Related discussion: https://discourse.numenta.org/t/nn-centroid-clustering-search/10443/28

T: The set of same-level patterns with the patterns to which they are connected are a map, or "a dictionary". The set and connections in all senses that this could be defined in your algorithm: "above average match" etc., adjacency (all adjacent to some anchor), all-miss; edge-patterns, "background"-patterns/negative/positive (sorry I'm not familiar with the details of the current state of the system) A "dictionary" or map is mathematical, a function, a set of mappings. A grammar is the set of rules for comparison and creating new nodes, graphs, links - that's the algorithm which builds the graph.

T: Thanks for the new picture, the first one was transparent and had issues when clicked.

Can you elaborate what's the difference between range-match and extended match?

B: There is no need for these terms, they only make things look more complicated. Same thing, just more suggestive. It means match over incremented comparison range. It's a rng+ vs der+ fork in sub_recursion_g: https://github.com/boris-kz/CogAlg/blob/master/frame_2D_alg/agg_recursion.py

Twenkid / Vsy-Jack-Of-All-Trades-AGI-Bulgarian-Internet-Archive-And-Search-Engine

Artificial General Intelligence (AGI) – General Discussion, Resources Etc. #21

Links

LLM as search engines, Transformers as big hierarchical maps/dictionaries/hash tables etc.