SwuduSusuwu
commented
1 month ago Just as a human who watches a video performs the following tasks:
Retinal nervous tissue has raw photons as inputs, and compresses such into splines + edges + motion vectors (close to how computers produce splines through edge detection plus do motion estimation, which is what the most advanced traditional codecs such as x264 do to compress)
passes millions/billions of those (through optic nerves) to the V1 visual cortex (as opposed to just dump those to a .mp4, which is what computers do),
which groups those to produce more abstract, sparse, compressed forms (close to a simulator's meshes / textures / animations),
passes those to V1 visual cortex,
which synthesizes those into more abstract datums (such as a simulator's specific instances of individual humans, tools, or houses),
and passes the most abstract (from V2 visual cortex) plus complex (from V1 visual cortex) to hippocampus (which performs temporary storage tasks while active, and, at rest, encodes this to neocortex).
Just as humans can use the neocortex's stored resources for synthesis of new animations/visuals,
so too could artificial central nervous systems (run on CPU or GPU) setup synapses to allow to compress gigabytes of visuals from videos into a few kilobytes of text (the hippocampus has dual uses, so can expand the compressed "text" back to good visuals).
2 routes to this: 1) Unsupervised CNS (fitness function of synapses is just to compress as much as can, plus reproduce as much of originals as can for us; layout of synapses is somewhat based on human CNS). 2) Supervised CNS (various sub-CNS's for various stages of compression, with examples used to setup the synapses for those various stages to compress, such as "raw bitmap -> Scalable Vector Graphics + partial texture synthesis", "video (vector of bitmaps) -> motion estimation vectors", "Scalable Vector Graphics/textures + motion estimation vectors -> mesh generation + animation + full texture synthesis", plus the inverses to decompress).
Humans process more complex experiences than just visual senses: humans also have layers of various auditory cortex tissues, so that sound compresses, plus a thalamus (which merges your various senses, thus the hippocampus has both audio+visual to access and compress, which, for a computer, would be as if you could all speech + lip motions down to the subtitles (.ass)).
Sources: https://wikipedia.org/wiki/Visual_cortex, Neuroscience for Dummies plus various such books.
Not sure if the arxiv.org articles[1][2] are about this, but if not, could produce this for us if someone sponsors.
- [1] A Computationally Efficient Neural Video Compression Accelerator Based on a Sparse CNN-Transformer Hybrid Network
- [2] Advances In Video Compression System Using Deep Neural Network: A Review And Case Studies
Because the arxiv.org pages do not list compression ratios, have doubts, but if someone has done this, won't waste resources to produce what someone else has. Expected compression ratios: parse inputs of 1024*1280@60fps (2.6gbps), output text at a few kbps, reproduce originals from text (with small losses.)
cxx/ClassResultList.cxx has correspondances to neocortex. which is what humans use as databases. cxx/VirusAnalysis.hxx & cxx/VirusAnalysis.cxx + cxx/AssistantCns.hxx & cxx/AssistantCns.cxx has some correspondances to Broca's area (produces language through recursive processes), Wernicke’s area (parses languages through recursive processes), plus hippocampus (integration to the
neocortex+ imagination through various regions). cxx/ClassCns.cxx (HSOM + apxr_run) is just templates for general-purpose emulations of neural mass. https://www.deviantart.com/dreamup has some equivalences to how visual cortex +Broca's area+hippocampus+ text inputs = texture generation + mesh generation visual outputs (from text inputs.) To have autonomous robots produce all goods for us [ https://swudususuwu.substack.com/p/program-general-purpose-robots-autonomous ] would require visual cortex (parses inputs from photoreceptors) + auditory cortex (parses inputs from malleus + cortical homunculus (parses inputs from touch sensors) + thalamus (merges information from various classes of sensors, thus the robot balances + produces maps) + hippocampus (uses outputs from sensors to setupneocortex, plus, runs inverses this for synthesis of new scenarios) +Wernicke's region/Broca's regions(recursive language processes)