xWuWux
commented
10 months ago Your insights into the interplay between quantization, crossover, and the efficiency of evolution in both biological and artificial systems are truly intriguing. Your observations on quantization, crossover, and activation functions raise thought-provoking questions that warrant further exploration.
The notion that quantization serves as a constraint and facilitator in biological systems is particularly fascinating. It prompts us to consider how the absence or presence of quantization might impact the representation and processing of information in artificial neural networks. Additionally, your observation regarding the perceived weakness of crossover in certain contexts raises questions about alternative optimization techniques in artificial systems.
Overall, your contributions to this discourse have opened up new avenues of investigation into the nature of evolution in neural networks. I eagerly anticipate further discussions on these topics.
Thank you for sharing your thoughts.
I wrote these comments somewhere else:
" Quantization is the enemy of evolution
It is fortunate that biological systems are heavily quantized, especially in bacteria and viruses. An atom is there or not, discrete point mutations are there or not, a plasmid is there or not . If the cost landscape where not so heavily quantized we simply wouldn’t exist. The crossover mechanism higher animals use is a weak optimizer but it does make the cost landscape less rough than what asexual microbes have to contend with.
Hence we can adapt to pathogens despite having a far longer time between generations and a far lower population count. It is also true (I think) that having a larger genome reduces the roughness of the cost landscape by giving more degrees of freedom.
In a non-quantized artificial system a perturbation in any of the basis directions gives a smoothly changing alteration in cost. A mutation in all dimensions gives a new cost that is a summary measure of multiple clues. Following mutations downhill in cost means following multiple clues about which way to go. If there were quantization in many basis directions a small movement in those directions would give you not information about whether such a movement was good or bad. You would get not clues in those directions, less clues overall, which is obviously detrimental.
A point here being that artificial evolution on digital computers can be far more efficient than biological evolutions. If you accept that back propagation is in some sense a form of evolution (at a slight stretch) then you can see that a GPU cluster can build in a few weeks the capacity to do vision that took biological evolution many millions of years to create. I have some kind of code here: https://github.com/S6Regen/Thunderbird ”
“Here’s a link for crossover being a weak optimizer: https://youtu.be/WoamKUfisVM It is actually to allow non-lethal mixability of traits. And that ends up implementing the multiplicative weights update algorithm, or so they say.
You might ask then, why are fungi not more lethal pathogens given what I said and that they reproduce by crossover. I don’t really know but I presume it has to do with crossover being a weak optimizer and maybe they have a smaller number of genes than a large animal.
A neural network can have squashing activation functions or non-squashing ones. What I noticed from my experiments with associative memory is that squashing type activation functions result in attractor states/error correction/(soft) quantization. That seems to be difficult for evolution to deal with, especially if you use hard binary threshold activation functions (the ultimate squashing function.)
One the other hand nets with non-squashing activation are very easy to evolve. And result in reasoning in sparse patterns. Of course, just because evolution favors non-squashing activation functions does not mean they are the best possible ones to use. It could be squashing ones are if you had a suitable algorithm.”