Rare hang

[HyperCodec]

Not sure how this is happening but in extremely rare circumstances it is possible to hang indefinitely. See #57 workflow run for more info.

My guess is there is one very small outlying situation that causes a rwlock to be locked and used by a child node, but this shouldn't be possible with the well-tested circulation prevention algorithm. This definitely requires further debugging, but it is so rare and obscure that it is difficult to catch it and the details about what happened.

[Bowarc]

After a couple of generation, it just stops and i don't know why.

It appears to be 100% of the time with my current test, I pushed it at https://github.com/Bowarc/doodlai_jump/tree/ea955a6b681fcbaa2a4e3ec6d81f14970d5414b7

(The /ring package is responsible for training (the one hanging after a couple of generations), game is a lib for a rly simple version of doodle jump and display is to see the ai play)

[HyperCodec]

After a couple of generation, it just stops and i don't know why.

It appears to be 100% of the time with my current test, I pushed it at https://github.com/Bowarc/doodlai_jump/tree/ea955a6b681fcbaa2a4e3ec6d81f14970d5414b7

(The /ring package is responsible for training (the one hanging after a couple of generations), game is a lib for a rly simple version of doodle jump and display is to see the ai play)

Hmm so it's probably something with a recursive RwLock. I'll have to look into it further. It's probably some internal function causing a cyclic neuron dependency (like DFS not working or something).

[HyperCodec]

Btw @Bowarc can you use the serde feature to dump a json (or ron) file on the generation that hangs? (Probably the easiest way to do this would be to overwrite the same file with each generation and then stop the program when it hangs)

[Bowarc]

Ok, i'll do that tomorrow

[Bowarc]

Well, i stayed up longer than expected 😅 Here is the dna of every genome of a sim that froze

DNA { network: NeuralNetworkTopology { input_layer: [RwLock { data: NeuronTopology { inputs: [], bias: 0.19538373, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.9611819, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.5509694, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.31042653, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.9654784, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.81183213, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.86611843, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.9298546, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.8283311, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.8759112, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.4996699, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.5423544, activation: linear_activation
 }, poisoned: false, .. }], hidden_layers: [], output_layer: [RwLock { data: NeuronTopology { inputs: [], bias: 0.010660529, activation: sigmoid
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [(Input(6), -0.1170296)], bias: 0.39411813, activation: sigmoid
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [(Input(7), -0.8857193), (Input(11), -0.97913766), (Input(2), 0.2923255), (Input(1), 0.26824117), (Input(2), -0.8934064), (Input(10), -0.19709682), (Input(9), -0.92098737), (Input(6), -0.9772694), (Input(7), 0.08727813), (Input(3), -0.61651254), (Input(9), 0.42674088), (Input(7), -0.801528), (Input(1), 0.6078919)], bias: 0.7993354, activation: sigmoid
 }, poisoned: false, .. }], mutation_rate: 0.01, mutation_passes: 3 } }
DNA { network: NeuralNetworkTopology { input_layer: [RwLock { data: NeuronTopology { inputs: [], bias: 0.19538373, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.9611819, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.5509694, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.31042653, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.9654784, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.81183213, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.86611843, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.9298546, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.8283311, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.8759112, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.4996699, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.5423544, activation: linear_activation
 }, poisoned: false, .. }], hidden_layers: [RwLock { data: NeuronTopology { inputs: [(Input(3), -0.5625169)], bias: 0.59482414, activation: sigmoid
 }, poisoned: false, .. }], output_layer: [RwLock { data: NeuronTopology { inputs: [(Hidden(0), 0.9505495)], bias: 0.010660529, activation: sigmoid
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [(Input(6), -0.1170296)], bias: 0.39411813, activation: sigmoid
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [(Input(7), -0.8857193), (Input(11), -0.97913766), (Input(2), 0.2923255), (Input(1), 0.26824117), (Input(2), -0.8934064), (Input(10), -0.19709682), (Input(9), -0.92098737), (Input(6), -0.9772694), (Input(7), 0.08727813), (Input(3), -0.61651254), (Input(9), 0.42674088), (Input(7), -0.801528), (Input(1), 0.6078919)], bias: 0.7993354, activation: sigmoid
 }, poisoned: false, .. }], mutation_rate: 0.01, mutation_passes: 3 } }
DNA { network: NeuralNetworkTopology { input_layer: [RwLock { data: NeuronTopology { inputs: [], bias: 0.19538373, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.9611819, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.5509694, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.31042653, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.9654784, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.81183213, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.86611843, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.9298546, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.8283311, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.8759112, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.4996699, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.5423544, activation: linear_activation
 }, poisoned: false, .. }], hidden_layers: [], output_layer: [RwLock { data: NeuronTopology { inputs: [], bias: 0.010660529, activation: sigmoid
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [(Input(6), -0.1170296)], bias: 0.39411813, activation: sigmoid
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [(Input(7), -0.8857193), (Input(11), -0.97913766), (Input(2), 0.2923255), (Input(1), 0.26824117), (Input(2), -0.8934064), (Input(10), -0.19709682), (Input(9), -0.92098737), (Input(6), -0.9772694), (Input(7), 0.08727813), (Input(3), -0.61651254), (Input(9), 0.42674088), (Input(7), -0.801528), (Input(1), 0.6078919)], bias: 0.7993354, activation: sigmoid
 }, poisoned: false, .. }], mutation_rate: 0.01, mutation_passes: 3 } }
DNA { network: NeuralNetworkTopology { input_layer: [RwLock { data: NeuronTopology { inputs: [], bias: 0.19538373, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.9611819, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.5509694, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.31042653, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.9654784, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.81183213, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.86611843, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.9298546, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.8283311, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.8759112, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.4996699, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.5423544, activation: linear_activation
 }, poisoned: false, .. }], hidden_layers: [], output_layer: [RwLock { data: NeuronTopology { inputs: [], bias: 0.010660529, activation: sigmoid
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [(Input(6), -0.1170296)], bias: 0.39411813, activation: sigmoid
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [(Input(7), -0.8857193), (Input(11), -0.97913766), (Input(2), 0.2923255), (Input(1), 0.26824117), (Input(2), -0.8934064), (Input(10), -0.19709682), (Input(9), -0.92098737), (Input(6), -0.9772694), (Input(7), 0.08727813), (Input(3), -0.61651254), (Input(9), 0.42674088), (Input(7), -0.801528), (Input(1), 0.6078919)], bias: 0.7993354, activation: sigmoid
 }, poisoned: false, .. }], mutation_rate: 0.01, mutation_passes: 3 } }
DNA { network: NeuralNetworkTopology { input_layer: [RwLock { data: NeuronTopology { inputs: [], bias: 0.19538373, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.9611819, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.5509694, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.31042653, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.9654784, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.81183213, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.86611843, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.9298546, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.8283311, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.8759112, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.4996699, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.5423544, activation: linear_activation
 }, poisoned: false, .. }], hidden_layers: [], output_layer: [RwLock { data: NeuronTopology { inputs: [], bias: 0.010660529, activation: sigmoid
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [(Input(6), -0.1170296)], bias: 0.39411813, activation: sigmoid
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [(Input(7), -0.8857193), (Input(11), -0.97913766), (Input(2), 0.2923255), (Input(1), 0.26824117), (Input(2), -0.8934064), (Input(10), -0.19709682), (Input(9), -0.92098737), (Input(6), -0.9772694), (Input(7), 0.08727813), (Input(3), -0.61651254), (Input(9), 0.42674088), (Input(7), -0.801528), (Input(1), 0.6078919)], bias: 0.7993354, activation: sigmoid
 }, poisoned: false, .. }], mutation_rate: 0.01, mutation_passes: 3 } }
DNA { network: NeuralNetworkTopology { input_layer: [RwLock { data: NeuronTopology { inputs: [], bias: 0.19538373, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.9611819, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.5509694, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.31042653, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.9654784, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.81183213, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.86611843, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.9298546, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.8283311, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.8759112, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.4996699, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.5423544, activation: linear_activation
 }, poisoned: false, .. }], hidden_layers: [], output_layer: [RwLock { data: NeuronTopology { inputs: [], bias: 0.010660529, activation: sigmoid
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [(Input(6), -0.1170296)], bias: 0.39411813, activation: sigmoid
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [(Input(7), -0.8857193), (Input(11), -0.97913766), (Input(2), 0.2923255), (Input(1), 0.26824117), (Input(2), -0.8934064), (Input(10), -0.19709682), (Input(9), -0.92098737), (Input(6), -0.9772694), (Input(7), 0.08727813), (Input(3), -0.61651254), (Input(9), 0.42674088), (Input(7), -0.801528), (Input(1), 0.6078919)], bias: 0.7993354, activation: sigmoid
 }, poisoned: false, .. }], mutation_rate: 0.01, mutation_passes: 3 } }
DNA { network: NeuralNetworkTopology { input_layer: [RwLock { data: NeuronTopology { inputs: [], bias: 0.19538373, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.9611819, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.5509694, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.31042653, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.9654784, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.81183213, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.86611843, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.9298546, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.8283311, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.8759112, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.4996699, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.5423544, activation: linear_activation
 }, poisoned: false, .. }], hidden_layers: [], output_layer: [RwLock { data: NeuronTopology { inputs: [], bias: 0.010660529, activation: sigmoid
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [(Input(6), -0.1170296)], bias: 0.39411813, activation: sigmoid
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [(Input(7), -0.8857193), (Input(11), -0.97913766), (Input(2), 0.2923255), (Input(1), 0.26824117), (Input(2), -0.8934064), (Input(10), -0.19709682), (Input(9), -0.92098737), (Input(6), -0.9772694), (Input(7), 0.08727813), (Input(3), -0.61651254), (Input(9), 0.42674088), (Input(7), -0.801528), (Input(1), 0.6078919)], bias: 0.7993354, activation: sigmoid
 }, poisoned: false, .. }], mutation_rate: 0.01, mutation_passes: 3 } }
DNA { network: NeuralNetworkTopology { input_layer: [RwLock { data: NeuronTopology { inputs: [], bias: 0.19538373, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.9611819, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.5509694, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.31042653, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.9654784, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.81183213, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.86611843, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.9298546, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.8283311, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.8759112, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.4996699, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.5423544, activation: linear_activation
 }, poisoned: false, .. }], hidden_layers: [], output_layer: [RwLock { data: NeuronTopology { inputs: [], bias: 0.010660529, activation: sigmoid
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [(Input(6), -0.1170296)], bias: 0.39411813, activation: sigmoid
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [(Input(7), -0.8857193), (Input(11), -0.97913766), (Input(2), 0.2923255), (Input(1), 0.26824117), (Input(2), -0.8934064), (Input(10), -0.19709682), (Input(9), -0.92098737), (Input(6), -0.97373414), (Input(7), 0.08727813), (Input(3), -0.61651254), (Input(9), 0.42674088), (Input(7), -0.801528), (Input(1), 0.6078919)], bias: 0.7993354, activation: sigmoid
 }, poisoned: false, .. }], mutation_rate: 0.01, mutation_passes: 3 } }
DNA { network: NeuralNetworkTopology { input_layer: [RwLock { data: NeuronTopology { inputs: [], bias: 0.19538373, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.9611819, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.5509694, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.31042653, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.9654784, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.81183213, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.86611843, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.9298546, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.8283311, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.8759112, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.4996699, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.5423544, activation: linear_activation
 }, poisoned: false, .. }], hidden_layers: [], output_layer: [RwLock { data: NeuronTopology { inputs: [], bias: 0.010660529, activation: sigmoid
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [(Input(6), -0.1170296)], bias: 0.39411813, activation: sigmoid
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [(Input(7), -0.8857193), (Input(11), -0.97913766), (Input(2), 0.2923255), (Input(1), 0.26824117), (Input(2), -0.8934064), (Input(10), -0.19709682), (Input(9), -0.92098737), (Input(6), -0.9772694), (Input(7), 0.08727813), (Input(3), -0.61651254), (Input(9), 0.42674088), (Input(7), -0.801528), (Input(1), 0.6078919)], bias: 0.7993354, activation: sigmoid
 }, poisoned: false, .. }], mutation_rate: 0.01, mutation_passes: 3 } }
DNA { network: NeuralNetworkTopology { input_layer: [RwLock { data: NeuronTopology { inputs: [], bias: 0.19538373, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.9611819, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.5509694, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.31042653, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.9654784, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.81183213, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.86611843, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.9298546, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.8283311, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.8759112, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.4996699, activation: linear_activation
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [], bias: 0.5423544, activation: linear_activation
 }, poisoned: false, .. }], hidden_layers: [], output_layer: [RwLock { data: NeuronTopology { inputs: [], bias: 0.010660529, activation: sigmoid
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [(Input(6), -0.1170296)], bias: 0.39411813, activation: sigmoid
 }, poisoned: false, .. }, RwLock { data: NeuronTopology { inputs: [(Input(7), -0.8857193), (Input(11), -0.97913766), (Input(2), 0.2923255), (Input(1), 0.26824117), (Input(2), -0.8934064), (Input(10), -0.19709682), (Input(9), -0.92098737), (Input(6), -0.9772694), (Input(7), 0.08727813), (Input(3), -0.61651254), (Input(9), 0.42674088), (Input(7), -0.801528), (Input(1), 0.6078919)], bias: 0.7993354, activation: sigmoid
 }, poisoned: false, .. }], mutation_rate: 0.01, mutation_passes: 3 } }

made a new commit if you wanna check it 8d75367

[HyperCodec]

Well, i stayed up longer than expected 😅 Here is the dna of every genome of a sim that froze...

Something I noticed here is that there are a lot inputs for Input layer neurons on one of the output neurons for each genome. I doubt this is just a result of evolution or something because of that huge ratio between it and the other neurons. Probably another issue to fix.

Anyways, I created #61 for the duplicate neuron references that are in the inputs to that output neuron.

[HyperCodec]

Merged #62, which is the main suspect of this issue.

@Bowarc Can you try to run with neat = { git = "https://github.com/hypercodec/neat", branch = "dev", features = ["whateveryouhadbefore"] } and see if it still hangs?

[Bowarc]

I've now tested over 3k generations, it seems to be stable, thank you for the fix (i had ["crossover", "rayon", "serde"] as features)

[HyperCodec]

Np

[HyperCodec]

You can use dev branch for now but it's not a good branch to stay on bc of large api changes, def change back to stable after next release.

[Bowarc]

Alright, thanks !

[Bowarc]

Oh I swapped to DivisionReproduction (i was on CrossoverReproduction before) and first try after about 125 generations it deadlocked

Here is the simulation data sim.backup.txt

I tried more tests, even went back to CrossoverReproduction w/ crossover_pruning_nextgen but it appears to be deadlocking 100% of the time again. After more tests i found that if i have a too low number of genome per generation (<100) it deadlocks in about 10/100 gens Seems fine with 1000 genomes / gen

DivisionReproduction hangs after a bit with 1000 genomes, here is the sim data: sim.backup.txt

[HyperCodec]

I swapped to DivisionReproduction (i was on CrossoverReproduction before) and first try after about 125 generations it deadlocked

I tried more tests, even went back to CrossoverReproduction w/ crossover_pruning_nextgen but it appears to be deadlocking 100% of the time again. After more tests i found that if i have a too low number of genome per generation (<100) it deadlocks in about 10/100 gens ~Seems fine with 1000 genomes / gen~

DivisionReproduction hangs after a bit with 1000 genomes, here is the sim data: sim.backup.txt

Interesting that it made it through ~3k generations without deadlocking when on CrossoverReproduction the first time but not the second time. Perhaps you just got really lucky on that run. At least this eliminates the premise that the double neuron input thing is causing a deadlock (although it probably also was causing a deadlock in and of itself, maybe there are just multiple issues here)

[HyperCodec]

After looking through your backup files, I noticed that there are still duplicate inputs. I am not sure this time how they are being made.

[Bowarc]

While testing performances & learning curves, i found out that high mutation rate (=>0.1) deadlocks in less than 50 gens 100% of the time, and now that i think of it, it might be the difference between me saying that it looks good and me saying that it doesn't work again

Example:

pub const NB_GAMES: usize = 3;
pub const GAME_TIME_S: usize = 20; // Nb of secconds we let the ai play the game before registering their scrore
pub const GAME_DT: f64 = 0.05; // 0.0166
pub const NB_GENERATIONS: usize = 100;
pub const NB_GENOME_PER_GEN: usize = 2000;

neat::NeuralNetworkTopology::new(0.2, 3, rng)

Deadlocks in 15 generations

sim15.backup.txt

[HyperCodec]

While testing performances & learning curves, i found out that high mutation rate (=>0.1) deadlocks in less than 50 gens 100% of the time, and now that i think of it, it might be the difference between me saying that it looks good and me saying that it doesn't work again

Example:

pub const NB_GAMES: usize = 3;

pub const GAME_TIME_S: usize = 20; // Nb of secconds we let the ai play the game before registering their scrore

pub const GAME_DT: f64 = 0.05; // 0.0166

pub const NB_GENERATIONS: usize = 100;

pub const NB_GENOME_PER_GEN: usize = 2000;

neat::NeuralNetworkTopology::new(0.2, 3, rng)

Deadlocks in 15 generations

sim15.backup.txt

So yeah the deadlock issue is probably one of the mutations.

[HyperCodec]

I wonder if the deadlock might be happening during the mutation phase, leading to something that can't be accurately debugged as it hasn't finished mutating the neural network before it deadlocks.

[HyperCodec]

Might not necessarily mean anything, but just ran some stress tests and such on windows in dev branch (rayon and crossover) and it didn't deadlock once.

Either I'm just really lucky or this has something to do with platform-specific things.

[Bowarc]

Have you tried high mutation rate ?

[HyperCodec]

Yeah I just got lucky, it happens on any platform.

I did more testing and found that the deadlock is during the running phase, meaning that it's still probably some type of recursive RwLock.

[HyperCodec]

Still can't find this deadlock even after weeks, it's being really evasive.

It's almost certainly a recursive RwLockor duped input, but I have code to prevent both of those from happening.

I thought it might be something like those while loops that reroll until a valid state is reached infinitely looping because there is no valid state, but the deadlock doesn't happen during mutation so it can't be that (although probably do want to patch that, it's extremely rare and unlikely to ever happen but is still a possibility).

I'm really just out of ideas for what could possibly cause this issue.

[HyperCodec]

While I think this is definitely a high-priority issue that urgently needs to be fixed, I'll take a break from it so it doesn't keep taking time away from new features and such.