Best way to merge two networks

[Leekao]

This is a question and not an issue, I couldn't find anywhere else to ask this, sorry if this is the wrong place. I'm trying to write a function that will take two networks as input and returns a third network that is 1/2 the first network and 1/2 the second network and I wondered what would be the best way to do this? I tried converting the networks to json and messing with it but it didn't really work. any advice or help will be welcomed.

[wagenaartje]

But how do you want to 'combine' them? Do you want to feed the output of one network in the input of the other? Or do you really want to 'merge' them averaging weight's and biases for example? And why do you want to combine them?

If the first in the case, you can use the myNetwork.project() function.

[Leekao]

I think the second option will be better, but suppose I want to try the first option, how would the code look like? I'll iterate over neurons, delete the connections object and make new projections? the reason I'm messing with this is that I've been experimenting with genetic algorithms and wanted to try and implement the same concepts using neural networks.

[wagenaartje]

Well i'm doing the exact same thing at the moment with genetic algorithms, so we are talking about performing the crossover operation on two networks?

It's easier to merge two networks that are the same size. This will allow you to iterate over both networks at the same time.

const INPUT = 1;
const HIDDEN = 4;
const OUTPUT = 1;

// Two networks of the same size
var network1 = new synaptic.Architect.Perceptron(INPUT, HIDDEN, OUTPUT);
var network2 = new synaptic.Architect.Perceptron(INPUT, HIDDEN, OUTPUT);

crossOver(network1, network2);

function crossOver(network1, network2){
  // Converting the networks to JSON makes it all a whole lot easier
  network1 = network1.toJSON();
  network2 = network2.toJSON();

  // Create a network, this will be the result of the two networks
  var offspring = new synaptic.Architect.Perceptron(INPUT, HIDDEN, OUTPUT);
  offspring = offspring.toJSON();

  var neurons = HIDDEN + OUTPUT; // you only need to crossover the bias of the hidden and output neurson
  var connections = INPUT * HIDDEN + HIDDEN * OUTPUT; // amount of connections of which you can modify the weight

  // Let's combine the neuron biases for the offspring
  for(var i = 0; i < neurons; i++){
    var bias1 = network1.neurons[INPUT+i].bias; // get's the bias of neuron i of network1
    var bias2 = network2.neurons[INPUT+i].bias; // get's the bias of neuron i of network2

    var new_bias = (bias1 + bias2) / 2; // this is the function that calculates the new bias, do whatever you want here

    offspring.neurons[INPUT+i].bias = new_bias; 
  }

  // Let's combine the neuron conection weights for the offspring
  for(var i = 0; i < connections; i++){
    var weight1 = network1.connections[i].weight; // get's the weight of connection i of network1
    var weight2 = network2.connections[i].weight; // get's the weight of connection i of network2

    var new_weight = (weight1 + weight2) / 2; // this is the function that calculates the new bias, do whatever you want here

    offspring.connections[i].weight = new_weight; 
  }

  // Now convert the offspring JSON back to a network and return it
  return synaptic.Network.fromJSON(offspring);
}

The example I wrote above looks at every neuron bias and connection weight and takes the average of the two networks and returns one 'average' network. There are way more crossover techniques out there.

For example, you could also randomly select neuron biases and connection weights of each parent to the offspring.

[Leekao]

@wagenaartje Wow, that look great, I'll try it out, thank you!

[wagenaartje]

I am creating an extension of synaptic having built in functions like this. Take a look here

[rokyed]

@wagenaartje thank you!! and also I propose this (nodejs):

const {Architect, Network, Layer, Trainer} = require('synaptic')
class Fusion {
    constructor () {

    }
    /* expecting network.toJSON as entries (Object, Object, Float(0 to 1)) */
    crossOver(network1, network2, ratio) {
        if (!ratio) ratio = 0.5
        let ratioPartA = ratio
        let ratioPartB = 1 - ratio

        let ntk1 = Network.fromJSON(network1)
        // Create a network, this will be the result of the two networks
        let offspring = ntk1.clone()
        let inputCount = offspring.inputs()

        offspring.clear()
        offspring = offspring.toJSON()

        // Let's combine the neuron biases for the offspring
        for (let i = inputCount; i < offspring.neurons.length; i++) {
            let bias1 = network1.neurons[i].bias * ratioPartA // get's the bias of neuron i of network1
            let bias2 = network2.neurons[i].bias * ratioPartB // get's the bias of neuron i of network2

            let new_bias = (bias1 + bias2) // this is the function that calculates the new bias, do whatever you want here

            offspring.neurons[i].bias = new_bias
        }

        // Let's combine the neuron conection weights for the offspring
        for (let i = 0; i < offspring.connections.length; i++) {
            let weight1 = network1.connections[i].weight * ratioPartA // get's the weight of connection i of network1
            let weight2 = network2.connections[i].weight * ratioPartB // get's the weight of connection i of network2

            let new_weight = (weight1 + weight2) // this is the function that calculates the new bias, do whatever you want here

            offspring.connections[i].weight = new_weight
        }

        // Now convert the offspring JSON back to a network and return it
        return offspring
    }
}

module.exports = Fusion

it's a spinoff of yours, it creates the offspring from the network1, i will use it to use browser based workers to train and merge my networks ( they will all start with the same network ), thank you again @cazala thank you too for creating such a beautiful library

(edit: for the n(th) time I fixed the typos and bugs from it :smile: ) last edit, i added a ratio because i use it with multiple computers and i am merging like 12 instances into one, this helps to not let the main base influenced that easly. it helps with keeping the neural net on track