Support of MLP Network with 2 or >2 hidden layers

[Elliot1299]

Good afternoon, dear respective jorgenkg!

If is it possible may I ask following, would it be possible to set MLP with more than 2 or >2 hidden layers in your code. For example if I need to create MLP with 4 layers (1 input layer, 2 hidden layers, 1 output layer), each layer have 3 neurons, 3 inputs at input layer, 3 outputs on the output layer. In main.py I should set following: "n_inputs" : 3, # Number of network input signals "layers" : [ (3, tanh_function), (3, tanh_function),(1, sigmoid_function) ], ?

Could you please say would it be right?

Could you please say in your question at http://stackoverflow.com/questions/17049321/mlp-neural-network-calculating-the-gradient-matrices you have enclosed Code which computes Error Gradient for MLP's with n hidden layers with n neurons, may I ask you how is it possible to launch it? I have seen there 3 tabs "My Python code (derived from Chuck Anderson's snippet):", "The NPNeuralNet class", "The final implementation", what should I do in order to launch it, I should perceive "My Python code (derived from Chuck Anderson's snippet):" as main? But in "The final implementation" there is no Class declaration, only Functions definitions (def forward(), def backprop(), def errorFunction(), def errorGradient() ), maybe I should transfer them in "The NPNeuralNet class", but then I do not see calling of all this functions in "My Python code (derived from Chuck Anderson's snippet):". Could you please say how can I launch program described in your question? Currently I am studying calculations of Error gradient with appliance of Partial derivatives I have done couple of calculations manually and wanted to check it with some program but on the internet there was no any frameworks for MLP's with support of n hidden layers, except your Code which I have found on stackoverflow, as in "The NPNeuralNet class" I have seen that it is possible to set n numbers of hidden layers.

self.n_inputs = n_inputs self.n_hiddens = n_hiddens self.n_outputs = n_outputs self.n_hidden_layers = n_hidden_layers

Thank you in advance for your reply!

Sincerely

[jorgenkg]

Hi @Elliot1299, You should really work with the code on this repo, rather than the old code available on StackOverflow.

Topology

The network can be instanced to a topology of your own choosing. If you construct the network using:

settings = {
    "n_inputs" : 3,
    "layers" : [ (3, tanh_function), (3, tanh_function),(1, sigmoid_function) ],
}

# initialize the neural network
network = NeuralNet( settings )

the network will be a 2-hidden layer neural network with a single output. In literature, this is either called a 2-hidden layer network or simply a 3-layered network. This seems to be what you are trying to achieve.

Gradients

If you are interested in computing the gradients, you can call the gradient calculation routine directly:

settings = {
    # Required settings
    "n_inputs"              : 2,       # Number of network input signals
    "layers"                : [ (3, tanh_function), (3, tanh_function), (1, sigmoid_function) ],
}

# initialize the neural network
network = NeuralNet( settings )

# calculate the gradient for the whole network
# @param training_data    This is a NumPy matrix with shape (n_samples, n_input_signals)
# @param training_targets This is a NumPy matrix with shape (n_samples, n_output_signals)
gradients = network.gradient( network.get_weights(), training_data, training_targets )

Important: Note that this routine returns all the gradiens in a single list! If you would like to have them structured by layer, swap out the following lines of code in neuralnet.py:

for i in layer_indexes:
            # Loop over the weight layers in reversed order to calculate the deltas

            # calculate the weight change
            dW = layers.append(np.dot( delta, add_bias(input_signals[i]) ).T.flat)

            if i!= 0:
                """Do not calculate the delta unnecessarily."""
                # Skip the bias weight
                weight_delta = np.dot( self.weights[ i ][1:,:], delta )

                # Calculate the delta for the subsequent layer
                delta = weight_delta * derivatives[i-1]
        #end weight adjustment loop

        return np.hstack( reversed(layers) )

with:

for i in layer_indexes:
            # Loop over the weight layers in reversed order to calculate the deltas

            # calculate the weight change
            layers.append(np.dot( delta, add_bias(input_signals[i]) ).T)

            if i!= 0:
                """Do not calculate the delta unnecessarily."""
                # Skip the bias weight
                weight_delta = np.dot( self.weights[ i ][1:,:], delta )

                # Calculate the delta for the subsequent layer
                delta = weight_delta * derivatives[i-1]
        #end weight adjustment loop

        return reversed(layers)

A bit more on training:

If you're only interested in training a multilayered network, you should emperiment with the scg() routine as long as you are not using too many nodes and layers,

[Elliot1299]

Good morning, dear respective jorgenkg!

Thank you very much for explaining creating network structure and Gradient Calculation in detail!