ElmanPattern null as activation function in output layer

[Dennis1111]

Hi

Once made a post about the ElmanPattern in the forums since I thaught it was strange that in the output layer null is used as the argument for the activation function. I see it as a bug but apologize in advance if I'm wrong about moving the issue to here. When I train the Elman network it produces extreme figures (I guess it depends on the learning task also). I have added generate2() to the Elman Network which instead uses network.addLayer(new BasicLayer(this.activation, false, this.outputNeurons)); and also made a small test program to show the differences.

Training Elman network without Sigmoid Epoch 0 Error = 3.340547587759162 Epoch 1 Error = 12.093709207462359 Epoch 2 Error = 20.10574781179527 Epoch 3 Error = 192.05958976530124 Epoch 4 Error = 584.3769644076101 Epoch 5 Error = 7792.268927035693 Epoch 6 Error = 73795.51179330552 Epoch 7 Error = 686300.045126438 Epoch 8 Error = 3871554.818886862 Epoch 9 Error = 2.3696703827344127E7 Training Elman network with Sigmoid Epoch 0 Error = 0.24208329326323957 Epoch 1 Error = 0.2252998899657926 Epoch 2 Error = 0.20599926542888564 Epoch 3 Error = 0.1845726317633126 Epoch 4 Error = 0.16181453130188853 Epoch 5 Error = 0.13884928778828928 Epoch 6 Error = 0.11686393694107555 Epoch 7 Error = 0.09679717323086938 Epoch 8 Error = 0.0791784626642417 Epoch 9 Error = 0.06417317172944495

The test code

package org.encog.neural.pattern;

import java.util.Random;

import org.encog.ml.data.basic.*; import org.encog.neural.pattern.ElmanPattern; import org.encog.neural.networks.BasicNetwork; import org.encog.neural.networks.training.propagation.back.Backpropagation; import org.encog.engine.network.activation.ActivationSigmoid; import org.encog.util.arrayutil.NormalizeArray;

public class ElmanPatternTest {

private BasicMLDataSet dataSet; private BasicNetwork elmanNetwork1; private BasicNetwork elmanNetwork2;

public ElmanPatternTest() { double noiseFactor=100; int patterns=10000; this.dataSet = createTestDataSet(patterns,noiseFactor); int inputSize=1; int outputSize=1; int hiddens=10; createElmanNetworks(inputSize,outputSize,hiddens); }

public void train(int epochs) { System.out.println("Training Elman network without Sigmoid"); train(epochs,elmanNetwork1); System.out.println("Training Elman network with Sigmoid"); train(epochs,elmanNetwork2); }

private void train(int epochs,BasicNetwork network) {
double learnRate = 0.0001; double momentum= 0.001; Backpropagation train = new Backpropagation(network,this.dataSet,learnRate,momentum); for(int epoch=0;epoch<epochs;epoch++) { train.iteration(); System.out.println("Epoch "+epoch+" Error = "+train.getError()); } }

private void createElmanNetworks(int inputSize,int outputSize,int hiddenNeurons) { ElmanPattern pattern = new ElmanPattern(); pattern.setActivationFunction(new ActivationSigmoid()); pattern.setInputNeurons(inputSize); pattern.addHiddenLayer(hiddenNeurons); pattern.setOutputNeurons(outputSize); this.elmanNetwork1=(BasicNetwork)pattern.generate(); this.elmanNetwork2=(BasicNetwork)pattern.generate2();
}

private BasicMLDataSet createTestDataSet(int size,double noiseFactor) { Random rd = new Random(); BasicMLDataSet samples = new BasicMLDataSet();

double[] inputs = new double[size];
double[] ideals = new double[size];

for(int pattern=0;pattern<size;pattern++)
  {
double noise = noiseFactor*((rd.nextDouble()-0.5)/100);
double x = (2*((double)pattern)/size)-1;//-1..1
inputs[pattern] = x;
ideals[pattern] = Math.sin(x)+noise-(Math.cos(x)/3);    
  }
NormalizeArray norm = new NormalizeArray();

norm.setNormalizedLow(0.05);
norm.setNormalizedHigh(0.95);
ideals = norm.process(ideals); 

double[][] inputDataSet = new double[size][1];
double[][] idealDataSet = new double[size][1];
for(int pattern=0;pattern<size;pattern++)
  {
inputDataSet[pattern][0]=inputs[pattern];
idealDataSet[pattern][0]=ideals[pattern];
  }

BasicMLDataSet dataSet= new BasicMLDataSet(inputDataSet,idealDataSet);
return dataSet;

}

public static void main(String[] args) {
ElmanPatternTest elmanTest= new ElmanPatternTest();
elmanTest.train(10); } }

[jeffheaton]

I will extend the class to support this.