No error tracking

[lewisrichardson2103]

Hi, This is a nice simple library so thank you for developing it.

One thing that would be very useful is a means to extract the error in some manner after each learning epoch. Perhaps mean error squared? this would allow us to track the progress and stop teaching the network when we are happy with the error is small enough. On top of this it will help us understand if we are improving the networks learning rate or making it worse

Many thanks

Lewis

[GiorgosXou]

_{i am just commenting to let you know that i am aware of the question, i just have to go for sleep as it is too late here, i am going to answer tomorow maybe after work [...] 😴 GN}

[GiorgosXou]

Hi

Hello, @lewisrichardson2103 (:

This is a nice simple library so thank you for developing it.

Thanks you so much for your your kind words!

One thing that would be very useful is a means to extract the error in some manner after each learning epoch. Perhaps mean error squared?

True [...] i am going to backpropagate your feedback through my biological neural-network and then start working again on the library 😂 but for now until the next update you will need to use something like this:

#define NumberOf(arg) ((unsigned int) (sizeof (arg) / sizeof (arg [0]))) //calculates the amount of layers (in this case 3)

#include <NeuralNetwork.h>
#include <Math.h>
const unsigned int layers[] = {2,4,1}; //3 layers (1st)layer with 2 input neurons (2nd)layer with 4 hidden neurons and (3rd)layer with 1 output neuron
float *outputs; // 3rd layer's outputs (in this case output)

//Default Inputs
const float inputs[4][2] = {
  {0, 0}, //0
  {0, 1}, //1
  {1, 0}, //1
  {1, 1}  //0
};

const float expectedOutput[4][1] = {{0},{1},{1},{0}}; // values that we were expecting to get from the 3rd/(output)layer of Neural-network, in other words something like a feedback to the Neural-network.

void setup()
{

  Serial.begin(9600);

  NeuralNetwork NN(layers,NumberOf(layers)); // Creating a NeuralNetwork with default learning-rates

  do{
    //Trains the NeuralNetwork for 1000 epochs = (Training loops), every time Until..
    for(int i=0; i < 1000; i++) // epochs = Training loops
    {
      for (int j = 0; j < NumberOf(inputs); j++)
      {
        outputs = NN.FeedForward(inputs[j]); // Feeds-Forward the inputs to the first layer of the NN and Gets the output.
        NN.BackProp(expectedOutput[j]); // Tells to the NN if the output was right/the-expectedOutput and then, teaches it.
      }
    }
  }while(compute_SquaredError() > 0.01); 
  //..Until compute_SquaredError() > 0.01
  // Trust me, don't simplify the loops 
  // (It consumes CPU power)

  //Goes through all inputs
  for (int i = 0; i < NumberOf(inputs); i++)
  {
    outputs = NN.FeedForward(inputs[i]); // Feeds-Forward the inputs[i] to the first layer of the NN and Gets the output 
    Serial.println(outputs[0], 7); // prints the first 7 digits after the comma.
  }

   NN.print(); // prints the weights and biases of each layer
   Serial.print("Mean-Squared-Error: ");
   Serial.println(compute_SquaredError(), 7);
}

int N = layers[NumberOf(layers)-1];
float compute_SquaredError() {

  float SquaredError = 0;
  for (int i=0; i< N; i++){
    SquaredError += sq(outputs[i] - expectedOutput[i][i]);
  }
  return (1/N)* SquaredError;
}

void loop() {}

Notice:

int N = layers[NumberOf(layers)-1];
float compute_SquaredError() {

  float SquaredError = 0;
  for (int i=0; i< N; i++){
    SquaredError += sq(outputs[i] - expectedOutput[i][i]);
  }
  return (1/N)* SquaredError;
}

_{⚠️ Please be aware that i lack of math knowledge so don't trust 100% of what i am claiming generally speaking, lol}

[lewisrichardson2103]

😂 I like it! I have implemented something similar to what you have suggested! Hopefully your suggestion is useful for others looking at this. Potentially for the update you can sum all the squared errors in the layer.cpp backprop function. And then a separate function can be called to retrieve the value as a mean... float GetMeanSqrdError(numberInputsPerEpoch) { return sumSqrdErr / numberInputsPerEpoch; }

Doing this should mean it doesn’t impact the way it currently works so no one will have a broken program !?!

[GiorgosXou]

To be honest.. i've thought about it too, but i wasn't sure about it... Because having to deal with the CPU power of most known to me microcontrollers, personally i see it as alot sensitive subject [...] _{(and especially now that i am more deticated and focused to the algorithmic part rather than to the mathematical part of the library)}

Soo.. some time ago when i had got back to the code again, i started to conflict alot about those situation., like: "should i use more CPU or more RAM?!", "should i use even more CPU! or split the algorithm in more parts..?" and the story goes on...

(After a conversation with members of the Arduino Forum) In conlusion what i decided to do, was to use preprocessor-directives^{_^1} such as things like CPU or RAM usage would be optimised based on user's preference... but because this is based on a slightly abstract model in my head that i haven't implimented it to the ideal state yet, i am also thoughtful-skeptical about it in some ways 😂 _{^{1^}(although there are some drawbacks)}

Good news is that it is as simple to its use as just defining a word like #Define CPU_LVL 1 and then something like calculating the MeanSqrdError is computed to the layer.cpp backprop function and when #Define CPU_LVL 2 then computed into the function.. _{^(this is the ideal 😂)}

[lewisrichardson2103]

Makes sense, no stress there is time. I am running it on an Adafruit nrf52480 so no memory issues there. I have implemented some changes on a local version of your library just so it suits my needs but thought it was worth mentioning in case others found it useful. I like to hear you are forward thinking and not rushing in to implementation!

[GiorgosXou]

UPDATE: hahahaha, I just realized how impractical my solution was 🤦‍♂️ (for anyone viewing this thread/issue pls forgive me for that, i was in a rush + i am mainly self taught with NNs and still making mistakes [good news is that i got motivated and i have some days now that i am working on the library])