How to load a NN from an SD? [Response to an email]

[GiorgosXou]

FOR THE CORRECT-VALID ANSWER CLICK RIGHT HERE FOR THE CORRECT-VALID ANSWER CLICK RIGHT HERE FOR THE CORRECT-VALID ANSWER CLICK RIGHT HERE FOR THE CORRECT-VALID ANSWER CLICK RIGHT HERE FOR THE CORRECT-VALID ANSWER CLICK RIGHT HERE FOR THE CORRECT-VALID ANSWER CLICK RIGHT HERE

Answer

Regarding a question I was asked in an email:

📜 Click to expand and see the code

```c++ #define NumberOf(arg) ((unsigned int) (sizeof (arg) / sizeof (arg [0]))) // calculates the number of layers (in this case 3) #define FILENAME "WEIGHTSA.txt" // make sure the name is simple because it seems like somethings wrong if not #define SELU #include // https://www.arduino.cc/reference/en/libraries/sd/ #include #include NeuralNetwork *NN; const unsigned int layers[] = {2,4,1}; //3 layers (1st)layer with 2 input neurons (2nd)layer with 4 hidden neurons and (3rd)layer with 1 output neuron float *output; // 3rd layer's output(s) File myFile; // The file from which weights will be loaded the next time //Default Inputs [for Training only] const float inputs[4][2] = { {0, 0}, // = 0 {0, 1}, // = 1 {1, 0}, // = 1 {1, 1} // = 0 }; const float expectedOutput[4][1] = {{0},{1},{1},{0}}; // Values that we are expecting to get from the 3rd/(output)layer of Neural-network, in other words something like a feedback to the Neural-network. void initialize() { Serial.begin(9600); Serial.print("Initializing SD"); if (!SD.begin()) { Serial.println(", failed!"); exit(0); } Serial.println(", done."); randomSeed(millis()); // Just for testing the NN later at loop() } void load_weights() { myFile = SD.open(FILENAME); if (myFile) { float weights[12]; // (2*4)+(4*1) float biases[2]; // input_to_hiden and hiden_to_output Serial.print("Loading weigths from '" + String(FILENAME) + "'"); int numberOflayers = myFile.readStringUntil('\n').toInt(); int i_j = 0; int inOuts; for(int n=0; nnumberOflayers+1); for(int n=0; nnumberOflayers; n++){ myFile.println(*NN->layers[n].bias, 7); myFile.println(NN->layers[n]._numberOfOutputs * NN->layers[n]._numberOfInputs); for(int i=0; ilayers[n]._numberOfOutputs; i++){ for(int j=0; jlayers[n]._numberOfInputs; j++){ myFile.println(NN->layers[n].weights[i][j], 7); // or ...weights[i_j] if defined B00010000 , where i_j is the sum of i + j } } } Serial.println(", done."); myFile.close(); } else {// if the file didn't open, print an error: Serial.println("error opening '" + String(FILENAME) + "' for writing."); exit(0); } } void train_weights() { NN = new NeuralNetwork(layers,NumberOf(layers)); //Initialization of NeuralNetwork object Serial.println("Training the NN"); do{ for (int j = 0; j < NumberOf(inputs); j++) // Epoch { NN->FeedForward(inputs[j]); // FeedForwards the input arrays through the NN | stores the output array internally NN->BackProp(expectedOutput[j]); // "Tells" to the NN if the output was the-expected-correct one | then, "teaches" it } // Prints the MSError. Serial.print("MSE: "); Serial.println(NN->MeanSqrdError,6); // Loops through each epoch Until MSE goes < 0.003 }while(NN->getMeanSqrdError(NumberOf(inputs)) > 0.003); Serial.println("Done"); save_weights(); } void setup() { initialize(); if (SD.exists(FILENAME)) load_weights(); else train_weights(); NN->print(); } float input[2]; // Input Array void loop() // testing a hypotherical scenarion { //For example: here you could have a live input from two buttons/switches (or a feed from a sensor if it was a different NN) input[0] = random(2); // ... lets say input from a button/Switch | random(2) = 0 or 1 input[1] = random(2); // ... lets say input from another button/Switch | random(2) = 0 or 1 output = NN->FeedForward(input); // FeedForwards the input-array through the NN | returns the predicted output(s) Serial.println((String)"Inputs: " + round( input[0]) + "," + round(input[1])); // Although you can kind of use casting like: Serial.println((String)"Output: " + round(output[0])); // "(int)output[0]" instead of round to reduces ROM usage, be careful delay(1500); } ```

Important note

Althrought it works (on Arduino UNO) there's an important issue to note! .toFloat() (which is atof() in it's core) it is not 100% accurate, because of the ULP error\issue (as from what I understand), eg.

Training phase weigths:

Saving weigths into 'WEIGHTS7.txt', done.

----------------------
2 4| bias:0.96
1  W:-0.9386852  W:-0.9057563 
2  W:-0.9714680  W:-0.9337891 
3  W:-0.9013161  W:-0.9461459 
4  W:-2.0541970  W:-2.0452339 
----------------------
4 1| bias:-0.37
1  W: 0.5785636  W: 0.4733289  W: 0.5075090  W:-1.2084372 
----------------------

Loading from SD weigths:

Loading weigths from 'WEIGHTS7.txt', done.

----------------------
2 4| bias:0.96
1  W:-0.9386852  W:-0.9057563 
2  W:-0.9714680  W:-0.9337891 
3  W:-0.9013161  W:-0.9461459 
4  W:-2.0541970  W:-2.0452339 
----------------------
4 1| bias:-0.37
1  W: 0.5785636  W: 0.4733289  W: 0.5075089  W:-1.2084370 
----------------------

Results to W: 0.5075090 and W:-1.2084372 becoming W: 0.5075089 and W:-1.2084370 something that could result into issues in larger NNs (I guess)

Solutions

• Consider edditing some of the weights from SD afterwords manually by adding an extra floating point might help to correct potential rounding issues related to ULP errors even thought the limit for arduino uno is 7 floating points
• Create some kind of sophisticated saving solution based on ^^^ ?
• Save the weights to the internal or external EEPROM or F-RAM instead

Thoughts

For simple NNs I don't think this is a huge issue, but for more complex ones it might be. So, it might be a good idea to consider EEPROM or F-RAM instead.

References

• SD:
• • Using SD Card Module with Arduino | Read/Write/Data Logger
• • Reading text line by line from SD
• Precision:
• • .toFloat() (which is atof() in it's core)
• • .toFloat() giving incorrect anwsers
• • atof like implementation
• • String to float on Arduino
• • Unit in the last place

[GiorgosXou]

Ok, I might be an Idiot, you can save them as long and do something like this, example:

void setup() {
  Serial.begin(9600);
  uint32_t myInt = 1078530000; // Replace with your 4-byte integer

  float myFloat = *((float*)(&myInt));

  Serial.println(myFloat, 7);
}

void setup() {
  Serial.begin(9600);
  float myFloat = 3.14159; // Replace with your float

  uint32_t myInt = *((uint32_t*)(&myFloat));

  Serial.println(myInt);
}

[GiorgosXou]

So this, should work, but for some reason, it is not accurate either (but in another way... wierd....)

📜 Click to expand and see the code

```c++ #define NumberOf(arg) ((unsigned int) (sizeof (arg) / sizeof (arg [0]))) // calculates the number of layers (in this case 3) #define FILENAME "WEIGHTS2.txt" // make sure the name is simple because it seems like somethings wrong if not #define SELU #include // https://www.arduino.cc/reference/en/libraries/sd/ #include #include NeuralNetwork *NN; const unsigned int layers[] = {2,4,1}; //3 layers (1st)layer with 2 input neurons (2nd)layer with 4 hidden neurons and (3rd)layer with 1 output neuron float *output; // 3rd layer's output(s) File myFile; // The file from which weights will be loaded the next time //Default Inputs [for Training only] const float inputs[4][2] = { {0, 0}, // = 0 {0, 1}, // = 1 {1, 0}, // = 1 {1, 1} // = 0 }; const float expectedOutput[4][1] = {{0},{1},{1},{0}}; // Values that we are expecting to get from the 3rd/(output)layer of Neural-network, in other words something like a feedback to the Neural-network. void initialize() { Serial.begin(9600); Serial.print("Initializing SD"); if (!SD.begin()) { Serial.println(", failed!"); exit(0); } Serial.println(", done."); randomSeed(millis()); // Just for testing the NN later at loop() } void load_weights() { myFile = SD.open(FILENAME); if (myFile) { float weights[12]; // (2*4)+(4*1) float biases[2]; // input_to_hiden and hiden_to_output Serial.print("Loading weigths from '" + String(FILENAME) + "'"); int numberOflayers = myFile.readStringUntil('\n').toInt(); int i_j = 0; int inOuts; long t; for(int n=0; nnumberOflayers+1); for(int n=0; nnumberOflayers; n++){ myFile.println(*((uint32_t*)(&*NN->layers[n].bias))); myFile.println(NN->layers[n]._numberOfOutputs * NN->layers[n]._numberOfInputs); for(int i=0; ilayers[n]._numberOfOutputs; i++){ for(int j=0; jlayers[n]._numberOfInputs; j++){ myFile.println(*((uint32_t*)(&NN->layers[n].weights[i][j]))); // or ...weights[i_j] if defined B00010000 , where i_j is the sum of i + j } } } Serial.println(", done."); myFile.close(); } else {// if the file didn't open, print an error: Serial.println("error opening '" + String(FILENAME) + "' for writing."); exit(0); } } void train_weights() { NN = new NeuralNetwork(layers,NumberOf(layers)); //Initialization of NeuralNetwork object Serial.println("Training the NN"); do{ for (int j = 0; j < NumberOf(inputs); j++) // Epoch { NN->FeedForward(inputs[j]); // FeedForwards the input arrays through the NN | stores the output array internally NN->BackProp(expectedOutput[j]); // "Tells" to the NN if the output was the-expected-correct one | then, "teaches" it } // Prints the MSError. Serial.print("MSE: "); Serial.println(NN->MeanSqrdError,6); // Loops through each epoch Until MSE goes < 0.003 }while(NN->getMeanSqrdError(NumberOf(inputs)) > 0.003); Serial.println("Done"); save_weights(); } void setup() { initialize(); if (SD.exists(FILENAME)) load_weights(); else train_weights(); NN->print(); } float input[2]; // Input Array void loop() // testing a hypotherical scenarion { //For example: here you could have a live input from two buttons/switches (or a feed from a sensor if it was a different NN) input[0] = random(2); // ... lets say input from a button/Switch | random(2) = 0 or 1 input[1] = random(2); // ... lets say input from another button/Switch | random(2) = 0 or 1 output = NN->FeedForward(input); // FeedForwards the input-array through the NN | returns the predicted output(s) Serial.println((String)"Inputs: " + round( input[0]) + "," + round(input[1])); // Although you can kind of use casting like: Serial.println((String)"Output: " + round(output[0])); // "(int)output[0]" instead of round to reduces ROM usage, be careful delay(1500); } ```

Based on the NN.print() weigths are the same, but somethign else is wrong when loading...

[GiorgosXou]

FIXED ANSWER

Now everything works fine! (Just make sure that your MCU\board's float and long are of the same length, eg. "on Arduino uno they are both 4bytes in length" else you should consider changing atol() and the casting to somethign equivalent to the float size\length )

#define NumberOf(arg) ((unsigned int) (sizeof (arg) / sizeof (arg [0]))) // calculates the number of layers (in this case 3)
#define FILENAME "/WEIGHTS2.txt" // make sure the name is simple because it seems like somethings wrong if not
#define SELU 
#include <SD.h> // https://www.arduino.cc/reference/en/libraries/sd/
#include <SPI.h>
#include <NeuralNetwork.h>
          NeuralNetwork *NN;

const unsigned int layers[] = {2,4,1}; //3 layers (1st)layer with 2 input neurons (2nd)layer with 4 hidden neurons and (3rd)layer with 1 output neuron
float *output; // 3rd layer's output(s)
File myFile;   // The file from which weights will be loaded the next time

//Default Inputs [for Training only]
const float inputs[4][2] = {
  {0, 0}, // = 0
  {0, 1}, // = 1
  {1, 0}, // = 1
  {1, 1}  // = 0
};
const float expectedOutput[4][1] = {{0},{1},{1},{0}}; // Values that we are expecting to get from the 3rd/(output)layer of Neural-network, in other words something like a feedback to the Neural-network.

void initialize()
{
  Serial.begin(9600);
  Serial.print("Initializing SD");
  if (!SD.begin()) { 
    Serial.println(", failed!"); 
    exit(0);
  }
  Serial.println(", done.");  
  randomSeed(millis()); // Just for testing the NN later at loop()
}

float weights[12]; // (2*4)+(4*1) 
float biases[2];   // input_to_hiden and hiden_to_output
void load_weights()
{
  myFile = SD.open(FILENAME);
  if (myFile) {
    Serial.print("Loading weigths from '" + String(FILENAME) + "'");  
    int numberOflayers = myFile.readStringUntil('\n').toInt();
    int i_j = 0;
    int inOuts;
    long tmp;
    for(int n=0; n<numberOflayers; n++){
      tmp = atol((char*)myFile.readStringUntil('\n').c_str());
      biases[n] = *((float*)(&tmp)); // toFloat() which is atof() is not accurate (at least on Arduino UNO)
      inOuts = myFile.readStringUntil('\n').toInt();
      for(int i=0; i<inOuts; i++){
          tmp = atol((char*)myFile.readStringUntil('\n').c_str());
          weights[i_j] = *((float*)(&tmp)); // toFloat() which is atof() is not accurate (at least on Arduino UNO)
          i_j++;
      }
    }
    Serial.println(", done.");  
    myFile.close();
    NN = new NeuralNetwork(layers, weights, biases, numberOflayers);//Initialization of NeuralNetwork object
  } else {// if the file didn't open, print an error:
    Serial.println("Error opening '" + String(FILENAME) + "' for reading.");      
    exit(0);
  }
}

void save_weights()
{
  myFile = SD.open(FILENAME, FILE_WRITE);
  if (myFile){
    Serial.print("Saving weigths into '" + String(FILENAME) + "'");  
    myFile.println(NN->numberOflayers+1); 
    for(int n=0; n<NN->numberOflayers; n++){
      myFile.println(*((int32_t*)(&*NN->layers[n].bias))); 
      myFile.println(NN->layers[n]._numberOfOutputs * NN->layers[n]._numberOfInputs); 
      for(int i=0; i<NN->layers[n]._numberOfOutputs; i++){
        for(int j=0; j<NN->layers[n]._numberOfInputs; j++){
          myFile.println(*((int32_t*)(&NN->layers[n].weights[i][j]))); // or ...weights[i_j] if defined B00010000 , where i_j is the sum of i + j
        }
      }
    }
    Serial.println(", done.");  
    myFile.close();
  } else {// if the file didn't open, print an error:
    Serial.println("error opening '" + String(FILENAME) + "' for writing.");      
    exit(0);
  }
}

void train_weights()
{
  NN = new NeuralNetwork(layers,NumberOf(layers)); //Initialization of NeuralNetwork object
  Serial.println("Training the NN");
  do{
    for (int j = 0; j < NumberOf(inputs); j++) // Epoch
    {
      NN->FeedForward(inputs[j]);      // FeedForwards the input arrays through the NN | stores the output array internally
      NN->BackProp(expectedOutput[j]); // "Tells" to the NN if the output was the-expected-correct one | then, "teaches" it
    }

    // Prints the MSError.
    Serial.print("MSE: "); 
    Serial.println(NN->MeanSqrdError,6);

    // Loops through each epoch Until MSE goes  < 0.003
  }while(NN->getMeanSqrdError(NumberOf(inputs)) > 0.003);
  Serial.println("Done");
  save_weights();
}

void setup()
{
  initialize();
  if (SD.exists(FILENAME))
    load_weights();
  else
    train_weights();

  NN->print();
}

float input[2]; // Input Array
void loop()  // testing a hypotherical scenarion
{
  //For example: here you could have a live input from two buttons/switches (or a feed from a sensor if it was a different NN)
  input[0] = random(2); // ... lets say input from a       button/Switch | random(2) = 0 or 1
  input[1] = random(2); // ... lets say input from another button/Switch | random(2) = 0 or 1

  output = NN->FeedForward(input); // FeedForwards the input-array through the NN | returns the predicted output(s)

  Serial.println((String)"Inputs: " + round( input[0]) + "," + round(input[1])); // Although you can kind of use casting like:
  Serial.println((String)"Output: " + round(output[0])); // "(int)output[0]" instead of round to reduces ROM usage, be careful

  delay(1500);
}

[GiorgosXou]

Ok just fixed 2 more things for ESP32 uint32_t to int32_t and / to #define FILENAME "/WEIGHTS2.txt"

[GiorgosXou]

Also many issues with esp32 related to -fpermissive and undifined actions taken in destractor that i'm fixing right now in a week or so i will have a new release

[GiorgosXou]

Oh... god ...I might have done something wrong, i'm not sure yet

[GiorgosXou]

whatever i did here doesn't matter, in a week I'll have a new version that supports SD.

[GiorgosXou]

https://github.com/GiorgosXou/NeuralNetworks/blob/master/examples/Media/Save_load_NN_from_SD/Save_load_NN_from_SD.ino