Simple MLP - NeuralNetwork Library For Microcontrollers

Nothing "Import ant", just a simple library for implementing Neural-Networks(NNs) easily and effectively on any Arduino board and other microcontrollers.

📚 Summary

Understanding the Basics of a Neural Network:
EXM 0 1 2 3 4 5 6 7 8 9 10 11

📦 Features

• Current

• • + Use of activation-functions per layer-to-layer.
• • + Optimizations based on user's preference.
• • + Support for custom activation functions.
• • + Basic ESP32-S3 SIMD acceleration.
• • + Both 16 and 8 bit, int quantization.
• • + MSE/BCE/CCE loss-functions.
• • + Support for double precision.
• • + Many activation-functions.
• • + Use of storage medias.
• • + Exceptional solutions.
• • + Simplicity!

• In the future, i want

• • - Your support.
• • - Better overall code.
• • - Other training methods.
• • - More Activation Functions.
• • - Training with int quantization.
• • - Support for convolutional layers.
• • - Support for fixed-point arithmetics.
• • - Different weight initialization methods.
• • - Support for external EEPROM and FRAM.
• • - Even more properties, for many different needs.

✏️ Examples

✨ (See also): training with Tensorflow section)

• 🔤 Basic:
• • Using a NN inside the loop
• • Pre-trained NN behaving as a 3-input-xor circuit
• • Training a NN to behave as a 3-input-xor circuit
• • Training a NN to behave as a single xor-gate
• • Using multiple Activ.-functions per layer-to-layer
• • Using multiple biases per layer-to-layer
• 💾 Media:
• • Save NN into SD and load it into RAM after restart
• • Saving a NN into the internal EEPROM for later use
• • Running NN mostly via internal EEPROM (not RAM)
• • Running NN mostly via programmable memmory
• 🎲 Other:
• • Pre-trained int8_t-quantized NN ✨
• • Using a custom function made by you
• • Support for 8Byte "double" instead of "float"
• • Recognizing handwritten digits (MNIST) ✨

⚠️ Important

• I am NOT a professional in any of those fields...
• In case of error with 'POINTER_REGS' click here
• bias means biases if MULTIPLE_BIASES_PER_LAYER is enabled
• If you don't want to USE_64_BIT_DOUBLE (which I also suggest you not to use), then make sure that you have used (4-byte)(32-bit)-precision variables when Training, Because Floats:"...are stored as 32 bits (4 bytes) of information...get more precision by using a double (e.g. up to 15 digits), on the Arduino, double is the same size as float."

🔬 Tested on

⚙️ Functions, Variables ...

Note that DFLOAT means float, unless you USE_64_BIT_DOUBLE, then it means double. IDFLOAT equals DFLOAT unless you USE_INT_QUANTIZATION, then it either means int16_t or int8_t. IS_CONST means nothing, unless you USE_PROGMEM, then it means const.

 IS_CONST IDFLOAT *default_Bias
 IS_CONST IDFLOAT *default_Weights
 byte *_ActFunctionPerLayer = NULL
 const unsigned int *layer_
 const unsigned int &NumberOflayers
 const DFLOAT &LRw
 const DFLOAT &LRb

Type Main Functions

DFLOAT Loss Functions

DFLOAT Loss Variables

DFLOAT Activation Functions

Due to (my uncertainty and) the strict RAM optimization that allows the library to use one array that stores only the values after the activation instead of two arrays storing values before and after the activation, the use of some derivative functions in backpropagation are not supported by this library at this moment, as also seen by the MACRO NO_BACKPROP below. This means that if you want to use and #define any function from 8-13 under the section "NO_BACKPROP support" , you won't be able to use backpropagation.

if you want to use other activation function from the default one, just define one other:

#define Sigmoid //[default] No need definition, for single activation across network
#define Tanh
#define ReLU
#define LeakyELU
#define ELU
#define SELU
...

Use any activation function per layer-to-layer, like :

#define ACTIVATION__PER_LAYER
#include <NeuralNetwork.h>

unsigned int layers[] = {3, 4, ..., 2, 1};
byte Actv_Functions[] = {   1, ..., 2, 0};

// Tanh > ... > ReLU > Sigmoid

If you want to drastically reduce ROM & slightly RAM size you can Define which Functions to use/compile, like:

#define ACTIVATION__PER_LAYER
        #define Sigmoid // 0
        //#define Tanh
        //#define ReLU
        //#define LeakyELU
        #define ELU     // 1
        #define SELU    // 2
        ...
#include <NeuralNetwork.h>

unsigned int layers[] = {3, 4, ..., 2, 1};
byte Actv_Functions[] = {   1, ..., 2, 0};

// ELU > ... > SELU > Sigmoid

^{⚠️ have in mind that because I used f(x) for derivatives there might be chances of mistakes (?)}

#define Custom Functions

(See also example) You can define up to 5. Every custom function, comes after every each non-custom one (numerically) eg:

#define ACTIVATION__PER_LAYER
        #define Sigmoid // 0
        //#define Tanh
        //#define ReLU
        //#define LeakyELU
        #define ELU  // 1
        #define SELU // 2
        #define CUSTOM_AF1 my_act_fun1 // 3
        #define CUSTOM_AF2 my_act_fun2 // 4
        ...

Define derivative-functions, by just definening ..._DFX:

        #define CUSTOM_AF1 my_act_fun1 
        #define CUSTOM_DF1 

And then use them in your sketch like:

// CUSTOM_DF1 is optional ...
#define ACTIVATION__PER_LAYER
        #define Tanh
        #define CUSTOM_AF1 my_sigmoid 
        #define CUSTOM_DF1

#include <NeuralNetwork.h>

// derivative function must end in "Der" | Limited to f(x), for optimization reasons
float NeuralNetwork::Layer::my_sigmoidDer(const float &fx){ return fx - fx * fx;      } 
float NeuralNetwork::Layer::my_sigmoid   (const float &x ){ return 1 / (1 + exp(-x)); }

unsigned int layers[] = {3, 4, ..., 2, 1};
byte Actv_Functions[] = {   0, ..., 0, 1};

// Tanh > ... > Tanh > my_sigmoid

IMPORTANT NOTE: Be careful commenting in front of #define, see issue #29

DFLOAT Variables Of Activation Functions

Type Other Variables

Note that except from _numberOfInputs and _numberOfOutputs everything else is not valid when you USE_INTERNAL_EEPROM

#define MACRO Properties

#define _1_OPTIMIZE 0B00000000

Don't use keywords to define optimizations, it won't work, use _X_OPTIMIZE

• ⚠️ = Backpropagation is not allowed
• 🟢 = Always enabled _{^{(not switchable yet.)}}
• ❌ = Not yet implimented
• 📌 = Recommended

👨‍💻 Training

To train a neural-network, you can use Tensorflow to do so. Here's a basic python example:

# pip install tensorflow
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.callbacks import LearningRateScheduler
import tensorflow as tf
import numpy as np

# Define if you want to use biases
IS_BIASED = True

# Enable 32-bit floating-point precision
tf.keras.backend.set_floatx('float32')

# Define the XOR gate inputs and outputs
inputs  = np.array([
    [ 0, 0, 0 ], 
    [ 0, 0, 1 ], 
    [ 0, 1, 0 ], 
    [ 0, 1, 1 ], 
    [ 1, 0, 0 ], 
    [ 1, 0, 1 ], 
    [ 1, 1, 0 ], 
    [ 1, 1, 1 ]
], dtype = np.float32)
outputs = np.array([[0], [1], [1], [0], [1], [0], [0], [1]], dtype = np.float32)
input_size = 3

# Create a simple convolutional neural network
model = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(input_size,)), # Input layer (no bias) 
    tf.keras.layers.Dense(3, activation='sigmoid', use_bias=IS_BIASED), # Dense  3 units 
    tf.keras.layers.Dense(1, activation='sigmoid', use_bias=IS_BIASED)  # Output 1 unit 
])

# Compile the model
optimizer = Adam(learning_rate=0.031)
model.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy'])

# Train the model
model.fit(inputs, outputs, epochs=900, verbose=0)

# Evaluate the model on the training data
loss, accuracy = model.evaluate(inputs, outputs)
print(f"Model accuracy: {accuracy * 100:.2f}%")

# Predict XOR gate outputs
predictions = model.predict(inputs)
print("Predictions:")
for i in range(len(inputs)):
    print(f"Input: {inputs[i]}, Predicted Output: {predictions[i][0]:.7f}")

# Print biases and weights
# (IMPORTANT NOTE! they are printed as w[i][j] not w[j][i] | outputs * inputs)
print()
weights_biases = model.get_weights()

print("#define _1_OPTIMIZE 0B01000000 // Highly-Recommended Optimization For RAM")
if IS_BIASED:
    print("#define _2_OPTIMIZE 0B00100000 // MULTIPLE_BIASES_PER_LAYER \n")
    print('float biases[] = {')
    for l, (w, b) in enumerate(zip(weights_biases[::2], weights_biases[1::2])):
        print('  ', end='')
        for j in range(0, w.shape[1]):
            print(b[j], end=', ')
        print()
    print('};\n')
else:
    print("#define _2_OPTIMIZE 0B01000000 // NO_BIAS \n")

print('float weights[] = {', end="")
for l, (w, b) in enumerate(zip(weights_biases[::2], weights_biases[1::2])):
    print()
    for j in range(0, w.shape[1]):
        print('  ', end='')
        for i in range(0, w.shape[0]):
            print(w[i][j], end=', ')
        print()
print('};\n')

Int quantization

Assuming you already have either enabled int16_t or int8_t... before proceeding with the example, you should know that the default range of weights (that maps floats to ints) , is set to 200.0 for int16_t and 51.0 for int8_t via this simple formula:

// FLOAT RANGE FOR INT16 = (100.0) - (-100.0) = 200.0 | MAX - MIN
// FLOAT RANGE FOR INT8  = ( 25.5) - (- 25.5) =  51.0 | MAX - MIN

You can change that by defining your own value in your sketch, like:

#define Q_FLOAT_RANGE 40.0 // (20.0) - (-20.0) = 40.0 | Ensure you used a dot!

IMPORTANT NOTE: See how weights and biases are printed at the end of the script and make sure you have (on top of your sketch) enabled\defined _2_OPTIMIZE 0B00100000 // MULTIPLE_BIASES_PER_LAYER or _2_OPTIMIZE 0B01000000 // NO_BIAS depending on your needs of use. Additionally, if you want to use just 1 bias per layer-to-layer don't use any of those 2 optimizations (Althought, just so you know... Tensorflow doesn't seem to support 1 bias per layer-to-layer). Finally make sure to use float32 unless your MCU is compatible and you want to USE_64_BIT_DOUBLE-optimization

(see also examples on how to train a NN directly on an MCU)

A HUGE THANK YOU!

I want to really thanks Underpower Jet for his amazing tutorial, by bringing it more to the surface. Because after all the videos and links I came across, he was the one that made the most significant difference to my understanding of backpropagation in neural networks. Plus, I would like to thanks: giant_neural_network for this and this, 3Blue1Brown for this, the authors of ✨ this scientific article for referencing me, Ivo Ljubičić for using my library for his ✨ master thesis, Arduino community and everyone else who gave me the oportunity to learn and make this library possible to exist [...]

🌐 Research

Here most of the resources I came across the internet, I recomend you to have a look if you want to (but please stay aware of the fact that for some of those sites, I had only opened them checked something and then closed them in a matter of seconds [so, please don't get them all seriously])

22\11\2023

• Code Related:
• • Handy C++ "cheat sheet"
• • Macros:
• • • Do not put comments in front of #define whatever
• • • is #ifdef __SD_H__ considered a bad practice?
• • • C++11-style [[unused]] attribute in gcc?
• • • comments in c macro definition
• • • Using #undef before #define
• • Arduino:
• • • StackOverflow:
• • • • PROGMEM : pgm_read_float_near() equivalent for double? (Arduino Due)
• • • • warning: inline variables are only available with ‘-std=c++17’ or ‘-std=gnu++17’ how to suppress it? [closed]
• Neural Network Related
• • General:
• • • Large values of weights in neural network

12\08\2021

• Neural Network Related
• • Videos:
• • • How convolutional neural networks work, in depth
• • • How to use the loss function categorical crossentropy
• • • Tutorial 4: How to train Neural Network with BackPropogation
• • • Neural Networks Part 8: Image Classification with Convolutional Neural Networks
• • Softmax:
• • • The Softmax : Data Science Basics
• • • Softmax Function in Deep Learning
• • • The SoftMax Derivative, Step-by-Step!!!
• • • Derivative of Sigmoid and Softmax Explained Visually
• • • Softmax and Cross Entropy Gradients for Backpropagation
• • StackOverflow\Exchange:
• • • What is batch size in neural network?
• • • how to do weight clipping after gradient updates?
• • • Cost function turning into nan after a certain number of iterations
• • • Why use softmax only in the output layer and not in hidden layers?
• • • Why is step function not used in activation functions in machine learning?
• • • if softmax is used as an activation function for output layer, must the number of nodes in the last hidden layer equal the number of output nodes?
• • General:
• • • Is Sigmoid the only activation function compatible with the binary crossentropy loss function
• • • Understanding binary cross-entropy / log loss: a visual explanation
• • • Understanding RMSprop
• • • MLP_C Arduino Library
• Code Related:
• • Tools:
• • • Compiler Explorer
• • Macros:
• • • 3.7.1 Standard Predefined Macros
• • • Diagnostic Pragmas
• • • PREPROCESSOR DIRECTIVES 2
• • Arduino:
• • • StackOverflow:
• • • • Why there's no erf() function in Arduino's math.h library?
• • • • How to Serial.println text and variable's values in the same line
• • • • Why AVR-GCC compiler throws an error when overloading with the same variables but as PROGMEM?
• • • General:
• • • • Natural Log of a Number
• General:
• • Math:
• • • What the cap over letters in math means
• • • When log is written without a base, is the equation normally referring to log base 10 or natural log?
• • Grammar:
• • • Plural of feedforward
• • Just "Random":
• • • Delving Deep into Rectifiers
• • • Linear & Identity Activation Functions
• • • “softmax derivative c++” Code Answer
• • • The Softmax function and its derivative
• • • -

xx\xx\202x

• Neural Network Related

• • kind of Intresting To me
• • • Training Deep Neural Nets
• • • Activation Functions - EXPLAINED!
• • • (Of course wiki lol) Activation function
• • • MNIST functions Intresting Simple Paper
• • General
• • • How many times is backprop used in epoch?
• • • Back-propagation in Neural Nets with >2 hidden layers
• • • Normalized Function, Normalized Data and Normalization
• • Activation Function Related
• • • Why the 6 in relu6?
• • • Are relus incapable of solving certain problems
• • • (Seems Intresting!) Modern Activations
• • • Machine learning using ReLu return NaN
• • • Machine learning using ReLu return NaN
• • • NaN loss when training regression network
• • • How to avoid NaN in using ReLU + Cross-Entropy? duplicate
• • • Activation Functions Explained - GELU, SELU, ELU, ReLU and more
• • • Compressing deep neural networks on FPGAs to binary and ternary precision with HLS4ML
• • • For deep learning, With activation relu the output becomes NAN during training while is normal with tanh
• • Gradient Explosion and clipping Related
• • • A clipped ReLU
• • • (Seems Intersting) Paper
• • • Complete Guide To Exploding Gradient Problem
• • • Understanding Gradient Clipping (and How It Can Fix Exploding Gradients Problem)
• • • Can the vanishing gradient problem be solved by multiplying the input of tanh with a coefficient?
• • • (Seems Intersting) Keras ML library: how to do weight clipping after gradient updates? TensorFlow backend
• • MNIST Related
• • • MNIST-Visualizer
• • • THE MNIST DATABASE
• • • ReLU vs sigmoid in mnist example
• • • Why not use more than 3 hidden layers for MNIST classification?
• • • Neural network classificator stuck at 85% accuracy for mnist dataset
• • • (Arduino Project) MNIST - (approach) downsampled to 9x9 - 170 neurons
• • • MNIST digit recognition: what is the best we can get with a fully connected NN only? (no CNN)
• • • What is the smallest model in terms of number of parameters that can achieve results over 99% on MNIST dataset?

• Related to Programming

• • to C-type Languages
• • • c++ making a function that always runs when any other function of a class is called
• • • int.. = {1,2,}; Weird comma allowed. Any particular reason?
• • • Declare an array of pointers to functions in Visual C++
• • • Function Pointer Array to a Method within a Class
• • • Inability to overload Dot '.' operator in c++
• • • How to get address of a pointer in c/c++?
• • • CSharp 16 bit float conversions
• • • Understanding * oppperator
• • • Array of function pointers
• • • Issue I came across c++
• • • Issue I came across c++
• • • c++ constexpr
• • • c++
• • to Python
• • • Python Floating Point Precision numpy
• • • Python Floating Point Precision Tensorflow
• • Other
• • • Aspect-oriented programming
• • • Metaprogramming

• Arduino Related

• • Reading and Writing Data to External EEPROM Using Arduino
• • avrdude: verification error, first mismatch at byte 0x7800
• • avrdude: verification error, first mismatch at byte 0x7800
• • KEYWORDS.TXT

• #MACROS / pre-processor directives

• • Is it possible to define same macro more than once?
• • (Very Useful) Print Content of Defined Variable
• • How to remove gcc warning on pragma region
• • (Very Useful) Generate Functions
• • (seems intresting) Add numbers
• • 3.8 Undefining and Redefining Macros
• • (Seems Intresting) Iterate through #defines
• • (Seems Intresting) Checking type of a variable
• • Macro functions
• • (Random)

#

Old Searches:

| ``` | Intresting |NN.| Neural Network(s) |A.| Arduino etc. |-| Mostly .NET & Other |*```| Maybe Intresting?

💗 Donation

Please consider donating something, even the least amount would be really appreciated. I need it. | Monero address: ^{87PVyQ8Vt768hnvVyR9Qw1NyGzDea4q9Zd6AuwHb8tQBU9VdRYjRoBL7Ya8yRPVQakW2pjt2UWEtzYoxiRd7xpuB4XSJVAW}

🕳️ Outro

• Forgive me if I've made any mistakes and please don't take me seriously with every claim i make, I am mainly "self taught" in this field of NeuralNetworks, I am not a professional programmer nor do I have a good knowledge in many of the fields used to create this library, I just make things because I love to [...]

• Also looking for jobs, if you are interested let me know, I really like working with embeded systems, C\C++, python, CLIs and etc.

if you want to help me&others to educate ourselves better and if you have a love and passion for sharing and helping, then I suggest you to join our discord server 🤍

My Instagram account is: giorgos.xou ;) feel free to ask me anything