Arduino BLE stuck on running IMU Tiny ML no output from model

Hi, I was following this tutorial of Arduino BLE 33 Sense to use it like a controler of Street Fighter on Raspberry Pi, I tried to use the BLE library to make it wireless with a baterry but I have problems running it like for two weeks, basically it get stuck on running the model I´m not sure if is because of the Bluetooth. Or something else but it don't show me the output from the gestures. It conects but after that it just show me on the serial monitor this:

Connecting ... Connected

on my Python code that receives all the information I tried with a simple int and it works but when I tried to use the model don't give me any result from the gestureid.

I would be awesome if someone can help me with this.

#include <ArduinoBLE.h>
#include <Arduino_LSM9DS1.h>

#include <TensorFlowLite.h>
#include <tensorflow/lite/micro/all_ops_resolver.h>
#include <tensorflow/lite/micro/micro_error_reporter.h>
#include <tensorflow/lite/micro/micro_interpreter.h>
#include <tensorflow/lite/schema/schema_generated.h>
#include <tensorflow/lite/version.h>

#include "model2.h"

BLEService sensorService("00001101-0000-1000-8000-00805f9b34fb");
//TX Characteristics
BLEStringCharacteristic txChar("00001143-0000-1000-8000-00805f9b34fb", BLERead | BLENotify,15);     

// last sensor data    
float oldXLevel = 0; 
float oldYLevel = 0; 
float oldZLevel = 0; 
long previousMillis = 0; 

const float accelerationThreshold = 2.5; // threshold of significant in G's
const int numSamples = 119;

int samplesRead = numSamples;
int gestureid=0;

// global variables used for TensorFlow Lite (Micro)
tflite::MicroErrorReporter tflErrorReporter;

// pull in all the TFLM ops, you can remove this line and
// only pull in the TFLM ops you need, if would like to reduce
// the compiled size of the sketch.
tflite::AllOpsResolver tflOpsResolver;

const tflite::Model* tflModel = nullptr;
tflite::MicroInterpreter* tflInterpreter = nullptr;
TfLiteTensor* tflInputTensor = nullptr;
TfLiteTensor* tflOutputTensor = nullptr;

// Create a static memory buffer for TFLM, the size may need to
// be adjusted based on the model you are using
constexpr int tensorArenaSize = 8 * 1024;
byte tensorArena[tensorArenaSize] __attribute__((aligned(16)));

// array to map gesture index to a name
const char* GESTURES[] = {
  "punch",
  "defend",
  "summon"
};

#define NUM_GESTURES (sizeof(GESTURES) / sizeof(GESTURES[0]))

void setup() {
  Serial.begin(115200);  
  while (!Serial);

  if (!IMU.begin()) {
    Serial.println("Failed to initialize IMU!");
    while (1);
  }

  pinMode(LED_BUILTIN, OUTPUT); 

  if (!BLE.begin()) {
    Serial.println("starting BLE failed!");
    while (1);
  }

  // get the TFL representation of the model byte array
  tflModel = tflite::GetModel(model);
  if (tflModel->version() != TFLITE_SCHEMA_VERSION) {
    Serial.println("Model schema mismatch!");
    while (1);
  }

  // Create an interpreter to run the model
  tflInterpreter = new tflite::MicroInterpreter(tflModel, tflOpsResolver, tensorArena, tensorArenaSize, &tflErrorReporter);

  // Allocate memory for the model's input and output tensors
  tflInterpreter->AllocateTensors();

  // Get pointers for the model's input and output tensors
  tflInputTensor = tflInterpreter->input(0);
  tflOutputTensor = tflInterpreter->output(0);

  BLE.setLocalName("NanoBLE33");
  BLE.setAdvertisedService(sensorService);

  sensorService.addCharacteristic(txChar);
  BLE.addService(sensorService); 

  // initialize default data
  txChar.writeValue(String(0)); 

  // start advertising
  BLE.advertise();
  Serial.println("Bluetooth device active, waiting for connections...");
}

void loop() {
  // wait for a BLE central
  BLEDevice central = BLE.central();
  if (central) {
    Serial.print("Connected to central: ");
    Serial.println(central.address());
    digitalWrite(LED_BUILTIN, HIGH);

    while (central.connected()) {
      //long currentMillis = millis();
      updateGyroscopeLevel();
      delay(300);
    }

    digitalWrite(LED_BUILTIN, LOW);
    Serial.print("Disconnected from central: ");
    Serial.println(central.address());
  }
}

void updateGyroscopeLevel() {

  float aX, aY, aZ, gX, gY, gZ;

  // check if new acceleration AND gyroscope data is available
  if (IMU.accelerationAvailable() && IMU.gyroscopeAvailable()) {
    // read the acceleration and gyroscope data
    IMU.readAcceleration(aX, aY, aZ);
    IMU.readGyroscope(gX, gY, gZ);

    // normalize the IMU data between 0 to 1 and store in the model's
    // input tensor
    tflInputTensor->data.f[samplesRead * 6 + 0] = (aX + 4.0) / 8.0;
    tflInputTensor->data.f[samplesRead * 6 + 1] = (aY + 4.0) / 8.0;
    tflInputTensor->data.f[samplesRead * 6 + 2] = (aZ + 4.0) / 8.0;
    tflInputTensor->data.f[samplesRead * 6 + 3] = (gX + 2000.0) / 4000.0;
    tflInputTensor->data.f[samplesRead * 6 + 4] = (gY + 2000.0) / 4000.0;
    tflInputTensor->data.f[samplesRead * 6 + 5] = (gZ + 2000.0) / 4000.0;

    samplesRead++;

    if (samplesRead == numSamples) {
      // Run inferencing
      TfLiteStatus invokeStatus = tflInterpreter->Invoke();
      if (invokeStatus != kTfLiteOk) {
        Serial.println("Invoke failed!");
        while (1);
        return;
      }

      // Loop through the output tensor values from the model
      for (int i = 0; i < NUM_GESTURES; i++) {
        if(tflOutputTensor->data.f[i]>0.7)
        {
          gestureid=i+1;
          txChar.writeValue(String(gestureid));
          Serial.print(txChar.writeValue(String(gestureid)));
        }
      }

      Serial.print(gestureid);
      gestureid=0;
      //Serial.println();
    }
  }
}

Dear Eduardo,

Is your model trained properly. Could you please send the Tensorflow Screenshot. Please note excel csv will not work! You have to use a excel to csv converter. Did you get 0% continually. Please let me know.

Regards, Bharat

On Tue, 6 Jul 2021, 07:20 Eduardo Padron, @.***> wrote:

Hi, I was following this tutorial https://create.arduino.cc/projecthub/zoromoth/machine-learning-with-nano-ble-33-raspberry-pi-958b2f of Arduino BLE 33 Sense to use it like a controler of Street Fighter on Raspberry Pi, I tried to use the BLE library to make it wireless with a baterry but I have problems running it like for two weeks, basically it get stuck on running the model I´m not sure if is because of the Bluetooth. Or something else but it don't show me the output from the gestures. It conects but after that it just show me on the serial monitor this:

Connecting ... Connected

on my Python code that receives all the information I tried with a simple int and it works but when I tried to use the model don't give me any result from the gestureid.

I would be awesome if someone can help me with this.

include

include

include

include <tensorflow/lite/micro/all_ops_resolver.h>

include <tensorflow/lite/micro/micro_error_reporter.h>

include <tensorflow/lite/micro/micro_interpreter.h>

include <tensorflow/lite/schema/schema_generated.h>

include <tensorflow/lite/version.h>

include "model2.h"

BLEService sensorService("00001101-0000-1000-8000-00805f9b34fb");

//TX Characteristics

BLEStringCharacteristic txChar("00001143-0000-1000-8000-00805f9b34fb", BLERead | BLENotify,15);

// last sensor data

float oldXLevel = 0;

float oldYLevel = 0;

float oldZLevel = 0;

long previousMillis = 0;

const float accelerationThreshold = 2.5; // threshold of significant in G's

const int numSamples = 119;

int samplesRead = numSamples;

int gestureid=0;

// global variables used for TensorFlow Lite (Micro)

tflite::MicroErrorReporter tflErrorReporter;

// pull in all the TFLM ops, you can remove this line and

// only pull in the TFLM ops you need, if would like to reduce

// the compiled size of the sketch.

tflite::AllOpsResolver tflOpsResolver;

const tflite::Model* tflModel = nullptr;

tflite::MicroInterpreter* tflInterpreter = nullptr;

TfLiteTensor* tflInputTensor = nullptr;

TfLiteTensor* tflOutputTensor = nullptr;

// Create a static memory buffer for TFLM, the size may need to

// be adjusted based on the model you are using

constexpr int tensorArenaSize = 8 * 1024;

byte tensorArena[tensorArenaSize] attribute((aligned(16)));

// array to map gesture index to a name

const char* GESTURES[] = {

"punch",

"defend",

"summon"

};

define NUM_GESTURES (sizeof(GESTURES) / sizeof(GESTURES[0]))

void setup() {

Serial.begin(115200);

while (!Serial);

if (!IMU.begin()) {
Serial.println("Failed to initialize IMU!");

while (1);
}

pinMode(LED_BUILTIN, OUTPUT);

if (!BLE.begin()) {
Serial.println("starting BLE failed!");

while (1);
}

// get the TFL representation of the model byte array

tflModel = tflite::GetModel(model);

if (tflModel->version() != TFLITE_SCHEMA_VERSION) {
Serial.println("Model schema mismatch!");

while (1);
}

// Create an interpreter to run the model

tflInterpreter = new tflite::MicroInterpreter(tflModel, tflOpsResolver, tensorArena, tensorArenaSize, &tflErrorReporter);

// Allocate memory for the model's input and output tensors

tflInterpreter->AllocateTensors();

// Get pointers for the model's input and output tensors

tflInputTensor = tflInterpreter->input(0);

tflOutputTensor = tflInterpreter->output(0);

BLE.setLocalName("NanoBLE33");

BLE.setAdvertisedService(sensorService);

sensorService.addCharacteristic(txChar);

BLE.addService(sensorService);

// initialize default data

txChar.writeValue(String(0));

// start advertising

BLE.advertise();

Serial.println("Bluetooth device active, waiting for connections...");

}

void loop() {

// wait for a BLE central

BLEDevice central = BLE.central();

if (central) {
Serial.print("Connected to central: ");

Serial.println(central.address());

digitalWrite(LED_BUILTIN, HIGH);

while (central.connected()) {

  //long currentMillis = millis();

  updateGyroscopeLevel();

  delay(300);

}

digitalWrite(LED_BUILTIN, LOW);

Serial.print("Disconnected from central: ");

Serial.println(central.address());
}

}

void updateGyroscopeLevel() {

float aX, aY, aZ, gX, gY, gZ;

// check if new acceleration AND gyroscope data is available

if (IMU.accelerationAvailable() && IMU.gyroscopeAvailable()) {
// read the acceleration and gyroscope data

IMU.readAcceleration(aX, aY, aZ);

IMU.readGyroscope(gX, gY, gZ);

// normalize the IMU data between 0 to 1 and store in the model's

// input tensor

tflInputTensor->data.f[samplesRead * 6 + 0] = (aX + 4.0) / 8.0;

tflInputTensor->data.f[samplesRead * 6 + 1] = (aY + 4.0) / 8.0;

tflInputTensor->data.f[samplesRead * 6 + 2] = (aZ + 4.0) / 8.0;

tflInputTensor->data.f[samplesRead * 6 + 3] = (gX + 2000.0) / 4000.0;

tflInputTensor->data.f[samplesRead * 6 + 4] = (gY + 2000.0) / 4000.0;

tflInputTensor->data.f[samplesRead * 6 + 5] = (gZ + 2000.0) / 4000.0;

samplesRead++;

if (samplesRead == numSamples) {

  // Run inferencing

  TfLiteStatus invokeStatus = tflInterpreter->Invoke();

  if (invokeStatus != kTfLiteOk) {

    Serial.println("Invoke failed!");

    while (1);

    return;

  }

  // Loop through the output tensor values from the model

  for (int i = 0; i < NUM_GESTURES; i++) {

    if(tflOutputTensor->data.f[i]>0.7)

    {

      gestureid=i+1;

      txChar.writeValue(String(gestureid));

      Serial.print(txChar.writeValue(String(gestureid)));

    }

  }

  Serial.print(gestureid);

  gestureid=0;

  //Serial.println();

}
}

}

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/bharatr1993/Arduino-Nano-33-BLE/issues/1, or unsubscribe https://github.com/notifications/unsubscribe-auth/APDV66KDJDCEFUH2DCFO6VLTWJOPLANCNFSM473TGWKQ .

Dear @bharatr1993 I'm using your tutorial Machine Learning with Nano BLE 33 & Raspberry Pi © MIT, the only change that I made was adding the BLE library to connect it via bluetooth. So I didn't change the model2.h that is in this repository. if you she code I'm using the same code lines and just added the BLE lines. But it doen't start the IMUmodel when I try to move the Arduino.

Dear Eduardo,

Yes, you are right... I remember now, as I had made it years earlier. Even when I tried to connect through BLE, it did hang up. But the reason is simple... BLE library is a while loop that doesn't exit... Which means execution never reaches the model code and your code too is an infinite loop.

You will have to create separate threads using library of C++ and then pass data between the threads. I didn't do this part though!

Regards, Bharat

On Tue, 6 Jul 2021, 09:28 Eduardo Padron, @.***> wrote:

Dear @bharatr1993 https://github.com/bharatr1993 I'm using your tutorial Machine Learning with Nano BLE 33 & Raspberry Pi © MIT https://create.arduino.cc/projecthub/zoromoth/machine-learning-with-nano-ble-33-raspberry-pi-958b2f?f=1#comments , the only change that I made was adding the BLE library to connect it via bluetooth. So I didn't change the model2.h that is in this repository. if you she code I'm using the same code lines and just added the BLE lines. But it doen't start the IMUmodel when I try to move the Arduino.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bharatr1993/Arduino-Nano-33-BLE/issues/1#issuecomment-874439935, or unsubscribe https://github.com/notifications/unsubscribe-auth/APDV66OGVLCMQ6J4JPNEJQTTWJ5ODANCNFSM473TGWKQ .

Dear Bharat,

I tried but is the first time doing the split for this but if you have some example or documentation to use it for it it would be great and thanks for the answer.

Regards, Eduardo

Dear Eduardo,

I think this stackoverflow post is a good place to start:

https://stackoverflow.com/questions/266168/simple-example-of-threading-in-c

I will explain the basics,

std::thread is the object of the thread library.

Use std::thread.create to make a thread..

You will have to create 2 threads, one for bluetooth and another for your model. Put both the infinite loops in a external function.

Then pass the function name as a paramet to the std::thread.create function. Please refer to the post for exact syntax!

Then create a global variable gesture id that will be written by your model code and read by the BLE code. Let the functions you created earlier handle this...

Thats it, ensure you do not reach end of main by putting a while(1) at the bottom as the function has the loop now...

The program will never exit but it should be sufficient for you to try.

This is just my logic, I do not have Computer Science background this is just an idea of mine.

Please do try and let me know.

Regards, Bharat

On Tue, 6 Jul 2021, 09:44 Eduardo Padron, @.***> wrote:

Dear Bharat,

I tried but is the first time doing the split for this but if you have some example or documentation to use it for it it would be great and thanks for the answer.

Regards, Eduardo

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bharatr1993/Arduino-Nano-33-BLE/issues/1#issuecomment-874445450, or unsubscribe https://github.com/notifications/unsubscribe-auth/APDV66IOZOCYW2IIXYMVBALTWJ7K7ANCNFSM473TGWKQ .

Dear Bharat, I almost get it but I need help with one thing, I need a function like this:

if value != "no value detected"

Serial.println (value)

Basically I need a code line to know when a move was made or not if not set a value like " "

Regards, Eduardo

Try

if(value != "")

As is

On Fri, 13 Aug 2021, 08:28 Eduardo Padron, @.***> wrote:

Dear Bharat, I almost get it but I need help with one thing, I need a function like this:

if value != "no value detected"

Serial.println (value)

Basically I need a code line to know when a move was made or not if not set a value like " "

Regards, Eduardo

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bharatr1993/Arduino-Nano-33-BLE/issues/1#issuecomment-898145747, or unsubscribe https://github.com/notifications/unsubscribe-auth/APDV66JPNTASDURZHMKWV5LT4SC4FANCNFSM473TGWKQ .

Another option is set the value to something like -1 or some string if nothing is detected (else condition) such that you can put it in an if condition

On Fri, 13 Aug 2021, 09:36 Bharat R, @.***> wrote:

Try

if(value != "")

As is

On Fri, 13 Aug 2021, 08:28 Eduardo Padron, @.***> wrote:

Dear Bharat, I almost get it but I need help with one thing, I need a function like this:

if value != "no value detected"

Serial.println (value)

Basically I need a code line to know when a move was made or not if not set a value like " "

Regards, Eduardo

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bharatr1993/Arduino-Nano-33-BLE/issues/1#issuecomment-898145747, or unsubscribe https://github.com/notifications/unsubscribe-auth/APDV66JPNTASDURZHMKWV5LT4SC4FANCNFSM473TGWKQ .

Thanks I forgot that I need to use this function in the gesture id part of your code, because the BLE library need a delay or set a value and then with that I can send the value by Bluetooth conection, so I'm trying to set the value if the value was 1 or 2 until a new move is detected the value is set as the last value detected so I want to try, a code line with a condition like if not movement detected set gesture id as " ".

bharatr1993 / Arduino-Nano-33-BLE

Arduino BLE stuck on running IMU Tiny ML no output from model #1

include

include

include

include <tensorflow/lite/micro/all_ops_resolver.h>

include <tensorflow/lite/micro/micro_error_reporter.h>

include <tensorflow/lite/micro/micro_interpreter.h>

include <tensorflow/lite/schema/schema_generated.h>

include <tensorflow/lite/version.h>

include "model2.h"

define NUM_GESTURES (sizeof(GESTURES) / sizeof(GESTURES[0]))