Feature Request: ml5 with Arduino Microcontrollers using webSerial

[hpssjellis]

I am working on a proof of concept of using TensorflowJS with microcontroller sensors to build machine learning models that can connect to microcontroller actuators after classification using webSerial and hopefully polyfill for Android. I feel that ml5js and p5js could probably do the same thing, but more professionally than my skills.

Is anyone on the ML5js team capable with Arduino MBED style microcontrollers such as the Nano33BleSense or the PortentaH7 or the new Seeedstudio XIAO SAMD or XIAO esp32S3?

Here is a youtube video of taking a Huggingface model and having the Arduino Servo motor respond to the classifications. I am working on Sensor uploading of data to tensorflowJS but testing needs an Arduino, hence the question about if anyone on your team is familiar with Arduino's.

I am not that interested in the extra steps of converting the tensorflowJS model to TensorflowLite and then using xxd -i to convert it into a c-header file for compiling to the Arduino.

[shiffman]

Hi @hpssjellis yes, this is a wonderful idea! At the moment, the project is undergoing a rebuild in order to refine the models and update all the dependencies. There is a team of students at NYU in NY and also NYU Shanghai that are collaborating this summer on this work. This work is happening in this repo at the moment:

https://github.com/ml5js/ml5-next-gen

I think this would be a wonderful addition to the project, do you see it requiring new features in m5.js itself or is it more about providing an example with webSerial? @yining1023 may also be interested as she has taught a class around ML + physical computing.

[hpssjellis]

Thanks @shiffman for replying. I will be presenting my findings at an education conference in Italy https://indico.ictp.it/event/10185 July 3-7, 2023. I am a part of tinyML4D - tinyml.seas.harvard.edu/team bringing ML to university students in developing countries (Look for me Jeremy Ellis, the only high school teacher in the group ). I have already met one of the p5js contributors @gohai in this thread who used my suggestions about polyfill (an android Pixel phone fallback) for webSerial to bring it to p5js.

Thanks for the link, my research about webSerial with machine learning such as edgeimpulse.com or huggingface is here. I will be working on vanilla Javascript files, but thought you could bring webSerial for microcontrollers with machine learning to a wider audience. Hopefully we can kind of work together.

[sproutleaf]

I'm transferring the issue to our new repo: https://github.com/ml5js/ml5-next-gen for more updated discussion and possible issue assignment.

[hpssjellis]

@shiffman and @sproutleaf now that it is the fall and students are back, is anyone interested in merging TensorflowJS with Arduinos? I have basically done all the work, I just need someone to make it compatible with the p5/ml5 community.

The steps are:

1. TinyMLjs. This webpage uses WebSerial to connect an Arduino (Nano33BleSense) to a webpage and then train a tensorflowJS model and export the model.json and .bin shard files.

2. Gitpod tfjs-converter. This gitpod (docker in the browser) Uses tensorflowjs-converter to convert the model.json file to keras then tflite and finally to a c.header text readable file to load onto an Arduino

3. Arduino examples. This repository shows multiple arduino sketches that work with the c.header file (Note: This step I am working on as my working code has been deprecated by the latest arduino tensorflowlite)

None of this is easy or has been well documented but I am willing to help someone wanting to use the ideas for ml5. I will continue to improve my methods. Bes way to reach me is probably to continue this issues thread and add me @hpssjellis .