search

mocleiri / tensorflow-micropython-examples

A custom micropython firmware integrating tensorflow lite for microcontrollers and ulab to implement the tensorflow micro examples.
MIT License
183 stars 87 forks source link

Micro speech example based on Edge Impulse #74

Open MichaelDu9226 opened 2 years ago

MichaelDu9226 commented 2 years ago

Hi, mocleiri.

After I successfully completed the image classification based on edge impulse and tensorflow-micropython-examples, I began to try the audio classification based on edge impulse.

At the beginning, my idea was the same as that of image classification. First, I trained the micro speech model based on edge impulse, and then imported the model into tensorflow-micropython-examples or tflite-micro-esp-examples. But I found that there were some problems with audio classification.

After some digging, I found that the audio feature provider of TF Lite Micro may be different from edge impulse. In the edge impulse, I choose MFE.

As shown in the following picture, it seems that the main parameters are the same as those of TF Lite micro, but many other parameters are still different, like log_scale and noise_reduction. image image

It seems that the best way is to use edge impulse SDK directly, which can ensure that the audio features provider is consistent with the model trained on edge impulse.

I see you mentioned here that you will look at wrapping the edge impulse sdk in micropython. Do you have any plans for this feature recently? https://forum.edgeimpulse.com/t/add-tensorflow-micropython-examples-as-edge-impulse-runtime/3628/2

I am not very professional in SDK adaptation, but I can provide testing and documentation support. Or if you don't have the time recently, can you provide some suggestions? Let me try. Thanks.

This is my micro speech project public on edge impulse. https://studio.edgeimpulse.com/public/81932/latest

mocleiri commented 2 years ago

The spectrogramming is all over the place with tensorflow.

See here: https://groups.google.com/a/tensorflow.org/g/micro/c/vzdrYXHF3Iw/m/YOHvtN1FAwAJ

The code you show is just copying the configuration used in the C++ example. If you are able to change the values to match what your model needs we could create a configuration object that would allow those values to be set from micropython.

Thanks for the example, I will try to use it as the driver for figuring out how to wrap their sdk into micropython.