search

mit-han-lab / tinyengine

[NeurIPS 2020] MCUNet: Tiny Deep Learning on IoT Devices; [NeurIPS 2021] MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning; [NeurIPS 2022] MCUNetV3: On-Device Training Under 256KB Memory
https://mcunet.mit.edu
MIT License
806 stars 131 forks source link

Using TinyEngine with TensorFlow Lite custom models & different/custom datasets. #9

Closed antragoudaras closed 1 year ago

antragoudaras commented 2 years ago

Is it possible to capture/improve the performance of (in terms of accuracy and peak memory usage) a custom tflite already trained model (that I converted from an originally simple keras model) using TinyEngine, when compared to the plain TensorFlow Lite implementation of the same model? Also do I need to add any extra functionality to the existing code-base in order to evaluate my model against my own dataset(dataset form: training & validation sets as numpy arrays, classification problem with 4 classes)?

Any suggestion/guidance would be deeply appreciated on how to conduct the performance analysis described above by using the TinyEngine inference library, given that my model only supports compatible TinyEngine operators(aka neural net layers). -Antonios. p.s. Novice fan/user of TinyEngine.

meenchen commented 2 years ago

Hi @antragoudaras thanks for your interest in our work. You can deploy your model with TFLite micro and Tinyengine to compare the performance improvement. You can refer to TFLite micro to see how to deploy your TFLite model. For Tinyenigne, you can refer to our examples. If you also want to evaluate the accuracy on-device, you may need to implement some evaluation scripts (both on MCU and PC) so that your PC can communicate with the MCU through a serial port and evaluate each input in the dataset.

meenchen commented 1 year ago

Close due to inactivity. Feel free to reopen.

ellial commented 1 year ago

Hi @meenchen . I am trying to validate the on-device accuracy for the visual wake words dataset. Is there any chance you could reopen this issue and share the scripts you used when measuring the on-device accuracy for STM32?

meenchen commented 1 year ago

Hi @ellial,

Our quantization implementation basically follows tensorflow lite micro, so we evaluate the accuracy on servers with TF Lite runtime (example: https://github.com/mit-han-lab/mcunet/blob/be404ea0dbb7402783e1c825425ac257ed35c5fc/eval_tflite.py). We are also working on supporting different ISA like x86 or ARM. It will be much easier to evaluate the on-device accuracy.