search

ARM-software / ML-examples

Arm Machine Learning tutorials and examples
https://developer.arm.com/technologies/machine-learning-on-arm
Apache License 2.0
434 stars 195 forks source link

Integration of Tensorflow lite and CMSIS-NN #84

Open Satyapreet opened 2 years ago

Satyapreet commented 2 years ago

We are stuck at a problem that requires the weights and biases to be represented in Qm.n formats from the trained Tensorflow Lite model. The trained CNN model on Tensorflow lite or Brevitas provides int8 weights, biases, and output activations. The model also specifies the scaling value and zero point, which it uses to convert the float32 weights to int8 weights during the computation internally. A screenshot of the trained weights from one convolution layer is attached.

image

CMSIS-NN requires the int8 weights to be represented in Qm.n format. It is essential to know the fixed-point representation(Qm.n) since the CMSIS-NN API calls require us to left-shift the biases and right-shift the output activations for correct computation. This task is performed by the CMSIS-API when we provide the exact left shift and right shift values in the header file. A screenshot of the header file and the API call is attached.

image

image

The problem is we do not know how to convert the scaling factor and zero-point values to a Qm.n format. We wondered if there is a way to know the Qm.n formats of the weights, biases, and output activations from Tensorflow lite. CMSIS-NN help guide provided by ARM mentions specific scripts that directly do these conversions like code_gen.py, but unfortunately, these help scripts are no longer available on the Github page (https://github.com/ARM-software/ML-examples/blob/master/cmsisnn-cifar10/code_gen.py). It would be great if you can help.

Burton2000 commented 2 years ago

For that specific link you can view it from the history in Git so this should work: https://github.com/ARM-software/ML-examples/blob/49627dc5be8824a774429a4fec32f6db88bd522a/cmsisnn-cifar10/code_gen.py but not sure how useful it will be for your issue.

Generally, we would recommend using TensorFlow Lite Micro built with CMSIS-NN to use CMSIS-NN acceleration rather than using just CMSIS-NN. This is the way that we have done it in this example (https://github.com/ARM-software/ML-examples/tree/master/tflu-kws-cortex-m).

Using TensorFlow Lite Micro will make things easier for you to deploy your model and perform inference compared to using bare CMSIS-NN and does not require much more memory from your application.

satyapreetsingh commented 2 years ago

@Burton2000

I am using a TI-based Cortex-M4 board. Can you please help guide me as to how can I compile the TensorFlow Lite Micro library onto that board? Is there any guide for the same?

Also, If I have a tflite model, How do I convert it to a C code. How do I dump that C code into my MCU?

Burton2000 commented 2 years ago

I believe TensorFlow provides some nice tutorials for getting started with TensorFlow Lite Micro: https://www.tensorflow.org/lite/microcontrollers

In particular for converting the model to C code this part of their guides shows how to do this: https://www.tensorflow.org/lite/microcontrollers/build_convert#convert_to_a_c_array