Coral

[ebgoldstein]

compile for edge TPU: https://coral.ai/docs/edgetpu/compiler/

And see how much faster the model is compared to the fp16 model (4s per inference)

[ebgoldstein]

Steps:

0. Use post-training quantization to produce a model with Full integer quantization — https://www.tensorflow.org/lite/performance/post_training_quantization#full_integer_quantization)
1. Compile an edgetpu compliant model from a TFLite model via the compiler (or the colab linked in the compiler docs): https://coral.ai/docs/edgetpu/compiler/
2. Put new model on device
3. Install EdgeTPU runtime on RPi: https://coral.ai/docs/accelerator/get-started/#runtime-on-linux
4. Install PyCoral on RPi: https://coral.ai/docs/accelerator/get-started/#2-install-the-pycoral-library
5. modify script to use pycoral https://coral.ai/docs/edgetpu/tflite-python/

[ebgoldstein]

note that currently our model uses float 16 quantization (and another model uses Dynamics range quantization). Full integer reqs some code tweaks, and a representative dataset...

[ebgoldstein]

one potential idea is the do full integer quant. and just omit quantizing the output.. i..e, include:

converter.inference_input_type = tf.int8 # or tf.uint8 but omit: converter.inference_output_type = tf.int8 # or tf.uint8

the model would run on the TPU except for the last layer, which would run on the CPU..

https://stackoverflow.com/questions/66984379/problem-in-conversion-of-pb-to-tflite-int8-for-coral-devboard-coral-ai

[ebgoldstein]

I'm going to close this for now — i think 4s per inference is pretty good, and 8-bit quant seems like it will be challenging to do for a regression model..