TFLite-Micro doesn't support models quantized to float16?

[zhuweizzz]

I built and trained a CNN model using TensorFlow, and used the TensorFlow Lite Converter to convert the trained model into a TFLite model.

# Create a model using high-level tf.keras.* APIs
model = tf.keras.models.load_model('./ECG_net_tf.h5')

# Convert the model.
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()

# Save the model.
with open('saved_models/model.tflite', 'wb') as f:
  f.write(tflite_model)

I compiled and successfully ran the model in TFLite-Micro.

I noticed that it's possible to perform float16 quantization when using the Converter.

# Create a model using high-level tf.keras.* APIs
model = tf.keras.models.load_model('./saved_models/ECG_net_tf.h5')

# Convert the model.
converter = tf.lite.TFLiteConverter.from_keras_model(model)

#float16 Quantization
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_types = [tf.float16]

tflite_model = converter.convert()

# Save the model.
with open('saved_models/model.tflite', 'wb') as f:
  f.write(tflite_model)

The quantized model compiled successfully in TFLite-Micro, but encountered errors during runtime.

tensorflow/lite/micro/kernels/dequantize_common.cc:45 input->type == kTfLiteInt8 || input->type == kTfLiteInt16 || input->type == kTfLiteUInt8 was not true.
Node DEQUANTIZE (number 0f) failed to prepare with status 1
Segmentation fault

Based on the error message, I located the problematic area in the file tensorflow/lite/micro/kernels/dequantize_common.cc, where the input type check for the Dequantize operation on line 43 seems to only support Int8, Int16, and Uint8.

  TF_LITE_ENSURE(context, input->type == kTfLiteInt8 ||
                              input->type == kTfLiteInt16 ||
                              input->type == kTfLiteUInt8
                              );

However, in a TFLite model quantized to float16, the input for the Dequantize operation is of float16 type. Does this mean that TFLite-Micro does not support models quantized to float16?

[rascani]

That is correct, TFLite-Micro does not support models quantized to float16. This is because the hardware targets using TFLite-Micro rarely have hardware floating point support, so it becomes prohibitively expensive to use a floating point model in production. We currently only support float32 with reference kernels. We generally recommend using full integer quantization. If applicable for your use case, TFLite does support float16.