Cannot convert ssd OD models into tflite

[xumengwei]

The pre-trained model I used: ssd_mobilenet_v1_fpn_640x640_coco17_tpu-8. The code I used is below:

converter = tf.lite.TFLiteConverter.from_saved_model(model_dir)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_model_quant = converter.convert()
interpreter = tf.lite.Interpreter(model_content=tflite_model_quant)

When I try on the saved model generated through object_detection/export_tflite_graph_tf2.py as described here, I got the following error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-6-ed31809d0211> in <module>
     15 tflite_model_quant = converter.convert()
     16 
---> 17 interpreter = tf.lite.Interpreter(model_content=tflite_model_quant)
     18 input_type = interpreter.get_input_details()[0]['dtype']
     19 print('input: ', input_type)

~/anaconda3/lib/python3.7/site-packages/tensorflow/lite/python/interpreter.py in __init__(self, model_path, model_content, experimental_delegates, num_threads)
    206       self._interpreter = (
    207           _interpreter_wrapper.CreateWrapperFromBuffer(
--> 208               model_content, self._custom_op_registerers))
    209     elif not model_content and not model_path:
    210       raise ValueError('`model_path` or `model_content` must be specified.')

ValueError: Did not get operators or tensors in subgraph 0.

I also tried the saved_model coming with the original pre-trained checkpoint coming from the model zoo, I got another error:

---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
~/anaconda3/lib/python3.7/site-packages/tensorflow/lite/python/convert.py in toco_convert_protos(model_flags_str, toco_flags_str, input_data_str, debug_info_str, enable_mlir_converter)
    198                                                  debug_info_str,
--> 199                                                  enable_mlir_converter)
    200       return model_str

~/anaconda3/lib/python3.7/site-packages/tensorflow/lite/python/wrap_toco.py in wrapped_toco_convert(model_flags_str, toco_flags_str, input_data_str, debug_info_str, enable_mlir_converter)
     37       debug_info_str,
---> 38       enable_mlir_converter)
     39 

Exception: <unknown>:0: error: loc("Func/StatefulPartitionedCall/input/_0"): requires all operands and results to have compatible element types
<unknown>:0: note: loc("Func/StatefulPartitionedCall/input/_0"): see current operation: %1 = "tf.Identity"(%arg0) {device = ""} : (tensor<1x?x?x3x!tf.quint8>) -> tensor<1x?x?x3xui8>

My ultimate goal is to test the accuracy of OD models after quantization. Now I cannot even convert model, let alone quantization. Can anyone help me on this?

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 18.04):
• TensorFlow installed from (binary through pip):
• TensorFlow version (2.3.1):
• Python version: 3.7.6

[huberemanuel]

Are you using the latest tf-nightly version? This is needed dependency as mentioned here #9394

[xumengwei]

Thanks @huberemanuel installing tf-nightly solved the above problem. However I still have a question.

I followed the step here for different ways to quantize the model. The original model size is 124630308 bytes. When I try "Convert using dynamic range quantization", the model size becomes 32380912 bytes. When I try "Convert using float fallback quantization", the model size is 32671552 bytes. But according to the tutorial, this approach should be able to further reduce the model size. Can you explain what's the difference between the two?

In addition, trying "Convert using integer-only quantization" gives the following error:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-52-a25e448a979d> in <module>
     11 converter.inference_output_type = tf.uint8
     12 
---> 13 tflite_model_quant = converter.convert()
     14 
     15 interpreter = tf.lite.Interpreter(model_content=tflite_model_quant)

~/anaconda3/lib/python3.7/site-packages/tensorflow/lite/python/lite.py in convert(self)
    760     calibrate_and_quantize, flags = quant_mode.quantizer_flags()
    761     if calibrate_and_quantize:
--> 762       result = self._calibrate_quantize_model(result, **flags)
    763 
    764     flags_modify_model_io_type = quant_mode.flags_modify_model_io_type(

~/anaconda3/lib/python3.7/site-packages/tensorflow/lite/python/lite.py in _calibrate_quantize_model(self, result, inference_input_type, inference_output_type, activations_type, allow_float)
    479       return calibrate_quantize.calibrate_and_quantize(
    480           self.representative_dataset.input_gen, inference_input_type,
--> 481           inference_output_type, allow_float, activations_type)
    482 
    483   def _is_unknown_shapes_allowed(self):

~/anaconda3/lib/python3.7/site-packages/tensorflow/lite/python/optimize/calibrator.py in calibrate_and_quantize(self, dataset_gen, input_type, output_type, allow_float, activations_type, resize_input)
    102         np.dtype(input_type.as_numpy_dtype()).num,
    103         np.dtype(output_type.as_numpy_dtype()).num, allow_float,
--> 104         np.dtype(activations_type.as_numpy_dtype()).num)
    105 
    106   def calibrate_and_quantize_single(self,

RuntimeError: Quantization not yet supported for op: 'CUSTOM'.

The issue has been reported in a previous post but seems there's no solution yet. Is it because full-integer quantization is not supported to SSD model yet?

[huberemanuel]

@xumengwei I think it would be nice to open your new questions on a new issue, to make it easily accessible.