RetinaNet compilation fails due to dynamic-size tensors

[samagalhaes]

Description

Dear all,

I am trying to compile a RetinaNet ResNet50 to a USB Coral TPU. I wrote the network using the Sequential API of TF2 Keras. After training the network, I removed the incompatible layers, i.e. the pre and post-processing layers and made a post-training quantisation.

For the network quantisation, I used the following code:

... # omitted code

model = tf.keras.models.load_model(model_path, custom_objects={"RetinaNetLoss": RetinaNetLoss}, compile=False)

... # omited code

def representative_dataset ():
  ds = train_dataset.take(10)
  for data in ds:
    yield [data[0]]

"""
## Quantization
"""

converter = tf.lite.TFLiteConverter.from_keras_model(model)

# This enables quantization
converter.optimizations = [tf.lite.Optimize.DEFAULT]
# This sets the representative dataset for quantization
converter.representative_dataset = representative_dataset
# This ensures that if any ops can't be quantized, the converter throws an error
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
# For full integer quantization, though supported types defaults to int8 only, we explicitly declare it for clarity.
converter.target_spec.supported_types = [tf.int8]
# These set the input and output tensors to uint8 (added in r2.3)
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8

tflite_model = converter.convert()

"""
## Save the quantized model
"""
quantized_filename_tflite = Path(os.path.join(quantized_dir, "quantized_ptq.tflite"))
print("[INFO] Saving TFLite quantized model at {}".format(quantized_filename_tflite))
#with open(quantized_filename_tflite, 'wb') as f:
#  f.write(tflite_model)
quantized_filename_tflite.write_bytes(tflite_model)

Any error was thrown.

However, when compiling the code for the TPU, the compiler fails.

$ edgetpu_compiler quantized_ptq.tflite

Error:

Edge TPU Compiler version 16.0.384591198
Started a compilation timeout timer of 180 seconds.
ERROR: Attempting to use a delegate that only supports static-sized tensors with a graph that has dynamic-sized tensors.
Compilation failed: Model failed in Tflite interpreter. Please ensure model can be loaded/run in Tflite interpreter.
Compilation child process completed within timeout period.
Compilation failed!

Attached, I share the float and quantised models: https://filesender.up.pt/?s=download&token=70e50377-6b87-4334-bf5b-4e043e7eb79f

Click to expand!

### Issue Type Bug ### Operating System Ubuntu ### Coral Device _No response_ ### Other Devices _No response_ ### Programming Language Python 3.8 ### Relevant Log Output ```shell Edge TPU Compiler version 16.0.384591198 Started a compilation timeout timer of 180 seconds. ERROR: Attempting to use a delegate that only supports static-sized tensors with a graph that has dynamic-sized tensors. Compilation failed: Model failed in Tflite interpreter. Please ensure model can be loaded/run in Tflite interpreter. Compilation child process completed within timeout period. Compilation failed! ```

[hjonnala]

Hello @samagalhaes the model has dynamic signatures shape('shape_signature': array([ -1, 384, 384, 3]). It should be ('shape_signature': array([ 1, 384, 384, 3]). Please make signatures static and convert to int8 tflite model to be able to compile with edgetpu compiler. Thanks!

[{'name': 'serving_default_image:0', 'index': 0, 'shape': array([ 1, 384, 384, 3], dtype=int32), 'shape_signature': array([ -1, 384, 384, 3], dtype=int32), 'dtype': <class 'numpy.uint8'>, 'quantization': (1.0774157047271729, 115), 'quantization_parameters': {'scales': array([1.0774157], dtype=float32), 'zero_points': array([115], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}}]

[samagalhaes]

Hi, @hjonnala Thank you for the help. Indeed.

I am trying to make the input static, but I do not know how to do it. May you aid me?

Thanks!

[hjonnala]

I am not sure of custom objects section. But with the below code you would be able to convert and compile the model with the compiler.

import tensorflow as tf
import numpy as np

model = tf.keras.models.load_model('float.h5')
print(model.input.shape)
model.input.set_shape((1,) + model.input.shape[1:])
print(model.input.shape)

def representative_dataset():
 for _ in range(100):
  data = np.random.rand(1, 384, 384, 3)
  yield [data.astype(np.float32)]

"""
## Quantization
"""

converter = tf.lite.TFLiteConverter.from_keras_model(model)

# This enables quantization
converter.optimizations = [tf.lite.Optimize.DEFAULT]
# This sets the representative dataset for quantization
converter.representative_dataset = representative_dataset
# This ensures that if any ops can't be quantized, the converter throws an error
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
# For full integer quantization, though supported types defaults to int8 only, we explicitly declare it for clarity.
converter.target_spec.supported_types = [tf.int8]
# These set the input and output tensors to uint8 (added in r2.3)
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8

tflite_model = converter.convert()

"""
## Save the quantized model
"""

with open('quant_model.tflite', 'wb') as f:
 f.write(tflite_model)

[samagalhaes]

Hi @hjonnala ,

Thank you for the support. Indeed solved the issue. Cheers!

[google-coral-bot[bot]]

Are you satisfied with the resolution of your issue? Yes No