Use TensorFlow Lite quantize RetinaFace get wrong result

[Lemiron24]

1. System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04):Linux Ubuntu 18.04
• TensorFlow installation (pip package or built from source):docker pull
• TensorFlow library (version, if pip package or github SHA, if built from source):tensorflow:2.4.1-gpu

2. Code

this is my script which intend to convert float32 to int8 type:

import tensorflow as tf import numpy as np import cv2 import os

from nets.mobilenet025 import relu6 from nets.layers import UpsampleLike from nets.retinaface_training import box_smooth_l1, conf_loss

save_folder = "./model_data/quantized" model_path = "./model_data/RetinaFace.h5"

def representative_dataset_gen(): """

generate representative image dataset

:return:
"""
input_size = [640, 640]
dataset = "./imagelist.txt"
filename = open(dataset).read().split()
for i in range(len(filename)):
    if os.path.exists(filename[i]):
        print ("filename: ", i, filename[i])
        orig_image = cv2.imread(filename[i])
        rgb_image = cv2.cvtColor(orig_image, cv2.COLOR_BGR2RGB)
        image_tensor = cv2.resize(rgb_image, dsize=tuple(input_size))
        image_tensor = np.asarray(image_tensor / 255.0, dtype=np.float32)
        image_tensor = image_tensor[np.newaxis, :]
        yield [image_tensor]

if name == "main": model = tf.keras.models.load_model(model_path, custom_objects={'relu6': relu6, 'UpsampleLike': UpsampleLike, '_smooth_l1': box_smooth_l1, '_conf_loss': conf_loss})

export_dir = "./model_data/saved_model"
tf.saved_model.save(model, export_dir)
model = tf.saved_model.load(export_dir)
concrete_func = model.signatures[tf.saved_model.DEFAULT_SERVING_SIGNATURE_DEF_KEY]
concrete_func.inputs[0].set_shape([1, 640, 640, 3])
converter = tf.lite.TFLiteConverter.from_concrete_functions([concrete_func])

converter.experimental_new_converter = True
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_dataset_gen
# Ensure that if any ops can't be quantized, the converter throws an error
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8, tf.lite.OpsSet.TFLITE_BUILTINS]
# Set the input and output tensors to int8 (APIs added in r2.3)
converter.inference_input_type = tf.int8
converter.inference_output_type = tf.int8

tflite_model_quant = converter.convert()

interpreter = tf.lite.Interpreter(model_content=tflite_model_quant)
input_type = interpreter.get_input_details()[0]['dtype']
print('input: ', input_type)
output_type = interpreter.get_output_details()[0]['dtype']
print('output: ', output_type)
# Save the quantized model:
tflite_model_quant_file = os.path.join(save_folder, "RetinaFace_quant.tflite")
open(tflite_model_quant_file, "wb").write(tflite_model_quant)

3. Failure after conversion

I followed the tflite spec write the above script, the conversion is successful, but the Model produces wrong results, the detected face have same conf score, the bbox location and landmask both is wrong..

[Lemiron24]

this is the origin tensorflow network link: https://github.com/bubbliiiing/retinaface-tf2

can someone give some helps, very appricated.