Quantization Result in Float32 Instead of Int8 Using TFLite with TensorFlow 2.9

[Mary-prh]

Environment

• TensorFlow Version: 2.9
• Python Version: 3.9.18
• Operating System: [Specify your OS]

Issue Description

I am trying to deploy a TensorFlow Keras model using TensorFlow Lite for an edge device, aiming for model quantization to int8 to optimize performance. However, after the quantization process, the data types of the input and output tensors remain float32 instead of int8 as expected.

Despite setting the inference_input_type and inference_output_type to tf.int8, the conversion result still shows float32 for both input and output tensor types.

Expected Behavior I expected the input and output tensor data types to be int8 after the quantization process.

Actual Behavior The input and output tensor data types remain float32 after quantization.

Steps to Reproduce Set up a TensorFlow environment with version 2.9 and Python 3.9.18. Execute the provided code snippet with a suitable model and dataset. Observe the data types of the input and output tensors after quantization. Has anyone encountered a similar issue, or does anyone have suggestions on how to ensure the tensors are correctly quantized to int8? Any insights or experiences with this problem would be greatly appreciated.

I have followed the standard process for converting and quantizing the model using TensorFlow Lite's TFLiteConverter API. Below is a simplified version of the code I used for the conversion and quantization process:

from tensorflow.keras.models import load_model
import tensorflow as tf

# Set logger level
tf.get_logger().setLevel('INFO')

# Load the model
path = 'GN_W0.5_S2_ArcFace_epoch16.h5'
ghost_model = load_model(path)

# Custom function to read and preprocess images
def tf_imread(file_path):
    img = tf.io.read_file(file_path)
    img = tf.image.decode_image(img, channels=3, expand_animations=False)  # Decode to [0, 255]
    img = tf.cast(img, tf.float32)  # Cast to float32
    img = (img - 127.5) * 0.0078125  # Normalize
    return img

def create_representative_dataset_generator(dataset_path, total_images=10, batch_size=64):
    def generator():
        image_paths = []
        pathList = os.listdir(dataset_path)
        for class_id in pathList[:20]:
            class_path = os.path.join(dataset_path, class_id)
            image_paths += glob.glob(os.path.join(class_path, '*.jpg'))

        # Shuffle to randomize the selection of images
        random.shuffle(image_paths)

        # Limit the total number of images to the specified subset size
        selected_image_paths = image_paths[:total_images]

        for n, image_path in enumerate(selected_image_paths, start=1):
            print(f"Processing image {n}/{total_images}: {image_path}")
            yield image_path

    dataset = tf.data.Dataset.from_generator(generator, output_types=tf.string)
    dataset = dataset.map(lambda x: tf_imread(x), num_parallel_calls=tf.data.experimental.AUTOTUNE)
    dataset = dataset.batch(batch_size)
    return dataset
dataset_path = './GhostFaceNets/datasets/ms1m-retinaface-t1/imgs/'
representative_dataset = create_representative_dataset_generator(dataset_path)

# use the dataset in quantization process
def representative_data_gen():
    for input_value in representative_dataset:
        yield [input_value]

converter = tf.lite.TFLiteConverter.from_keras_model(ghost_model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8,
                                       tf.lite.OpsSet.TFLITE_BUILTINS, # enable TensorFlow Lite ops.
                                       tf.lite.OpsSet.SELECT_TF_OPS ]# enable TensorFlow ops.]
converter.inference_input_type = tf.int8  # or tf.uint8
converter.inference_output_type = tf.int8 
converter.representative_dataset = representative_data_gen
tflite_model_quant  = converter.convert()
interpreter = tf.lite.Interpreter(model_content=tflite_model_quant)
input_type = interpreter.get_input_details()[0]['dtype']
print('input: ', input_type)
output_type = interpreter.get_output_details()[0]['dtype']
print('output: ', output_type)
tensor_details = interpreter.get_tensor_details()
print(tensor_details)

[Mary-prh]

@HamadYA Your help is much appreciated

[Mary-prh]

the problem is solved for quantizing to int8 with the following change: cloned_model = tf.keras.models.clone_model( ghost_model, clone_function=lambda layer: layer.class.from_config( {**layer.get_config(), "dtype": "float32"} ), ) However, the problem persists with tf.lite.OpsSet.EXPERIMENTAL_TFLITE_BUILTINS_ACTIVATIONS_INT16_WEIGHTS_INT8 as TFLite does not support PReLU

[Mary-prh]

The previous solution I mentioned changes the weights remarkably and the prediction is no more valid

[HamadYA]

The previous solution I mentioned changes the weights remarkably and the prediction is no more valid

I am sorry, I cannot work on big issues right now. I am being busy with my PhD Research Dissertation. Sorry again.

[Anshulgada]

Hey @Mary-prh did you find any suitable approach? I too was trying to quantize the model mainly because the inference time for each is too much and I wanted to deploy it to my software where I'll be inferencing on multiple images and the collective time increases alot.

My aim too is to decrease the 32bit floating to int8. Please drop down a solution if you've found one.