A PTQ tflite model fails to pass benchmark test

[liamsun2019]

My use case: Apply post training quantization to a pth model and convert to tflite. The generated tflite model fails to pass benchmark test with following error message: STARTING! Log parameter values verbosely: [0] Graph: [out/ptq_model.tflite] Loaded model out/ptq_model.tflite ERROR: tensorflow/lite/kernels/concatenation.cc:179 t->params.scale != output->params.scale (3 != -657359264) ERROR: Node number 154 (CONCATENATION) failed to prepare. Failed to allocate tensors! Benchmarking failed.

Pls refer to the attachment. Thanks. test.zip

[liamsun2019]

My quantization strategy: quantizer = PostQuantizer(model, dummy_input, work_dir='out', config={'force_overwrite': True, 'rewrite_graph': True, 'is_input_quantized': None, 'asymmetric': False, 'per_tensor': False}) 。。。。。。。。。。。。。。。。。。。。。。。。。。 converter = TFLiteConverter(ptq_model, dummy_input, tflite_path='out/ptq_model.tflite', strict_symmetric_check=False, quantize_target_type='int8')

[liamsun2019]

The following strategy works: quantizer = PostQuantizer(model, dummy_input, work_dir='out', config={'force_overwrite': True, 'rewrite_graph': True, 'is_input_quantized': None, 'asymmetric': True, 'per_tensor': True}) 。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。 converter = TFLiteConverter(ptq_model, dummy_input, tflite_path='out/ptq_model.tflite', strict_symmetric_check=False, quantize_target_type='uint8')

[liamsun2019]

Looks int8 per-channel quantization may incur errors.

[peterjc123]

The following pattern in your model is the root cause of the problem.

A = sigmoid(X)
B = cat(A, Y)

The output tensor of the sigmoid op has a constant quantization parameter. There are several ways to get it fixed.

1. Unify the quantization parameters of (Y, B) to the quantization parameters of A and also we need to disable the observers in those variables.
2. Insert requantization after A, so that we have

   A = sigmoid(X)
   A_ = requantize(A)
   B = cat(A_, Y)

   Then, we may just unify the quantization parameters of (A_, Y, B), just like what we do as usual.

[peterjc123]

Or you may just skip the quantization for this kind of pattern, which seems to be the simplest solution.

[peterjc123]

1. Unify the quantization parameters of (Y, B) to the quantization parameters of A and also we need to disable the observers in those variables.

This is simpler I guess. We will try to fix it this way.