Closed freedomtan closed 3 years ago
@renjie-liu could you take a look at this? This is a good example case where the quantization + NNAPI is involved.
Hi Freedom, do you have a tflite model we can debug internally? thanks!
@renjie-liu you can find the PTQ resnet50 tflite generated by the aforementioned PTQ script at https://drive.google.com/file/d/1af1ucoBg4zQ0KJdcyEfVSWyqcXYJdOym/view?usp=sharing
Hi Freedom, https://github.com/tensorflow/tensorflow/commit/8c90a182b8e200a870ddc5dc4fb9d7ee10f04cbe
Can you try bazel run //tensorflow/compiler/mlir/lite/experimental/tac:tac-translate -- <INPUT_MODEL> -o=<OUTPUT_MODEL> --device-specs=NNAPI
And see if it works for you case?
Note we are replacing the mean with a avg_pool -> requantize, that may cause accuracy issue so it would be great to get your help to validate the model.
Thanks!
@renjie-liu Thanks. So far, it works for all the models I tested functionally. I'll check accuracy later.
FYI. Yes, there is accuracy loss, but seems to be acceptable for image classification. For this resnet case, verifying with TFLite image classification evaluation tool. The one with MEAN:
Top-1 Accuracy: 0.76332
Top-2 Accuracy: 0.86308
Top-3 Accuracy: 0.8974
Top-4 Accuracy: 0.91452
Top-5 Accuracy: 0.92644
Top-6 Accuracy: 0.93486
Top-7 Accuracy: 0.9403
Top-8 Accuracy: 0.94442
Top-9 Accuracy: 0.94764
Top-10 Accuracy: 0.95084
For MEAN replaced with avg_pool + requant
Top-1 Accuracy: 0.76286
Top-2 Accuracy: 0.8619
Top-3 Accuracy: 0.89688
Top-4 Accuracy: 0.91464
Top-5 Accuracy: 0.92534
Top-6 Accuracy: 0.9334
Top-7 Accuracy: 0.93922
Top-8 Accuracy: 0.94358
Top-9 Accuracy: 0.94674
Top-10 Accuracy: 0.94996
1. System information
2. Code
Post-training quantization (PTQ) may result in MEAN with different quantization scales for inputs and outputs. Because NNAPI only supports MEAN with same quantization parameters for inputs and output, this kind of mean ops could not be delegated to NNAPI via the NNAPI delegate.
E.g., if we do PTQ on the ResNet 50 from tfhub,
then
adb push /tmp/resnet50_quant.tflite /data/local/tmp/
, and runbenchmark_model
on device (here I ran it on a Pixel 4)It shows something like:
3. Possible solutions
a. force MEAN to have same scale for inputs and outputs like what I did at https://github.com/tensorflow/tensorflow/pull/51373
With this constraint applied, for the same ResNet 50 model, I got something like the following:
However, as discussed in https://github.com/tensorflow/tensorflow/pull/51373, this may run into problems
b. extending NNAPI MEAN op to support different quantization parameters
@miaowang14 is this possible?
c. if modifying original model is viable, as @renjie-liu suggested in https://github.com/tensorflow/tensorflow/pull/51373, replacing MEAN with reduce_sum or global avg_pool is mathematically equivalent.
Maybe this could an option in converter or quantizer?