search

ARM-software / armnn

Arm NN ML Software. The code here is a read-only mirror of https://review.mlplatform.org/admin/repos/ml/armnn
https://developer.arm.com/products/processors/machine-learning/arm-nn
MIT License
1.14k stars 307 forks source link

Noisy inference results with TF2 int8 tflite models #637

Closed Rahn80643 closed 5 months ago

Rahn80643 commented 2 years ago

Hi,

I'm migrating our segmentation model from TensorFlow1 to TensorFlow2, and the model is retrained with TensorFlow quantization aware training technique; however, the inference results of TF2 tflite model from ArmNN are not ideal, the comparisons and environment settings are shown below:

armnn2202_1

armnn2202_2

Based on the inference results from TensorFlow tflite API, the weights in tflite models seems normal, but the inference results from ArmNN contain many noisy pixels, I want to ask:

  1. Is this issue resulted from the version of TensorFlow or tflite?
  2. How to effectively improve the inference results in ArmNN?

Best Regards, Rahn

TeresaARM commented 2 years ago

Hi @Rahn80643,

thank you very much for your comment. which backend are you using when you run ArmNN, CpuAcc, GpuAcc or CpuRef?

Rahn80643 commented 2 years ago

Hi @TeresaARM,

I've tested with CpuAcc and GpuAcc backend, and the inference results are similar

TeresaARM commented 2 years ago

Hi @Rahn80643,

could you give it a go to CpuRef? and see if the results are closer to TensorFlow? The performance will be worse than CpuAcc and GpuAcc, as CpuRef is just a "reference", but this information will help us to find out where the issue is.

Thank you!

Rahn80643 commented 2 years ago

Hi @TeresaARM,

The followings are the inference results using CpuRef, and the results are not closer to those from TensorFlow

armnn2202_1

armnn2202_2

Thank you

Rahn80643 commented 2 years ago

Hi,

I compared two int8 tflite models trained in TensorFlow1 and TensorFlow2 respectively, and I found the input zero point values are different. Is this the main reason why inferring TensorFlow2 int8 tflite model generates noisy results? If so, how to include the input zero point into ArmNN inference code to solve this issue?

zero_point

Rahn

MatthewARM commented 2 years ago

yes, this could be the problem. In your application code, when you convert the input image into a tensor for Arm NN, are you taking the zero-point into account?

MatthewARM commented 2 years ago

If this is the problem, there is some good explanation of it here: https://github.com/ARM-software/armnn/issues/161

Rahn80643 commented 2 years ago

Hi,

I've tried the following modifications but the inference results still contain noisy results:

  1. Based on thread #161, the author had tried not to do preprocess the image and divide the output by 255. I tried the same process in ArmNN; however, since the values of output results are within the range of [0, 255], if the values are divided by 255, the results will be all 0s.

  2. the input images are preprocessed based on the following formula in ArmNN, new_value = ((old_value/scale_factor) - mean)/ stddev. Therefore, I've tried changing mean values, but the inference results still contain noisy results:

different mean values

Apart from modifying the mean values in preprocess stage, are there other approaches to take the zero-point into account in ArmNN settings?

morgolock commented 5 months ago

I'm closing this due to inactivity, if you require support please reopen.