search

google-coral / edgetpu

Coral issue tracker (and legacy Edge TPU API source)
https://coral.ai
Apache License 2.0
426 stars 125 forks source link

The results produced by edgeTPU are completely different from int8 tflite #685

Closed Dnlee17 closed 1 year ago

Dnlee17 commented 1 year ago

Description

I successfully converted the int8 tflite (input:uint8,output:float32) to edgetpu tflite, but when using the same image as input, the output of the two models is inconsistent, Not sure which part went wrong that causes the output to be so different, or is there a value loss when int8 tflite compiles to edgetpu tflite?what should I do to minimize the value loss?

1.CMD edgeTPUcompile:

Edge TPU Compiler version 16.0.384591198 Started a compilation timeout timer of 180 seconds.

Model compiled successfully in 893 ms.

Input model: uint8_11.tflite Input size: 1.08MiB Output model: uint8_11_edgetpu.tflite Output size: 1.41MiB On-chip memory used for caching model parameters: 1.06MiB On-chip memory remaining for caching model parameters: 6.55MiB Off-chip memory used for streaming uncached model parameters: 115.00KiB Number of Edge TPU subgraphs: 1 Total number of operations: 101 Operation log: uint8_11_edgetpu.log

Model successfully compiled but not all operations are supported by the Edge TPU. A percentage of the model will instead run on the CPU, which is slower. If possible, consider updating your model to use only operations supported by the Edge TPU. For details, visit g.co/coral/model-reqs. Number of operations that will run on Edge TPU: 100 Number of operations that will run on CPU: 1 See the operation log file for individual operation details. Compilation child process completed within timeout period. Compilation succeeded!

2.log file

uint8_11_edgetpu.log: Edge TPU Compiler version 16.0.384591198 Input: uint8_11.tflite Output: uint8_11_edgetpu.tflite

Operator Count Status

DEPTHWISE_CONV_2D 17 Mapped to Edge TPU QUANTIZE 1 Mapped to Edge TPU CONV_2D 34 Mapped to Edge TPU DEQUANTIZE 1 Operation is working on an unsupported data type FULLY_CONNECTED 1 Mapped to Edge TPU ADD 12 Mapped to Edge TPU PRELU 34 Mapped to Edge TPU RESHAPE 1 Mapped to Edge TPU

int8 tflite process

import tensorflow as tf (tensorflow-gpu version:2.10.0) import cv2

input_tensor = cv2.imread(img_path) # RGB order png

interpreter = tf.lite.Interpreter("uint8_11.tflite") interpreter.allocate_tensors() interpreter.set_tensor(interpreter.get_input_details()[0]['index'], input_tensor) interpreter.invoke() output_data = interpreter.get_tensor(interpreter.get_output_details()[0]['index'])

score = tf.squeeze(output_data).numpy()

edgetpu tflite process

import tflite_runtime.interpreter as tflite import cv2 import platform

EDGETPU_SHARED_LIB = { 'Linux': 'libedgetpu.so.1', 'Darwin': 'libedgetpu.1.dylib', 'Windows': 'edgetpu.dll' }[platform.system()]

def make_interpreter(model_file): model_file, *device = model_file.split('@') return tflite.Interpreter( model_path=model_file, experimental_delegates=[ tflite.load_delegate(EDGETPU_SHARED_LIB, {'device': device[0]} if device else {}) ])

interpreter = make_interpreter("uint8_11_edgetpu.tflite") interpreter.allocate_tensors()

input_tensor = cv2.imread(img_path) # RGB order png

interpreter.set_tensor(interpreter.get_input_details()[0]['index'], input_tensor) interpreter.invoke()

score = interpreter.get_tensor(interpreter.get_output_details()[0]['index'])

3.different output

testing two models with same image as input,but output different results. (file name: int8 tflite score, edgetpu tflite score) 090.png:11.257883, 15.173668 091.png:8.810517, -10.76841 092.png:25.942078, 26.921024 093.png:-0.9789463, 12.726302 094.png:11.747356, 27.410498 095.png:22.026293, 16.642088 096.png:-5.873678, 7.8315706 097.png:23.494713, 23.00524 098.png:17.131561, 34.263123 099.png:18.110508, 25.942078

Click to expand! ### Issue Type Performance ### Operating System Ubuntu ### Coral Device Dev Board ### Other Devices _No response_ ### Programming Language Python 3.9 ### Relevant Log Output _No response_
hjonnala commented 1 year ago

can you run the int8 tflite model with edgetpu tflite process code and share the results.

Dnlee17 commented 1 year ago

Issue

I retried compile with a new model, both run on coral board with same code,but the result is still different.

run on coral board

with int8 tflite model

edgeTPU$ python3 val_list.py --model models/uint8_11.tflite --input images/TPU

099.png , [[-14.896992]] 098.png , [[-11.704779]] 097.png , [[-10.108673]] 096.png , [[-5.320354]] 095.png , [[-13.832921]] 094.png , [[-0.5320354]] 093.png , [[-1.5961063]] 092.png , [[4.7883186]] 091.png , [[-6.384425]] 090.png , [[4.7883186]]

with edgetpu tflite model

edgeTPU$ python3 val_list.py --model models/uint8_11_edgetpu.tflite --input images/TPU

099.png , [[-4.7883186]] 098.png , [[5.8523893]] 097.png , [[22.877523]] 096.png , [[-16.493097]] 095.png , [[-22.345488]] 094.png , [[14.896992]] 093.png , [[17.557169]] 092.png , [[13.832921]] 091.png , [[-0.5320354]] 090.png , [[6.384425]]

discover new issue

the result of int8 tflite running on coral dev board and ubuntu is different.

int8 tflite model run on Ubuntu

--model models/uint8_11.tflite --input images/TPU

099.png , -16.493097 098.png , -12.76885 097.png , -15.429027 096.png , -9.576637 095.png , -15.961062 094.png , -1.0640708 093.png , -6.9164605 092.png , -7.448496 091.png , -20.217346 090.png , 2.1281416

hjonnala commented 1 year ago

can you please share the tensor flow saved model here..

hjonnala commented 1 year ago

Is there any reason for your model has 6 channels and you are using only 3 channel input(RGB) to the model?

Dnlee17 commented 1 year ago

int8 tflite model uint8_11.zip

The original input is RGB+HSV, in order to simplify the issue, so I only put 3 channels as input in the description. Only image pre-processing is added to the code.

RGB_img = cv2.imread(img_path) # RGB order image HSV_img = cv2.cvtColor(RGB_img, cv2.COLORRGB2HSV) input = np.concatenate([RGB_img,HSV_img], -1) # same result as tf.concat([RGB_img,HSV_img], -1) input_tensor = np.expanddims(input, 0) # same result as tf.expanddims(input, 0)

hjonnala commented 1 year ago

can you lease sample image files also for testing..Thanks!

Dnlee17 commented 1 year ago

GIT.zip

images and int8 tflite model predict score (run on ubuntu) file in zip,Thanks!

hjonnala commented 1 year ago 
import tflite_runtime.interpreter as tflite
from pycoral.adapters import common
import cv2
import platform
import numpy as np
from PIL import Image
import glob

path = r'/home/mendel/GIT/*.bmp'
images = glob.glob(path)
models = ['/home/mendel/uint8_11.tflite', '/home/mendel/uint8_11_edgetpu.tflite']
EDGETPU_SHARED_LIB = {
'Linux': 'libedgetpu.so.1',
'Darwin': 'libedgetpu.1.dylib',
'Windows': 'edgetpu.dll'
}[platform.system()]

def make_interpreter(model_file):
    model_file, *device = model_file.split('@')
    return tflite.Interpreter(
    model_path=model_file,
    experimental_delegates=[
    tflite.load_delegate(EDGETPU_SHARED_LIB,
    {'device': device[0]} if device else {})
    ])

for img_path in images:
    for model in models:
        interpreter = make_interpreter(model)
        interpreter.allocate_tensors()

        RGB_img = cv2.imread(img_path) # RGB order image
        HSV_img = cv2.cvtColor(RGB_img, cv2.COLOR_RGB2HSV)
        input_ = np.concatenate([RGB_img,HSV_img], -1) # same result as tf.concat([RGB_img,HSV_img], -1)
        input_tensor = np.expand_dims(input_, 0) # same result as tf.expand_dims(input_, 0)
        # print(input_tensor)
        interpreter.set_tensor(interpreter.get_input_details()[0]['index'], input_tensor)
        interpreter.invoke()

        score = interpreter.get_tensor(interpreter.get_output_details()[0]['index'])
        print(img_path.split('/')[-1],  score[0][0], model.split('/')[-1])

The results seems to be similar for most of the images, except for some images.

Are there any expected results for these images to check the accuracy? Wondering have you compared the results with tensorflow vs tflite model?

000365.bmp -59.05593 uint8_11.tflite
000365.bmp -59.05593 uint8_11_edgetpu.tflite
000036.bmp -56.395752 uint8_11.tflite
000036.bmp -59.587967 uint8_11_edgetpu.tflite
000042.bmp -56.927788 uint8_11.tflite
000042.bmp -53.20354 uint8_11_edgetpu.tflite
000049.bmp 19.153275 uint8_11.tflite
000049.bmp 36.17841 uint8_11_edgetpu.tflite
000346.bmp -56.927788 uint8_11.tflite
000346.bmp -55.331684 uint8_11_edgetpu.tflite
000032.bmp -51.607433 uint8_11.tflite
000032.bmp -53.20354 uint8_11_edgetpu.tflite
000056.bmp -54.79965 uint8_11.tflite
000056.bmp -53.735577 uint8_11_edgetpu.tflite
000057.bmp -59.587967 uint8_11.tflite
000057.bmp -61.716106 uint8_11_edgetpu.tflite
000058.bmp -54.267612 uint8_11.tflite
000058.bmp -54.79965 uint8_11_edgetpu.tflite
000379.bmp -41.49876 uint8_11.tflite
000379.bmp -31.922125 uint8_11_edgetpu.tflite
000383.bmp 11.172744 uint8_11.tflite
000383.bmp 33.51823 uint8_11_edgetpu.tflite
000394.bmp -50.543365 uint8_11.tflite
000394.bmp -37.774513 uint8_11_edgetpu.tflite
000387.bmp -60.65204 uint8_11.tflite
000387.bmp -61.18407 uint8_11_edgetpu.tflite
000064.bmp -46.28708 uint8_11.tflite
000064.bmp -44.690975 uint8_11_edgetpu.tflite
000369.bmp -60.120003 uint8_11.tflite
000369.bmp -60.120003 uint8_11_edgetpu.tflite
000062.bmp 46.819115 uint8_11.tflite
000062.bmp 48.947258 uint8_11_edgetpu.tflite
000352.bmp -61.18407 uint8_11.tflite
000352.bmp -60.65204 uint8_11_edgetpu.tflite
000385.bmp -48.415222 uint8_11.tflite
000385.bmp -52.671505 uint8_11_edgetpu.tflite
000074.bmp -53.20354 uint8_11.tflite
000074.bmp -54.267612 uint8_11_edgetpu.tflite
000344.bmp -55.86372 uint8_11.tflite
000344.bmp -56.927788 uint8_11_edgetpu.tflite

the result of int8 tflite running on coral dev board and ubuntu is different.

to debug this please compare the input tensor values on ubuntu vs dev board.

Dnlee17 commented 1 year ago

1.

It seems that the value is easily affected when the value is in the range of -40~40.

and most of my target images predictions are in this range,

so I was wondering what causes the value loss.

2.

Is the value loss problem definitely happen when int8 tflite converting to edgetpu?

(the edgetpu_costom_op or the dequantize part went wrong ? or something else like int8 tflite output is tf.float32 not uint8?)

update

Change int8 tflite output type to int8, value loss issue still exists. 099.bmp, -34, uint8_11_d.tflite 099.bmp, -21, uint8_11_d_edgetpu.tflite 098.bmp, 3, uint8_11_d.tflite 098.bmp, 10, uint8_11_d_edgetpu.tflite 097.bmp, 0, uint8_11_d.tflite 097.bmp, 2, uint8_11_d_edgetpu.tflite 096.bmp, -34, uint8_11_d.tflite 096.bmp, -39, uint8_11_d_edgetpu.tflite 095.bmp, -6, uint8_11_d.tflite 095.bmp, -16, uint8_11_d_edgetpu.tflite 094.bmp, 38, uint8_11_d.tflite 094.bmp, 31, uint8_11_d_edgetpu.tflite 093.bmp, -11, uint8_11_d.tflite 093.bmp, 16, uint8_11_d_edgetpu.tflite 092.bmp, -6, uint8_11_d.tflite 092.bmp, -2, uint8_11_d_edgetpu.tflite 091.bmp, 4, uint8_11_d.tflite 091.bmp, 5, uint8_11_d_edgetpu.tflite 090.bmp, 5, uint8_11_d.tflite 090.bmp, 9, uint8_11_d_edgetpu.tflite

input tensor values on ubuntu vs dev board is exactly the same, and same as code.

the results with tensorflow vs tflite model is similar.

hjonnala commented 1 year ago

There could be slight variations for tflite to edgetpu results. But the results posted are not expected. To debug the operations causing the difference

  1. model needs to be compiled with -i flag.
  2. And the output of edgetpu-custom-op needs to be compared with respective tflite operation output(by creating the interpreter with experimental_preserve_all_tensors=True). https://github.com/google-coral/edgetpu/issues/607#issuecomment-1149029064

If possible can you try your model with relu or relu6 instead of PRelu. Thanks!

google-coral-bot[bot] commented 1 year ago

Are you satisfied with the resolution of your issue? Yes No