different results from cpu and edgetpu

[tino926]

I ran the sample and got different results from edgetpu model and cpu model. Is that normal?

This is what I got: (the scores of "tabby, tabby cat" are obviously different)

user@server2:~/workdir/edgetpu-minimal-example-dev/build$ sudo ../out/k8/minimal \
                         ../test_data/mobilenet_v1_1.0_224_quant.tflite \
                        ../test_data/resized_cat.bmp ../test_data/imagenet_labels.txt

------
Class: tabby, tabby cat
Score: 0.046875
------
Class: tiger cat
Score: 0.109375
------
Class: Siamese cat, Siamese
Score: 0.00390625
------
Class: Egyptian cat
Score: 0.804688
------
Class: doormat, welcome mat
Score: 0.0078125
------
Class: radiator
Score: 0.0078125
------
Class: space heater
Score: 0.0117188

user@server2:~/workdir/edgetpu-minimal-example-dev/build$ sudo ../out/k8/minimal \
                         ../test_data/mobilenet_v1_1.0_224_quant_edgetpu.tflite \
                         ../test_data/resized_cat.bmp ../test_data/imagenet_labels.txt

------
Class: tabby, tabby cat
Score: 0.0390625
------
Class: tiger cat
Score: 0.128906
------
Class: Siamese cat, Siamese
Score: 0.00390625
------
Class: Egyptian cat
Score: 0.792969
------
Class: doormat, welcome mat
Score: 0.0078125
------
Class: radiator
Score: 0.0078125
------
Class: space heater
Score: 0.0117188

[manoj7410]

@tino926 I think it is not fair to expect exact same numbers from both the models. There can be a slight difference.

[hjonnala]

@tino926 for some models there might be slight difference between cpu and edgeptu results. But for this model I am getting the same results. Have you used this TENSORFLOW_COMMIT = "a4dfb8d1a71385bd6d122e4f27f86dcebb96712d" to build the code.

python3 examples/classify_image.py   --model ../../test_data/mobilenet_v1_1.0_224_quant_edgetpu.tflite    --labels ../../test_data/imagenet_labels.txt   --input ../../test_data/resized_cat.bmp --top_k 5
----INFERENCE TIME----
Note: The first inference on Edge TPU is slow because it includes loading the model into Edge TPU memory.
12.4ms
2.7ms
2.7ms
2.5ms
2.5ms
-------RESULTS--------
Egyptian cat: 0.82031
tiger cat: 0.11328
tabby, tabby cat: 0.03125
space heater: 0.01172
radiator: 0.00781

python3 examples/classify_image.py   --model ../../test_data/mobilenet_v1_1.0_224_quant.tflite    --labels ../../test_data/imagenet_labels.txt   --input ../../test_data/resized_cat.bmp --top_k 5
----INFERENCE TIME----
Note: The first inference on Edge TPU is slow because it includes loading the model into Edge TPU memory.
60.7ms
32.7ms
34.2ms
34.4ms
33.1ms
-------RESULTS--------
Egyptian cat: 0.82031
tiger cat: 0.11328
tabby, tabby cat: 0.03125
space heater: 0.01172
radiator: 0.00781

[tino926]

@hjonnala , thanks for your reply. I found that I post the question at the wrong place, I was going to post at https://github.com/Namburger/edgetpu-minimal-example . However, still wish you can kindly help me about one more question:

By following Namburger's steps, the libedgetpu.so.1 is pre-built and included in download files. I thought I don't need to change the "TENSORFLOW_COMMIT" since it should match the pre-built libedgetpu.so.1. If I need to change that, how do I know the exact commit I should use?

[hjonnala]

As the repo that you are working has not been updated from 2 years, you would be working with old edgetpu runtime. Please find the latest ones here: https://github.com/google-coral/pycoral/tree/master/libedgetpu_bin

To install it form packages, please refer to software page: https://coral.ai/software/#debian-packages

If you want build the edgetpu runtime for yourself, please check this link: https://coral.ai/docs/notes/build-coral/#build-the-edge-tpu-runtime-libedgetpu

[tino926]

@hjonnala

I want to duplicate your result at (https://github.com/google-coral/edgetpu/issues/503#issuecomment-964479232), however, I still can't get the same result as you got.

This is how I get the library and sample codes (follow https://coral.ai/docs/accelerator/get-started/):

apt-get install apt-utils -y

# 1. Install the Edge TPU runtime
echo "deb https://packages.cloud.google.com/apt coral-edgetpu-stable main" | sudo tee /etc/apt/sources.list.d/coral-edgetpu.list

curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

sudo apt-get update

sudo apt-get install libedgetpu1-std

apt-get install python3-pycoral -y

# 2. Install the PyCoral library
mkdir coral && cd coral

git clone https://github.com/google-coral/pycoral.git

cd pycoral

bash examples/install_requirements.sh classify_image.py

And these are what I got:

python3 examples/classify_image.py --model ../../../edgetpu_minimal/edgetpu-minimal-example/test_data/mobilenet_v1_1.0_224_quant.tflite --labels ../../../edgetpu_minimal/edgetpu-minimal-example/test_data/imagenet_labels.txt --input ../../../edgetpu_minimal/edgetpu-minimal-example/test_data/resized_cat.bmp --top_k 5
----INFERENCE TIME----
Note: The first inference on Edge TPU is slow because it includes loading the model into Edge TPU memory.
42.3ms
37.4ms
44.3ms
38.1ms
37.7ms
-------RESULTS--------
Egyptian cat: 0.80469
tiger cat: 0.10938
tabby, tabby cat: 0.04688
space heater: 0.01172
radiator: 0.00781

python3 examples/classify_image.py --model ../../../edgetpu_minimal/edgetpu-minimal-example/test_data/mobilenet_v1_1.0_224_quant_edgetpu.tflite --labels ../../../edgetpu_minimal/edgetpu-minimal-example/test_data/imagenet_labels.txt --input ../../../edgetpu_minimal/edgetpu-minimal-example/test_data/resized_cat.bmp --top_k 5
----INFERENCE TIME----
Note: The first inference on Edge TPU is slow because it includes loading the model into Edge TPU memory.
13.1ms
3.6ms
3.6ms
3.6ms
3.7ms
-------RESULTS--------
Egyptian cat: 0.79297
tiger cat: 0.12891
tabby, tabby cat: 0.03906
space heater: 0.01172
radiator: 0.00781

[hjonnala]

can you try with models from this test data

[hjonnala]

Please try with updated models: https://github.com/google-coral/test_data/blob/master/mobilenet_v1_1.0_224_quant.tflite https://github.com/google-coral/test_data/blob/master/mobilenet_v1_1.0_224_quant_edgetpu.tflite and let us if you have any questions

Thanks

[google-coral-bot[bot]]

Are you satisfied with the resolution of your issue? Yes No

[tino926]

@hjonnala Thanks for your replying, I got the same result with the models you provided.