search

ARM-software / armnn

Arm NN ML Software. The code here is a read-only mirror of https://review.mlplatform.org/admin/repos/ml/armnn
https://developer.arm.com/products/processors/machine-learning/arm-nn
MIT License
1.16k stars 308 forks source link

Issue with TfLiteMnasNet-Armnn #165

Closed georgeokelly closed 4 years ago

georgeokelly commented 5 years ago

I try to test the TfLiteMnasNet-Armnn with MnasNet_1.3_224 but get a weird output. Could it be possible that the confidence is bigger than 100% ?

./armnn-test/TfLiteMnasNet-Armnn -d testdata/ -m tflite_model/ -c CpuAcc
ArmNN v20190200

= Prediction values for test #0
Top(1) prediction is 436 with confidence: 864.037%
Top(2) prediction is 282 with confidence: 668.454%
Top(3) prediction is 107 with confidence: 272.195%
Top(4) prediction is 18 with confidence: 225.788%
Top(5) prediction is 2 with confidence: 181.305%
Prediction for test case 0 (436) is incorrect (should be 209)
= Prediction values for test #1
Top(1) prediction is 436 with confidence: 864.037%
Top(2) prediction is 282 with confidence: 668.454%
Top(3) prediction is 107 with confidence: 272.195%
Top(4) prediction is 18 with confidence: 225.788%
Top(5) prediction is 2 with confidence: 181.305%
Prediction for test case 1 (436) is incorrect (should be 283)
= Prediction values for test #2
Top(1) prediction is 3 with confidence: 994.089%
Top(2) prediction is 2 with confidence: 84.3158%
Top(3) prediction is 1 with confidence: 55.1689%
Top(4) prediction is 0 with confidence: 7.75329%
Total time for 3 test cases: 0.512 seconds
Average time per test case: 170.618 ms
One or more test cases failed
Runtime::UnloadNetwork(): Unloaded network with ID: 0
file /data/local/tmp/testdata/*
./Cat.jpg:                       JPEG image data
./Dog.jpg:                       JPEG image data
./shark.jpg:                     JPEG image data
georgeokelly commented 5 years ago

It also happens with TfLiteMobiletV2Quantized-Armnn.

./armnn-test/TfLiteMobiletV2Quantized-Armnn -d testdata/ -m tflite_model/ -c CpuAcc
ArmNN v20190200

= Prediction values for test #0
Top(1) prediction is 163 with confidence: 1602.06%
Top(2) prediction is 160 with confidence: 484.573%
Top(3) prediction is 157 with confidence: 336.235%
Top(4) prediction is 127 with confidence: 286.788%
Top(5) prediction is 114 with confidence: 227.453%
Prediction for test case 0 (163) is incorrect (should be 209)
= Prediction values for test #1
Top(1) prediction is 877 with confidence: 1137.26%
Top(2) prediction is 436 with confidence: 1127.37%
Top(3) prediction is 432 with confidence: 682.359%
Top(4) prediction is 286 with confidence: 662.58%
Top(5) prediction is 282 with confidence: 652.691%
Prediction for test case 1 (877) is incorrect (should be 283)
= Prediction values for test #2
Top(1) prediction is 3 with confidence: 1196.6%
Top(2) prediction is 1 with confidence: 276.899%
Top(3) prediction is 0 with confidence: -39.557%
Total time for 3 test cases: 0.348 seconds
Average time per test case: 115.927 ms
One or more test cases failed
Runtime::UnloadNetwork(): Unloaded network with ID: 0
MatthewARM commented 5 years ago

Very strange. Are you using Mobilenet_V2_1.0_224_quant from https://www.tensorflow.org/lite/guide/hosted_models ?

georgeokelly commented 5 years ago

Hi @MatthewARM I use the model from Hosted models on GitHub. Anyway, I also tried with that one on tensorflow.org and got the same results.

MatthewARM commented 5 years ago

Pass me the URLs for your shark, dog and cat and I'll try to see what results I get :-)

MatthewARM commented 5 years ago

Also, do you get the same results with CpuRef?

ggardet commented 4 years ago

I am having similar problems (wrong recognition) on ThunderX2 99xx machine (CpuRef and CpuAcc). Images used:

TfLiteInceptionV3Quantized-Armnn returns:

ArmNN v20190800
= Prediction values for test #0
Top(1) prediction is 274 with value: 7.97737
Top(2) prediction is 228 with value: 7.2936
Top(3) prediction is 226 with value: 5.69812
Top(4) prediction is 236 with value: 4.71045
Top(5) prediction is 169 with value: 4.10265
Prediction for test case 0 (274) is incorrect (should be 209)
= Prediction values for test #1
Top(1) prediction is 282 with value: 11.0923
Top(2) prediction is 283 with value: 8.8131
Top(3) prediction is 286 with value: 7.06567
Top(4) prediction is 288 with value: 3.11497
Top(5) prediction is 754 with value: 2.88705
Prediction for test case 1 (282) is incorrect (should be 283)
= Prediction values for test #2
Top(1) prediction is 3 with value: 11.7001
Top(2) prediction is 4 with value: 4.63447
Top(3) prediction is 5 with value: 2.7351
Top(4) prediction is 149 with value: 2.58315
Top(5) prediction is 95 with value: 2.27925
Total time for 3 test cases: 0.449 seconds
Average time per test case: 149.543 ms
One or more test cases failed
Runtime::UnloadNetwork(): Unloaded network with ID: 0

Same errors with TfLiteMnasNet-Armnn TfLiteMobilenetQuantized-Armnn TfLiteMobilenetV2Quantized-Armnn.

[EDIT]: Models used: http://download.tensorflow.org/models/tflite/mnasnet_1.3_224_09_07_2018.tgz http://download.tensorflow.org/models/tflite_11_05_08/inception_v3_quant.tgz http://download.tensorflow.org/models/mobilenet_v1_2018_08_02/mobilenet_v1_1.0_224_quant.tgz http://download.tensorflow.org/models/tflite_11_05_08/mobilenet_v2_1.0_224_quant.tgz

MatthewARM commented 4 years ago

Hi @ggardet looks like that's fine: 274 = dingo / 228 = kelpie 282 = tabby cat 3 = great white shark

And the output activations show a reasonable range. Looks good to me.

ggardet commented 4 years ago

Hi @ggardet looks like that's fine: 274 = dingo / 228 = kelpie 282 = tabby cat 3 = great white shark

Ok, One or more test cases failed is a bit misleading then. Maybe we should check for multiple labels? Btw, where could I get the labels?

And the output activations show a reasonable range. Looks good to me.

Ok, thanks!

MatthewARM commented 4 years ago

The labels.txt should be available wherever you got the models.

Those tests need redesigning as they are a holdover from when Arm NN was developed in-house at Arm. They're really regression tests for 3 specific images that we have lying around internally, but can't upstream due to copyright. I think the right thing to do would be to provide a script that pulls them from the internet, and does any cropping needed for the test. I'll raise this internally.

MatthewARM commented 4 years ago

Urgh that's strange, I can't find the labels.txt on the tensorflow website. That's odd considering the models are pretty much useless without them. There's a copy here: https://github.com/leferrad/tensorflow-mobilenet/blob/master/imagenet/labels.txt

ggardet commented 4 years ago

Urgh that's strange, I can't find the labels.txt on the tensorflow website. That's odd considering the models are pretty much useless without them. There's a copy here: https://github.com/leferrad/tensorflow-mobilenet/blob/master/imagenet/labels.txt

I also found them in a MobileNet V1 archive: https://storage.googleapis.com/download.tensorflow.org/models/tflite/mobilenet_v1_1.0_224_quant_and_labels.zip

Colm-in-Arm commented 4 years ago

Hi @georgeokelly and @ggardet.

I presume you're both happy with the resolution of this issue? If I hear nothing back I'll close it in a week or so.

Colm.

ggardet commented 4 years ago

Hi @georgeokelly and @ggardet.

I presume you're both happy with the resolution of this issue? If I hear nothing back I'll close it in a week or so.

Colm.

Hmm, not really, we should have access to images known to work.

Colm-in-Arm commented 4 years ago

As a short term solution I searched for the specific images so we can at least refer to what appears to be the origin:

shark.jpg appears to be a stock image provided by Norbert Wu: http://www.norbertwu.com/nwp/landscape-subjects-for-calendars/australia_web/detail.np/detail-178.html

Dog.jpg also appears to be a stock image but I can't find an original source. Several sites refer to it being CCO public domain but I couldn't find it in there. Here's a link to just one site that had it: https://pixabay.com/photos/dog-labrador-light-brown-pet-1210559/

Finally Cat.jpg is from Wikimedia: https://upload.wikimedia.org/wikipedia/commons/thumb/4/4d/Cat_November_2010-1a.jpg/767px-Cat_November_2010-1a.jpg

Colm.

ggardet commented 4 years ago

Dog.jpg does not seem to work, at least with TfLiteMnasNet-Armnn. It matches label 160 instead of 209. Could you double check this is the file used in CI/CD, please?

Colm-in-Arm commented 4 years ago

It's a pity I can't find an attributable source for that image. Can you try this link instead please:

https://pixabay.com/static/uploads/photo/2016/02/19/15/46/dog-1210559_960_720.jpg

It is the correct image. It should be 136886 bytes in size.

Colm.

ggardet commented 4 years ago

With that one, it works again with the subset of tests I run. Thanks!