[Bug] Model accuracy decreases dramatically using NCS2

[lorytony]

• OpenVINO =>2021.4.2
• Operating System / Platform => Windows10 64 Bit and Raspbian10
• Problem classification: Model accuracy decreases dramatically using NCS2
• Framework: TensorFlow 2.9 (if applicable)
• Model name: Vgg16 (if applicable)

Detailed description

Hi all, when I do OpenVINO inference of my network on CPU I get 86% accuracy. Instead, if I make the inference using the NCS2 it drops to 72%. For simplicity I have reported only the accuracy metric, but also the others (f1-score, precision, recall ) behave in the same way.

Steps to reproduce

Initially this phenomenon was observed on a raspberry4 with Raspbian10 and Openvino Runtime 2021.4.2 . To exclude that the problem was a bad conversion of my network in .xml and .bin formats, or a wrong configuration of OpenVINO, I chose to install Openvino also on my Windows Notebook. A first experiment consisted in comparing the results of the metrics (accuracy, f1-score ect) obtained from the model executed in keras (h5 format) with the model (xml bin format) executed on openvino 2021.4.2 (CPU) on the same dataset. The results obtained were all identical. 'accuracy':0.8709677419354839 ect.. both for the model run on keras and for the model run on openvino. But in changing the string "CPU" to "MYRIAD" of the following statement and calculating the related metrics on the same test set

compiled_model = core.compile_model(model=model, device_name="CPU")

I get 'accuracy': 0.7217741935483871 which matches the metric value returned by raspberry4 with NCS2.

At this point I think the problem is the Neural Compute Stick2.

Can I do something to avoid this performance degradation of my model running on NCS2?

Meanwhile I upload the code used in this repository https://github.com/lorytony/OpenVINO-NCS2

Issue submission checklist

• [ ] I report the issue, it's not a question
• [ ] I checked the problem with documentation, FAQ, open issues, Stack Overflow, etc and have not found solution
• [ ] There is reproducer code and related data files: images, videos, models, etc.

[zulkifli-halim]

Hi, Could you try to convert the model again and add --scale 255 to the model optimizer command?

And please share with us your inference code, as I see in this repo, the inference.py is empty.

[lorytony]

hi @zulkifli-halimi, I tried the conversions both with -255 and with weights at fp16 or fp32. But it didn't affect the results.

I thought of retraining our model on the same dataset using the pytorch framework. The model in pt format was converted to onnx and finally to xml bin format.

The xml file was run in the following three environments in the same testset and with the same version of Openvino(2021.4.2): 1) Windows10 pc using CPU 2) Windows10 pc using Neural Compute Stick 2 3) RaspberryPi4 Raspbian10 32bit using Neural Compute Stick 2

The resulting metrics (accuracy,f1-score,precision,recall) from each environment were equivalent. The phenomenon of accuracy degradation did not occur!

At this point in the previous comment my intuition was wrong. The problem is in the xml file related to the model in keras and not in the ncs2.

Tell me what you think. Anyway I'll upload the code used on the pc and on the raspberry to do the model inference in xml.

[zulkifli-halim]

This issue may be due to certain layers in the model that get processed differently with the MYRIAD and with the CPU. The relative accuracy difference between any of the target platforms and the reference metrics should be within 1%.

[zulkifli-halim]

Closing issue. Feel free to reopen in further assistance is needed.