search

openvinotoolkit / openvino

OpenVINO™ is an open-source toolkit for optimizing and deploying AI inference
https://docs.openvino.ai
Apache License 2.0
7.36k stars 2.3k forks source link

person-reidentification-retail-0300 on HDDL #1126

Closed ghost closed 4 years ago

ghost commented 4 years ago

Output values are very different from CPU.

import os
import numpy as np
from openvino.inference_engine import IECore

MODEL_REID = os.path.join("..", "models", "intel",
                          "person-reidentification-retail-0300", "%s",
                          "person-reidentification-retail-0300.%s")
HEIGHT = 256
WIDTH = 128

ie = IECore()
net_fp32 = ie.read_network(model=(MODEL_REID % ("FP32", "xml")),
                           weights=(MODEL_REID % ("FP32", "bin")))
net_fp16 = ie.read_network(model=(MODEL_REID % ("FP16", "xml")),
                           weights=(MODEL_REID % ("FP16", "bin")))

cpu_exec = ie.load_network(network=net_fp32, device_name="CPU") # reference
vpu_exec = ie.load_network(network=net_fp16, device_name="HDDL")

np.random.seed(0)
image = np.random.randint(0, 256, (1, 3, 256, 128))

input_blob = "data"
output_blob = "reid_embedding"

out_cpu = cpu_exec.infer({input_blob: image})[output_blob][0]
out_vpu = vpu_exec.infer({input_blob: image})[output_blob][0]
print(out_cpu[:32]) # reference
=>
[ 6.8710447e-06 -7.0161977e-06 -3.8732423e-06  1.0893067e-05
  7.0405008e-06 -6.3986095e-06  1.6915390e-05 -2.3208102e-05
  7.8262947e-06  1.3483948e-01  4.9406625e-02 -1.7084892e-06
 -5.2083509e-05  1.3480984e-01  1.6241057e-01  2.9326179e-03
  3.0082154e-01  1.3527721e-01 -2.4468700e-05  2.1820955e-01
 -9.8195905e-03  7.5303344e-03 -3.7727198e-06  1.3817801e-05
 -1.8803392e-02 -1.7836462e-01 -3.0745574e-05  6.5909452e-03
  1.0697336e-05  5.7600856e-02 -1.6748710e-02 -2.7515578e-01]
print(out_vpu[:32])
=>
[ 0.         -0.          0.          0.          0.         -0.
  0.          0.          0.          0.06359863  0.01040649  0.
  0.          0.09625244  0.12445068  0.00069189  0.20129395  0.10974121
 -0.          0.16101074 -0.02455139 -0.00648499  0.          0.
 -0.02659607 -0.12963867  0.         -0.0034771   0.          0.01727295
 -0.04885864 -0.25756836]
ilya-lavrenov commented 4 years ago

@AndrewBakalinIntel could you please help?

avitial commented 4 years ago

@isakamoto may I ask which type of HDDL card are you using? Also help provide the OpenVINO version and OS reproduced on.

ghost commented 4 years ago

Hi @avitial, thank you for your response. I am using Intel Vision Accelerator Design with Intel Movidius VPUs(Mustang-V100-MX8). OpenVINO version: 2020.3 OS: Windows

I was able to reproduce this on iei-mustang-v100-mx8 node of Intel DevCloud for the Edge as well.

avitial commented 4 years ago

@isakamoto the results are close enough, not identical but very close. The print statement for output results slightly differs in notation. In your reference results for example out_cpu[31] = -2.7515578e-01 and out_vpu[31] = -0.25756836 are the same.

Also note that you are running an FP32 on CPU vs FP16 on HDDL, try running same model precision on both devices for better comparison results.

cpu_exec = ie.load_network(network=net_fp32, device_name="CPU") # reference vpu_exec = ie.load_network(network=net_fp16, device_name="HDDL")

ghost commented 4 years ago

@avitial output values of FP16 on CPU are good.

import os
import numpy as np
from openvino.inference_engine import IECore

MODEL_REID = os.path.join("..", "models", "intel",
                          "person-reidentification-retail-0300", "%s",
                          "person-reidentification-retail-0300.%s")
CHANNELS = 3
HEIGHT = 256
WIDTH = 128

ie = IECore()
net_fp32 = ie.read_network(model=(MODEL_REID % ("FP32", "xml")),
                           weights=(MODEL_REID % ("FP32", "bin")))
net_fp16 = ie.read_network(model=(MODEL_REID % ("FP16", "xml")),
                           weights=(MODEL_REID % ("FP16", "bin")))

cpu_exec_32 = ie.load_network(network=net_fp32, device_name="CPU")
cpu_exec_16 = ie.load_network(network=net_fp16, device_name="CPU")
vpu_exec = ie.load_network(network=net_fp16, device_name="HDDL")

np.random.seed(0)
image = np.random.randint(0, 256, (1, CHANNELS, HEIGHT, WIDTH))

input_blob = "data"
output_blob = "reid_embedding"
out_cpu_32 = cpu_exec_32.infer({input_blob: image})[output_blob][0]
print(out_cpu_32[:32])
=>
[ 6.8710447e-06 -7.0161977e-06 -3.8732423e-06  1.0893067e-05
  7.0405008e-06 -6.3986095e-06  1.6915390e-05 -2.3208102e-05
  7.8262947e-06  1.3483948e-01  4.9406625e-02 -1.7084892e-06
 -5.2083509e-05  1.3480984e-01  1.6241057e-01  2.9326179e-03
  3.0082154e-01  1.3527721e-01 -2.4468700e-05  2.1820955e-01
 -9.8195905e-03  7.5303344e-03 -3.7727198e-06  1.3817801e-05
 -1.8803392e-02 -1.7836462e-01 -3.0745574e-05  6.5909452e-03
  1.0697336e-05  5.7600856e-02 -1.6748710e-02 -2.7515578e-01]
out_cpu_16 = cpu_exec_16.infer({input_blob: image})[output_blob][0]
print(out_cpu_16[:32])
=>
[ 6.8545341e-06 -7.0439264e-06 -3.8475227e-06  1.0907650e-05
  7.0333481e-06 -6.3928069e-06  1.6927719e-05 -2.3203522e-05
  7.8082085e-06  1.3506928e-01  4.9596179e-02 -1.7165871e-06
 -5.2089537e-05  1.3563587e-01  1.6268867e-01  2.9325716e-03
  3.0169013e-01  1.3577530e-01 -2.4446568e-05  2.1864854e-01
 -9.8624090e-03  7.5557828e-03 -3.7587758e-06  1.3828278e-05
 -1.8371228e-02 -1.7892802e-01 -3.0720988e-05  6.5794052e-03
  1.0669231e-05  5.8196954e-02 -1.6953975e-02 -2.7505499e-01]
out_vpu = vpu_exec.infer({input_blob: image})[output_blob][0]
print(out_vpu[:32])
=>
[ 0.         -0.          0.          0.          0.         -0.
  0.          0.          0.          0.06359863  0.01040649  0.
  0.          0.09625244  0.12445068  0.00069189  0.20129395  0.10974121
 -0.          0.16101074 -0.02455139 -0.00648499  0.          0.
 -0.02659607 -0.12963867  0.         -0.0034771   0.          0.01727295
 -0.04885864 -0.25756836]

Additionally I tested these networks for actual person images. Test person-reidentification-retail-0300

Thank you.

avitial commented 4 years ago

@isakamoto thanks for the update, so you see an issue with FP16 on VPU correct? Do you see the same with FP32 on VPU?

ghost commented 4 years ago

@avitial yes, I see the same with FP32 on VPU.

I updated Test person-reidentification-retail-0300

avitial commented 4 years ago

@isakamoto I can see the same results with such model for HDDL/Myriad plugins, there is an older version of the model available from previous releases person-reidentification-retail-0200 that doesn't seem to behave the same. Please give that a try and see if it meets your needs, find the model available for download here.

ghost commented 4 years ago

@avitial thank you for your information. I got expected results with previous model. Test person-reidentification-retail-0200

Will current model be fixed?

avitial commented 4 years ago

@isakamoto I've shared this problem with development team, no ETA or commit to fix. I will share any updates if I have any, apologies for the inconvenience this may cause.

ghost commented 4 years ago

@avitial thank you for your support.

vladimir-dudnik commented 4 years ago

we confirm that accuracy of model person-reidentification-retail-0300 was different between CPU/GPU and MyriadX/HDDL-R in OpenVINO 2020.3.

topology metric REF CPU GPU MyriadX HDDL-R
person-reidentification-retail-0300 reid_map 0.885 0.881488 0.881488 0.831629 0.831629
person-reidentification-retail-0300 rank@1 0.963 0.963777 0.963777 0.958432 0.958432

Note, models person-reidentification-retail-0031, person-reidentification-retail-0249 and person-reidentification-retail-0200 were end of lifed in OpenVINO 2020.4. New models person-reidentification-retail-0265, person-reidentification-retail-0267 and person-reidentification-retail-0270 were added in OpenVINO 2020.4 and these models doesn't have the difference in accuracy between CPU/GPU and HDDL-R.

avitial commented 4 years ago

@isakamoto kindly refer to vladimir-dudnik's comment above. I've tested these new models available in v2020.4 release and I don't see the problem you were experiencing before, please give these models a try in latest release:

I will go ahead now and close this issue as it has been addressed, but feel free to reopen and ask any additional questions related to this topic.