performance of the model deployed in openvino is low

[andeyeluguo]

Hi AlexeyAB，

I try to deploy the yolov3-tiny in openvino according to theofficial tutorial of openvino but the performance is bad , I also try to get the reason why, and found lots of people meet this problem.

• issue from convert tool tensorflow-yolo-v3 which openvino official example usethinks that maybe it is the upsample layer

• issue from the openvino-yolov3 which is inspired by the tensorflow-yolo-v3 think that there was a mistake in the logic of preprocessing and postprocessing, only change this code

      self.new_w = int(camera_width * min(self.m_input_size/camera_width, self.m_input_size/camera_height))
      self.new_h = int(camera_height * min(self.m_input_size/camera_width, self.m_input_size/camera_height))

  into

      self.new_w = int(camera_width * self.m_input_size/camera_width)
      self.new_h = int(camera_height * self.m_input_size/camera_height)

  this code change the size of resized the input image. The author declare he has solved

• this article thinks that the resize mode when we train model is letterbox, but the OpenVINO ony support the bilinear and area, not the letterbox, so we need to resize on CPU by ourselfs.

The question is:

• Is it the resize way that reduce the performance ?
• Does the upsample layer use the letterbox in darknet?
• how to avoid this decrease in performance?

Regards

[andeyeluguo]

@AlexeyAB I see your results https://github.com/AlexeyAB/darknet/issues/5079#issue-585403577

the performance of the yolov3-tiny on coco is 33.1% Would you please give the procedure how you process.

[AlexeyAB]

but the performance is bad

• what is the performance, speed or accuracy?
• how did you measure performance?
• what value of performance did you get?
• OpenVINO uses hardcoded anchors for yolov3.cfg, so you should change it to yolov3-tiny.cfg and recompile:
  • https://github.com/opencv/open_model_zoo/blob/master/demos/object_detection_demo_yolov3_async/main.cpp#L118-L119
  • https://github.com/AlexeyAB/darknet/blob/0c7305cf638bd0e692c6195e412aff93200000e4/cfg/yolov3-tiny.cfg#L176
• how to test accuracy of any model: https://github.com/AlexeyAB/darknet/wiki/How-to-evaluate-accuracy-and-speed-of-YOLOv4#how-to-evaluate-ap-of-yolov4-on-the-ms-coco-evaluation-server
• how to test speed of any model on VPU: https://github.com/AlexeyAB/darknet/wiki/Converting-Yolo-v3-models-to-TensorFlow-and-OpenVINO(IR)-models#run-yolo-v3-models-on-opencv-dnn-with-openvino-dl-ie-backend
• all your links are broken

[andeyeluguo]

the links now is available

• I just use an video tested on darknet_alexey and openvino(converted from darknet), the openvino can not detect anything
• I try your suggetion to change the anchors, find it did not work in my case, may it will get anchors from the json file and replace the initial anchors

[AlexeyAB]

Try to use OpenCV-dnn + OpenVINO-backend: https://github.com/AlexeyAB/darknet/wiki/Converting-Yolo-v3-models-to-TensorFlow-and-OpenVINO(IR)-models#run-yolo-v3-models-on-opencv-dnn-with-openvino-dl-ie-backend

[andeyeluguo]

I have worked on flatform of the openvino toolkit for a month, do many research and solved many problems.

• Maybe this is the last step of my work, but the openvino offical tutorial seems to be wrong. many people has meet this issue.
• Have you used the toolkit or just just use this? I see you got 33.1% MAP of the yolov3-tiny.
• Does it still has hope to be solved? Maybe I have no much time to transform to another scheme
• My platform is windows

[AlexeyAB]

• I successfully used yolov3-tiny on both OpenVINO (Linux) and OpenCV-dnn+OpenVINO-IE (Win/Linux)

• 33.1% AP50 is measured by using Darknet on mscoco test-dev evaluation server (I didn't measure AP by using OpenVINO, but detections look good)

[andeyeluguo]

your detections looks good

• Will you please give me the pictures(or videos) and results you test, I want to compare with you, and try to debug what's wrong with me

[andeyeluguo]

Hi AlexeyAB,

• I have confirmed that it is the resize mode lead to the phenomenon openvino can not detect object
• what this article describe is right, the resize mode really have a big effect on performance
• Just crop the ROI with a size(416, 416) from the image
• people who use the openvino toolkit must rewrite the resize mode
• what I can not confirm is the issue, does the upsample layer have the resize operation ?

[andeyeluguo]

not ony the resize mode cause it

• I have notice your code does not use the letterbox
• the same video on darknet_alexyAB is good, but on openvino it can't detect anything. Cropped a ROI with a size(416, 416), openvino will detect some boxes, but still not so good as on darknet.

[AlexeyAB]

• what this article describe is right, the resize mode really have a big effect on performance

Yes, resizing in TF was broken, but it isn't related to OpenVINO.

Current yolo models use resize instad of letter_box.

But using letterbox for models which was trained with resizing should not spoil detection much. OpenCV-dnn uses letterbox, but all yolo models works well with OpenCV-dnn.

I think the issue is in anchors: https://github.com/opencv/open_model_zoo/blob/master/demos/object_detection_demo_yolov3_async/main.cpp#L118-L119

[andeyeluguo]

the issue about anchor

• I have debugged, found this function will obtain the anchors from the xml file when I change

  std::vector<float> anchors = {10.0, 13.0, 16.0, 30.0, 33.0, 23.0, 30.0, 61.0, 62.0, 45.0, 59.0, 119.0, 116.0, 90.0,156.0, 198.0, 373.0, 326.0};

  into

  std::vector<float> anchors;

  or initial it to be other value, it does't affect the result.

[andeyeluguo]

I just compare one video processed by yolov3 in darknet and openvino respectively, many boxes is missed in openvino .

[andeyeluguo]

sorry, I change the threshold, then the result of detecting a video using yolov3 and yolov3-tiny on darknet and openvino is very likely

• what bad performance is my one-class model based yolov3-tiny.
• Maybe something is wrong with my process, or my model is hard to be quantized to FP32（I am wondering if it needs to be quantized here, or it is simple float）
• some people meet the same question, this issue trained a model based yolov3 drops a lot.
• my model trained based on your darknet, all setting is default, while the yolov3.weights and yolov3-tiny.weights is from the pjreddie. Does the train platform affect the result ?

[gganduu]

sorry, I change the threshold, then the result of detecting a video using yolov3 and yolov3-tiny on darknet and openvino is very likely

• what bad performance is my one-class model based yolov3-tiny.
• Maybe something is wrong with my process, or my model is hard to be quantized to FP32（I am wondering if it needs to be quantized here, or it is simple float）
• some people meet the same question, this issue trained a model based yolov3 drops a lot.
• my model trained based on your darknet, all setting is default, while the yolov3.weights and yolov3-tiny.weights is from the pjreddie. Does the train platform affect the result ?

Hi, I met the same problem, and you have closed this issue. Have you solve this problem already?