Can't get a custom tiny-yolov3 model to work (RPi+NCS2)

[lbcastro]

I'm using the following commands to convert a tiny yolov3 weights file to a .bin/.xml pair:

From https://github.com/mystic123/tensorflow-yolo-v3:

python convert_weights_pb.py \
        --weights_file custom_tiny_yolov3.weights \
        --class_names custom_tiny_yolov3.names \
        --data_format NHWC \
        --tiny \
        --output_graph custom_tiny_yolov3.pb

sudo python3 /opt/intel/computer_vision_sdk_2018.5.445/deployment_tools/model_optimizer/mo_tf.py \
        --input_model custom_tiny_yolov3.pb \
        --output_dir custom_tiny_yolov3/ \
        --data_type FP16 \
        --batch 1 \
        --tensorflow_use_custom_operations_config modified-tiny-yolov3.json

My modified-tiny-yolov3.json file looks like this:

[
  {
    "id": "TFYOLOV3",
    "match_kind": "general",
    "custom_attributes": {
      "classes": 6,
      "coords": 4,
      "num": 6,
      "mask": [0,1,2],
      "entry_points": ["detector/yolo-v3-tiny/Reshape","detector/yolo-v3-tiny/Reshape_4"]
    }
  }
]

Modified the openvino_tiny-yolov3_test.py: classes, num and LABELS variables to reflect the custom tiny yolov3 model; changed the model_xml variable in the "main_IE_infer()" function to point to the folder containing the .bin and .xml files.

classes = 6
coords = 4
num = 6
anchors = [10,14, 23,27, 37,58, 81,82, 135,169, 344,319]
LABELS = ("SIX", "DIFFERENT", "LABELS", "ARE", "INCLUDED", "HERE")

After executing "python3 openvino_tiny-yolov3_test.py -d MYRIAD" I always get errors like:

Traceback (most recent call last):
  File "openvino_tiny-yolov3_test.py", line 244, in <module>
    sys.exit(main_IE_infer() or 0)
  File "openvino_tiny-yolov3_test.py", line 203, in main_IE_infer
    objects = ParseYOLOV3Output(output, m_input_size, m_input_size, camera_height, camera_width, 0.2, objects)
  File "openvino_tiny-yolov3_test.py", line 125, in ParseYOLOV3Output
    scale = output_blob[obj_index]
IndexError: index 25012 is out of bounds for axis 0 with size 22308

I know your version is very early, and I really appreciate the effort. If you have any idea why I'm getting this error, it would be great if you could help me fix it. It seems to me the problem is related to the conversion from .weights/.cfg to .bin/.xml, since I've tried the same with the default tiny yolov3 files and I got a similar error. However, I've tried following every advice from the Intel forums, with no success. Please let me know if you need additional information.

Thanks a lot!

[PINTO0309]

@lbcastro

Please try the correction below. Just adjust it to the number of classes you defined.

openvino_tiny-yolov3_test.py

classes = 80
↓
classes = 6

[PINTO0309]

@lbcastro

Sorry, I made a mistake. I was a little panicked.

openvino_tiny-yolov3_test.py

classes = 6
coords = 4
num = 3

[lbcastro]

No problem, thanks a lot for answering so quickly. I tried that and I get a different error:

Traceback (most recent call last):
  File "openvino_tiny-yolov3_test.py", line 245, in <module>
    sys.exit(main_IE_infer() or 0)
  File "openvino_tiny-yolov3_test.py", line 212, in main_IE_infer
    if (IntersectionOverUnion(objects[i], objects[j]) >= 0.4):
  File "openvino_tiny-yolov3_test.py", line 84, in IntersectionOverUnion
    return (area_of_overlap / area_of_union)
ZeroDivisionError: division by zero

[PINTO0309]

@lbcastro

Your point is very helpful to me. Please tell me what behavior will be taken if you try the following things. There seems to be a very serious bug in my program.

def IntersectionOverUnion(box_1, box_2):
    width_of_overlap_area = min(box_1.xmax, box_2.xmax) - max(box_1.xmin, box_2.xmin)
    height_of_overlap_area = min(box_1.ymax, box_2.ymax) - max(box_1.ymin, box_2.ymin)
    area_of_overlap = 0.0
    if (width_of_overlap_area < 0.0 or height_of_overlap_area < 0.0):
        area_of_overlap = 0.0
    else:
        area_of_overlap = width_of_overlap_area * height_of_overlap_area
    box_1_area = (box_1.ymax - box_1.ymin)  * (box_1.xmax - box_1.xmin)
    box_2_area = (box_2.ymax - box_2.ymin)  * (box_2.xmax - box_2.xmin)
    area_of_union = box_1_area + box_2_area - area_of_overlap
    retval = 0.0
    if area_of_union <= 0.0:
        retval = 0.0
    else:
        retval = (area_of_overlap / area_of_union)
    return retval

        # Filtering overlapping boxes
        objlen = len(objects)
        for i in range(objlen):
            if (objects[i].confidence == 0.0):
                continue
            for j in range(i + 1, objlen):
                if (IntersectionOverUnion(objects[i], objects[j]) >= 0.4):
                    if objects[i].confidence < objects[j].confidence:
                        objects[i], objects[j] = objects[j], objects[i]
                    objects[j].confidence = 0

[lbcastro]

I'll be glad to help in any way I can, however I left my Movidius sticks at work. In the meantime I can send you the model I'm using, otherwise I'll get back to you on this issue in a few days.

[lbcastro]

I think there's something wrong with my conversion process from yolo weights/cfg to bin/xml. After your suggestion, the "objects" variable inside the main inference loop has thousands of entries, whereas using the default tiny-yolov3 model I only get a very small amount of them. I'm going to try to find what's wrong with the procedure I've described in my first post here, since I've already verified that my yolo model is working correctly in the weights/cfg format.

[lbcastro]

It seems there are some reported problems with using https://github.com/mystic123/tensorflow-yolo-v3 to convert tiny-yolov3 to pb, as noted in the repo's issues.

[PINTO0309]

@lbcastro

It seems there are some reported problems with using https://github.com/mystic123/tensorflow-yolo-v3 to convert tiny-yolov3 to pb, as noted in the repo's issues.

Yes. I know the issue. The reason why so many bounding boxes are displayed is probably because the accuracy of the model is extremely low. I am trying to regenerate tiny-YoloV3 using another repository. I have repeatedly failed, so it seems to take several days to succeed.

[PINTO0309]

I solved the problem of low precision. There was a mistake in the logic of preprocessing and postprocessing.

And there seems to be a bug in NCS 2's internal API. https://software.intel.com/en-us/forums/computer-vision/topic/804818

[umbralada]

Hi.

Katsuya Hyodo, thank you for the actual tasks and development for connecting OpenVINO+yolov3+RPi+NCS

I use your repository and https://github.com/mystic123/tensorflow-yolo-v3 to test the conversion of my network my-tiny-yolov3. In theory, the num parameter in the my-tiny-yolov3.json file and at the beginning of the openvino_tiny-yolov3_MultiStick_test.py script should have the same value as in the network architecture file my-tiny-yolov3.cfg. When I setting num=6 (as at the beginning of this issue) errors appear: "Traceback (most recent call last): File "/home/OpenVINO-YoloV3/openvino_tiny-yolov3_MultiStick_test.py", line 336, in predict_async objects = ParseYOLOV3Output (output, self.new_h, self.new_w, self.camera_height, self.camera_width, self.threshould, objects) File "/home/OpenVINO-YoloV3/openvino_tiny-yolov3_MultiStick_test.py", line 122, in ParseYOLOV3Output scale = output_blob [obj_index] IndexError: index 8281 is out of bounds for axis 0 with size 7605 "

With the values ​​of the parameter num=3 and the threshold value confidence>0.85, the detection results are more or less normal, but for some reason at the end of the video (8 seconds) a lot of false detections appear. Do you think this is due to errors in your script or conversion?

[PINTO0309]

@umbralada

I understand that the original meaning of "num" is the number of "anchor". However, "num" described in my program represents the size of "mask". I felt quite uncomfortable when porting the Intel sample program to Python, but the Intel C ++ sample program implements "num" as the size of "mask". Note that in implementation "num" is not directly related to the number of "anchor".

    // --------------------------- Extracting layer parameters -------------------------------------
    auto num = layer->GetParamAsInt("num");
    try { num = layer->GetParamAsInts("mask").size(); } catch (...) {}
    auto coords = layer->GetParamAsInt("coords");
    auto classes = layer->GetParamAsInt("classes");
    std::vector<float> anchors = {10.0, 13.0, 16.0, 30.0, 33.0, 23.0, 30.0, 61.0, 62.0, 45.0, 59.0, 119.0, 116.0, 90.0, 156.0, 198.0, 373.0, 326.0};
    try { anchors = layer->GetParamAsFloats("anchors"); } catch (...) {}
    auto side = out_blob_h;
    int anchor_offset = 0;

"num" of JSON = number of anchor "num" of Program = size of mask

Below is my article that describes how to convert one class tiny-YoloV3. https://qiita.com/PINTO/items/7dd7135085a7249bf17a (Japanese)

[umbralada]

@PINTO0309 Thank you so much for the quick answer! If possible, please specify the parameter "coords" - is this the count of BB corners? It is in the JSON file, but not in the configuration file *.cfg.