Incorrect connection order in Concat

[PINTO0309]

1. OS you are using e.g. Ubuntu 20.04, WIndows10, etc: Ubuntu20.04 (on Docker)

2. OS Architecture e.g. x86_64, armv7l, aarch64, etc: x86_64

3. Version of OpenVINO e.g. 2021.2.185, etc: 2021.4.582

4. Version of TensorFlow e.g. v2.4.1, tf-nightly==2.5.0.dev20210128, etc: v2.6.0-rc1

5. Version of TensorRT e.g. TensorRT6.0 GA, etc: 8.0

10. Download URL for OpenVINO IR (.bin/.xml) model

https://github.com/PINTO0309/PINTO_model_zoo/tree/main/132_YOLOX

11. URL of the repository from which the transformed model was taken:

https://github.com/PINTO0309/PINTO_model_zoo/tree/main/132_YOLOX

13. Issue Details

In models such as YOLOv5, YOLOR, and YOLOX, the connection order of input layers is different when the starting point is concat via multiple StridedSlices. Therefore, even if the structure of the intermediate layer is correctly converted, the data at the input will be corrupted, and the output result will be meaningless. The order of tensor decomposition and the order of concatenation by Concat need to be controlled correctly.

• ONNX (Before conversion)

• TFLite (openvino2tensorflow / after conversion)

• TFLite (onnx-tensorflow / after convertion)

[NobuoTsukamoto]

If my understanding is correct, As far as YOLOX is concerned, the Strided Slice of the input layer has the same parameters. Therefore, it seems that there is no effect even if the order is different.

The problem seems to be with Concat in the middle layer.

• There are two in YOLOX 320X320.
• Both have resizing in one of the Opes.

The following is one of them. OpenVINO model

TensorFlow Lite model. Although it is shown in red, the order of Concat is different between the OpenVINO model and the TensorFlow Lite model.

In the case of a concatenate like Skip connection like U-Net, it seems to be because the id of input and the actual order are reversed.

[PINTO0309]

@NobuoTsukamoto Thank you for your support! :smile_cat:

Well, you have a point. However, I am very worried about how to fix it. The only way I can think of is to sort the layer IDs entered in the INPUT of OpenVINO's Concat in ascending order and then Concat them. However, since OpenVINO often reverses the order of operations displayed in INPUT and sometimes not, I am not sure if this fix will work for all models.

Anyway, I'm going to apply the workaround, regenerate the model, and provide the tflite model to you again.

[PINTO0309]

This is a part of the log of the tflite conversion process of concat. The assumption was the exact opposite. When importing openvino concat layers, the layer IDs were sorted in ascending order. I may need to modify it to accept the order as it is, without sorting.

[PINTO0309]

However, the xml file is connected to concat in the order of this image. In other words, the conversion behavior to tflite appears to be working correctly. "304" = Concat, "228" = Mul, "303" = Interporate

    # edges
    added_key_list = []
    for edge in edges:
        to_layer = edge.attrib['to-layer']
        from_layer = edge.attrib['from-layer']
        from_layer_port = edge.attrib['from-port']

        for layer in layers:
            if layer.attrib['id'] == to_layer:
                output_layer_ports = layer.find('output')
                if layer.attrib['type'] != 'Result' and len(output_layer_ports) >= 2:
                    for port in output_layer_ports:
                        tf_edges.setdefault('{}:{}'.format(to_layer, port.attrib['id']), []).append(from_layer)
                    added_key_list.append(to_layer)
                else:
                    tf_edges.setdefault(to_layer, [])

[PINTO0309]

I understand. The port numbers are reversed. It was found that there was a bug that the port number was incorrectly sorted in ascending order only when the destination of the skip connection was Concat.

[PINTO0309]

Fixes f037817dcaeebe9208293dd598d93f39d7d13602