Different output of inference results between (ONNX and TFLITE 32bit)

[beekeeper23]

Issue Type

Support

OS

Windows

OS architecture

x86_64

Programming Language

Python

Framework

OpenVINO, PyTorch, ONNX, TensorFlow

Download URL for ONNX / OpenVINO IR

https://drive.google.com/file/d/1h03treR__OHGcAwPL0Qq7nIk5fMi0P_n/view?usp=sharing

Description

Thank you for the immediate solution to the previous problem!

So, now I tried to convert the yolov5 algorithm with ShuffleNet ( only 4d vectors were used in the post-processing part of the algorithm and one head in the detection part) to tflite using the openvino2tensorflow, but the received TensorFlow model had different results on inference, compared to the onnx model.

Below the JSON file and output of inference testing are provided.

Relevant Log Output

{
    "format_version": 2,
    "layers": [
        {
            "layer_id": "269",
            "type": "Const",
            "replace_mode": "direct",
            "values": [
                0,
                3,
                4,
                1,
                2
            ]
        },
        {
            "layer_id": "271",
            "type": "Reshape", 
            "replace_mode": "insert_after",
            "values": [
                16,
                16,
                6,
                6
            ]
        },
        {
            "layer_id": "272",
            "type": "Const",
            "replace_mode": "direct",
            "values": [
                3
            ]
        },
        {
            "layer_id": "279",
            "type": "Reshape",
            "replace_mode": "insert_after",
            "values": [
                16,
                16,
                2,
                1
            ]
        },
        {
            "layer_id": "285",
            "type": "Const",
            "replace_mode": "direct",
            "values": [
                0,2,0,0

            ]
        },
        {
            "layer_id": "286",
            "type": "Const",
            "replace_mode": "direct",
            "values": [
                0,4,0,0

            ]
        },
        {
            "layer_id": "287",
            "type": "Const",
            "replace_mode": "direct",
            "values": [
                1,1,1,1

            ]
        },
        {
            "layer_id": "292",
            "type": "Transpose",
            "replace_mode": "insert_after",
            "values": [
                1,
                2,
                3,
                0
            ]
        },
        {
            "layer_id": "294",
            "type": "Concat",
            "replace_mode": "change_axis",
            "values": 2
        },
        {
            "layer_id": "296",
            "type": "Transpose",
            "replace_mode": "insert_before",
            "values": [
              0,
              3,
              1,
              2
            ]
        }
    ]
}

Source code for simple inference testing code

[PINTO0309]

It is very uncomfortable to deal with individual requirements. If there are any more consecutive posts with the same content, it will be closed immediately.

user@5b0c463cd830:~/workdir$ python3 onnx_openvino_tflite_test.py 
[Optimized] ONNX output @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
elapsed time: 0.8974075317382812ms
shape: (1, 1536, 6)
array([[[3.01748753e+00, 6.53862858e+00, 0.00000000e+00, 1.27895343e+00,
         1.09314919e-04, 9.70405817e-01],
        [1.11926670e+01, 6.86662388e+00, 0.00000000e+00, 1.16248798e+00,
         8.74400139e-05, 9.74289775e-01],
        [1.89740047e+01, 6.81226444e+00, 0.00000000e+00, 1.39419115e+00,
         8.24332237e-05, 9.74288583e-01],
        ...,
        [1.09703598e+02, 1.28077484e+02, 1.08411636e+01, 7.07119560e+00,
         1.31487846e-04, 9.84525442e-01],
        [1.17063286e+02, 1.28603699e+02, 9.92232323e+00, 6.81786585e+00,
         3.07798386e-04, 9.85273361e-01],
        [1.27492783e+02, 1.26608719e+02, 9.58653355e+00, 8.26488495e+00,
         8.28564167e-04, 9.84302998e-01]]], dtype=float32)

OpenVINO output @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
elapsed time: 1.4874935150146484ms
shape: (1, 1536, 6)
array([[[3.01748800e+00, 6.53862953e+00, 0.00000000e+00, 1.27895343e+00,
         1.09334644e-04, 9.70405877e-01],
        [1.11926670e+01, 6.86662483e+00, 0.00000000e+00, 1.16248739e+00,
         8.74732141e-05, 9.74289775e-01],
        [1.89740047e+01, 6.81226540e+00, 0.00000000e+00, 1.39419115e+00,
         8.24443923e-05, 9.74288583e-01],
        ...,
        [1.09703598e+02, 1.28077484e+02, 1.08411608e+01, 7.07119560e+00,
         1.31498673e-04, 9.84525442e-01],
        [1.17063286e+02, 1.28603699e+02, 9.92231941e+00, 6.81786394e+00,
         3.07835813e-04, 9.85273361e-01],
        [1.27492783e+02, 1.26608719e+02, 9.58653164e+00, 8.26488209e+00,
         8.28570395e-04, 9.84302998e-01]]], dtype=float32)

INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
tflite float32 output @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
elapsed time: 1.1420249938964844ms
shape: (1, 1536, 6)
array([[[3.01748800e+00, 6.53862953e+00, 0.00000000e+00, 1.27895343e+00,
         1.09334542e-04, 9.70405817e-01],
        [1.11926670e+01, 6.86662483e+00, 0.00000000e+00, 1.16248739e+00,
         8.74731268e-05, 9.74289775e-01],
        [1.89740047e+01, 6.81226349e+00, 0.00000000e+00, 1.39419115e+00,
         8.24443850e-05, 9.74288583e-01],
        ...,
        [1.09703598e+02, 1.28077484e+02, 1.08411636e+01, 7.07119465e+00,
         1.31498557e-04, 9.84525442e-01],
        [1.17063286e+02, 1.28603699e+02, 9.92232132e+00, 6.81786585e+00,
         3.07835784e-04, 9.85273361e-01],
        [1.27492775e+02, 1.26608719e+02, 9.58653164e+00, 8.26488304e+00,
         8.28570104e-04, 9.84302998e-01]]], dtype=float32)

{
    "format_version": 2,
    "layers": [
        {
            "layer_id": "266",
            "type": "Transpose",
            "replace_mode": "insert_after",
            "values": [
                0,
                3,
                1,
                2
            ]
        }
    ]
}