conversion of Yolov7-tiny ends with ERROR: axes don't match array

[ingura]

Issue Type

Bug

OS

Windows

OS architecture

x86_64

Programming Language

Python

Framework

TensorFlowLite

Download URL for ONNX / OpenVINO IR

pip install onnx==1.12.0 pip install onnxruntime onnxsim onnx-tf pip install openvino-dev==2022.3.0 pip install tensorflow-intel==2.10.0 pip install tensorflow-datasets pip install openvino2tensorflow --upgrade

Convert Script

Download the model

wget https://github.com/WongKinYiu/yolov7/releases/download/v0.1/yolov7-tiny.pt

Convert the Pytorch model to ONNX model

python export.py --weights yolov7-tiny.pt --grid --end2end --simplify --topk-all 100 --iou-thres 0.65 --conf-thres 0.35 --img-size 640 640 --max-wh 640

Convert ONNX model to OpenVINO IR

mo --input_model yolov7-tiny.onnx --input_shape [1,3,640,640] --output_dir model_openvino/ --data_type FP32

Convert model input shape to NHWC

python openvino2tensorflow.py --model_path model_openvino/yolov7-tiny.xml --model_output_path models-NHWC-final/ --output_saved_model --output_h5 --output_pb --output_no_quant_float32_tflite

Description

I am trying to use to convert the Pythorch version of the Yolov7-tiny model (https://github.com/WongKinYiu/yolov7) to the tflite version of the same model. If I take the conversion path PyTorch -> ONNX -> TF/TFlite the ONNX model works fine however the order of the input channels of the resulting TF model is NcHW instead of the NHWc required by TfLite. To correct the issue I am trying to use openvino2tensorflow using the conversion path PyTorch -> ONNX -> OpenVINO ->openvino2tensorflow -> TensorFlow / Tensorflow Lite. The conversion works without exceptions until i reach the openvino2tensorflow step where i use the following script:

python openvino2tensorflow.py --model_path model_openvino/yolov7-tiny.xml --model_output_path models-NHWC-final/ --output_saved_model --output_h5 --output_pb --output_no_quant_float32_tflite

Relevant Log Output

This leads to the following output:

`
...
......
====================================================================================
ERROR: axes don't match array
ERROR: model_path  : model_openvino/yolov7-tiny.xml
ERROR: weights_path: model_openvino/yolov7-tiny.bin
ERROR: layer_id    : 289
ERROR: input_layer0 layer_id=287: KerasTensor(type_spec=TensorSpec(shape=(1, 80, 3, 85, 2), dtype=tf.float32, name=None), name='tf.math.multiply/Mul:0', description="created by layer 'tf.math.multiply'")
ERROR: input_layer1 layer_id=288: Const(ndarray).shape  (1, 1, 80, 80, 2)
array([[[[[ -4.,  -4.],
          [  4.,  -4.],
          [ 12.,  -4.],
          ...,
          [612.,  -4.],
          [620.,  -4.],
          [628.,  -4.]],

         [[ -4.,   4.],
          [  4.,   4.],
          [ 12.,   4.],
          ...,
          [612.,   4.],
          [620.,   4.],
          [628.,   4.]],

         [[ -4.,  12.],
          [  4.,  12.],
          [ 12.,  12.],
          ...,
          [612.,  12.],
          [620.,  12.],
          [628.,  12.]],

         ...,

         [[ -4., 612.],
          [  4., 612.],
          [ 12., 612.],
          ...,
          [612., 612.],
          [620., 612.],
          [628., 612.]],

         [[ -4., 620.],
          [  4., 620.],
          [ 12., 620.],
          ...,
          [612., 620.],
          [620., 620.],
          [628., 620.]],

         [[ -4., 628.],
          [  4., 628.],
          [ 12., 628.],
          ...,
          [612., 628.],
          [620., 628.],
          [628., 628.]]]]], dtype=float32)
ERROR: The trace log is below.
Traceback (most recent call last):
  File "C:\...\openvino2tensorflow.py", line 1077, in convert
    tf_layers_dict[layer_id] = tf.math.add(tf_layers_dict[edge_id0], tf_layers_dict[edge_id1])
  File "C:\...\tensorflow\python\util\traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "C:\...\keras\layers\core\tf_op_layer.py", line 119, in handle
    return TFOpLambda(op)(*args, **kwargs)
  File "C:\...\keras\utils\traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
ValueError: Exception encountered when calling layer "tf.math.add_43" (type TFOpLambda).

Dimensions must be equal, but are 3 and 80 for '{{node tf.math.add_43/Add}} = AddV2[T=DT_FLOAT](Placeholder, tf.math.add_43/Add/y)' with input shapes: [1,80,3,85,2], [1,1,80,80,2].
Call arguments received by layer "tf.math.add_43" (type TFOpLambda):
  • x=tf.Tensor(shape=(1, 80, 3, 85, 2), dtype=float32)
  • y=array([[[[[ -4.,  -4.],
          [  4.,  -4.],
          [ 12.,  -4.],
          ...,
          [612.,  -4.],
          [620.,  -4.],
          [628.,  -4.]],

         [[ -4.,   4.],
          [  4.,   4.],
          [ 12.,   4.],
          ...,
          [612.,   4.],
          [620.,   4.],
          [628.,   4.]],

         [[ -4.,  12.],
          [  4.,  12.],
          [ 12.,  12.],
          ...,
          [612.,  12.],
          [620.,  12.],
          [628.,  12.]],

         ...,

         [[ -4., 612.],
          [  4., 612.],
          [ 12., 612.],
          ...,
          [612., 612.],
          [620., 612.],
          [628., 612.]],

         [[ -4., 620.],
          [  4., 620.],
          [ 12., 620.],
          ...,
          [612., 620.],
          [620., 620.],
          [628., 620.]],

         [[ -4., 628.],
          [  4., 628.],
          [ 12., 628.],
          ...,
          [612., 628.],
          [620., 628.],
          [628., 628.]]]]], dtype=float32)
  • name=None

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\...\openvino2tensorflow.py", line 1079, in convert
    tf_layers_dict[layer_id] = tf.math.add(tf_layers_dict[edge_id0], tf_layers_dict[edge_id1].transpose(0,2,3,1))
ValueError: axes don't match array
ERROR: Please refer to 6-7 in the README first. https://github.com/PINTO0309/openvino2tensorflow
`

Source code for simple inference testing code

import cv2
import time
import requests
import random
import numpy as np
import onnxruntime as ort
from PIL import Image
from pathlib import Path
from collections import OrderedDict,namedtuple
import tensorflow as tf
import matplotlib.pyplot as plt

def scaleAndFill(im, new_shape=(640, 640), color=(114, 114, 114), auto=True, scaleup=True, stride=32):
    # Resize and pad image while meeting stride-multiple constraints
    shape = im.shape[:2]  # current shape [height, width]
    if isinstance(new_shape, int):
        new_shape = (new_shape, new_shape)

    # Scale ratio (new / old)
    scale = min(new_shape[0] / shape[0], new_shape[1] / shape[1])
    if not scaleup:  # only scale down, do not scale up (for better val mAP)
        scale = min(scale, 1.0)

    # Compute padding
    new_unpad = int(round(shape[1] * scale)), int(round(shape[0] * scale))
    fillW, fillH = new_shape[1] - new_unpad[0], new_shape[0] - new_unpad[1]  #  padding

    if auto:  # minimum rectangle
        fillW, fillH = np.mod(fillW, stride), np.mod(fillH, stride)  #  padding

    fillW /= 2  # divide padding into 2 sides
    fillH /= 2

    if shape[::-1] != new_unpad:  # resize
        im = cv2.resize(im, new_unpad, interpolation=cv2.INTER_LINEAR)
    top, bottom = int(round(fillH - 0.1)), int(round(fillH + 0.1))
    left, right = int(round(fillW - 0.1)), int(round(fillW + 0.1))
    im = cv2.copyMakeBorder(im, top, bottom, left, right, cv2.BORDER_CONSTANT, value=color)  # fill border
    return im, scale, (fillW, fillH)

#Name of the classes according to class indices.
names = ['person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'traffic light', 
         'fire hydrant', 'stop sign', 'parking meter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 
         'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 
         'skis', 'snowboard', 'sports ball', 'kite', 'baseball bat', 'baseball glove', 'skateboard', 'surfboard', 
         'tennis racket', 'bottle', 'wine glass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 
         'sandwich', 'orange', 'broccoli', 'carrot', 'hot dog', 'pizza', 'donut', 'cake', 'chair', 'couch', 
         'potted plant', 'bed', 'dining table', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cell phone', 
         'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddy bear', 
         'hair drier', 'toothbrush']

#Creating random colors for bounding box visualization.
colors = {name:[random.randint(0, 255) for _ in range(3)] for i,name in enumerate(names)}

#Load and preprocess the test image.
img = cv2.imread("\\..\\image2.jpg")
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

image = img.copy()
image, ratio, dwdh = scaleAndFill(image,(640,640), auto=False)

image = np.expand_dims(image, 0)
image = np.ascontiguousarray(image)

im = image.astype(np.float32)
im /= 255

# Load the TFLite model and allocate tensors.
interpreter = tf.lite.Interpreter(model_path="./Models\\yolov7-tiny-fp32.tflite")
#Allocate tensors.
interpreter.allocate_tensors()
# Get input and output tensors.
input_details = interpreter.get_input_details()
interpreter.set_tensor(input_details[0]['index'], im)
interpreter.invoke()

output_data = interpreter.get_tensor(output_details[0]['index'])

## Visualize results

ori_images = [img.copy()]
testOutputSize=0
for i,(batch_id,x0,y0,x1,y1,cls_id,score) in enumerate(output_data):
    testOutputSize= testOutputSize+1
    print('batch_id: {}  clsID: {}  score: {}'.format(batch_id,cls_id,score ))
    image = ori_images[int(batch_id)]
    box = np.array([x0,y0,x1,y1])
    box -= np.array(dwdh*2)
    box /= ratio
    box = box.round().astype(np.int32).tolist()
    cls_id = int(cls_id)
    score = round(float(score),3)
    name = names[cls_id]
    color = colors[name]
    name += ' '+str(score)
    cv2.rectangle(image,box[:2],box[2:],color,2)
    cv2.putText(image,name,(box[0], box[1] - 2),cv2.FONT_HERSHEY_SIMPLEX,0.75,[225, 255, 255],thickness=2)  
plt.imshow(ori_images[0])
plt.title('TfLite Indications',  fontweight ="bold")
plt.show()

[PINTO0309]

I recommend the use of the latest tool here. No conversion errors occur and no accuracy errors are introduced. It also supports complex models such as Transformer. openvino2tensorflow is not actively maintained at this time. https://github.com/PINTO0309/onnx2tf

Converting a Yolov5 model to tensorflow #135

pip install -U onnx \
&& pip install -U nvidia-pyindex \
&& pip install -U onnx-graphsurgeon \
&& pip install -U onnxruntime \
&& pip install -U onnxsim \
&& pip install -U simple_onnx_processing_tools \
&& pip install -U onnx2tf \
&& pip install -U h5py==3.7.0

onnx2tf -i yolov7-tiny.onnx -osd -oh5 -cotof

https://user-images.githubusercontent.com/33194443/215622581-cbe52d48-5fc3-4cf9-bc8b-51f61e923ec2.mp4

[ingura]

Awesome! thank you.