[Consolidate all YOLOv5 issues] YOLOv5 export error `ValueError: axes don't match array`

glenn-jocher commented 1 year ago

Issue Type

Other

OS

Ubuntu

OS architecture

x86_64

Programming Language

Python

Framework

OpenVINO, PyTorch

Download URL for ONNX / OpenVINO IR

yolov5n_openvino_model.zip

Convert Script

!openvino2tensorflow \
  --model_path yolov5n_openvino_model/yolov5n.xml \
  --output_saved_model \
  --output_pb \
  --output_weight_quant_tflite \
  --output_float16_quant_tflite \
  --output_no_quant_float32_tflite

Description

In Google Colab I exported YOLOv5n model from PyTorch to OpenVINO, then used openvino2tensorflow command above which produced error message below.

Relevant Log Output

...

          ...,
          [612., 620.],
          [620., 620.],
          [628., 620.]],

         [[ -4., 628.],
          [  4., 628.],
          [ 12., 628.],
          ...,
          [612., 628.],
          [620., 628.],
          [628., 628.]]],

        [[[ -4.,  -4.],
          [  4.,  -4.],
          [ 12.,  -4.],
          ...,
          [612.,  -4.],
          [620.,  -4.],
          [628.,  -4.]],

         [[ -4.,   4.],
          [  4.,   4.],
          [ 12.,   4.],
          ...,
          [612.,   4.],
          [620.,   4.],
          [628.,   4.]],

         [[ -4.,  12.],
          [  4.,  12.],
          [ 12.,  12.],
          ...,
          [612.,  12.],
          [620.,  12.],
          [628.,  12.]],

         ...,

         [[ -4., 612.],
          [  4., 612.],
          [ 12., 612.],
          ...,
          [612., 612.],
          [620., 612.],
          [628., 612.]],

         [[ -4., 620.],
          [  4., 620.],
          [ 12., 620.],
          ...,
          [612., 620.],
          [620., 620.],
          [628., 620.]],

         [[ -4., 628.],
          [  4., 628.],
          [ 12., 628.],
          ...,
          [612., 628.],
          [620., 628.],
          [628., 628.]]]]], dtype=float32)
  • name=None

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/openvino2tensorflow", line 1079, in convert
    tf_layers_dict[layer_id] = tf.math.add(tf_layers_dict[edge_id0], tf_layers_dict[edge_id1].transpose(0,2,3,1))
ValueError: axes don't match array
ERROR: Please refer to 6-7 in the README first. https://github.com/PINTO0309/openvino2tensorflow

Source code for simple inference testing code

!git clone https://github.com/ultralytics/yolov5  # clone
%cd yolov5
%pip install -qr requirements.txt onnx openvino2tensorflow openvino-dev # install

!python export.py --weights yolov5n.pt --include openvino
!python detect.py --weights yolov5n_openvino_model

Using https://colab.research.google.com/github/ultralytics/yolov5/blob/master/tutorial.ipynb

PINTO0309 commented 1 year ago

convert - first try

openvino2tensorflow \
--model_path yolov5n.xml \
--output_saved_model \
--output_pb \
--non_verbose \
--output_no_quant_float32_tflite

Here. Shape Error occurred.

ERROR: axes don't match array
ERROR: model_path  : yolov5n.xml
ERROR: weights_path: yolov5n.bin
ERROR: layer_id    : 271 <<<<<<<<<<<--------------------- Shape Error occured
ERROR: input_layer0 layer_id=269: KerasTensor(type_spec=TensorSpec(shape=(1, 80, 3, 85, 2), dtype=tf.float32, name=None), name='tf.math.multiply/Mul:0', description="created by layer 'tf.math.multiply'")
ERROR: input_layer1 layer_id=270: Const(ndarray).shape  (1, 3, 80, 80, 2)
array([[[[[ -4.,  -4.],
      [  4.,  -4.],
      [ 12.,  -4.],
:
:
ERROR: Please refer to 6-7 in the README first. https://github.com/PINTO0309/openvino2tensorflow

The conversion of the unpredictable transposition operation from NCDHW to NCHWD fails at this point. Mechanically unpredictable transposition operations require the tool's behavior to be modified using JSON files. The cause of this error is that the transpose before the OP in which the error occurs failed to correct the transposition order.

layer_type: Reshape
layer_id: 261
input_layer0: layer_id=259: KerasTensor(type_spec=TensorSpec(shape=(1, 80, 80, 255), dtype=tf.float32, name=None), name='tf.math.add_52/Add:0', description="created by layer 'tf.math.add_52'")
input_layer1_shape: layer_id=260: (5,)
tf_layers_dict: KerasTensor(type_spec=TensorSpec(shape=(1, 80, 80, 3, 85), dtype=tf.float32, name=None), name='tf.reshape/Reshape:0', description="created by layer 'tf.reshape'")
====================================================================================
layer_type: Const
layer_id: 262
tf_layers_dict_shape: (5,)
====================================================================================
layer_type: Transpose
layer_id: 263 <<<<<<<<<<<--------------------- The conversion of the unpredictable transposition operation from NCDHW to NCHWD fails at this point.
input_layer0: layer_id=261: KerasTensor(type_spec=TensorSpec(shape=(1, 80, 80, 3, 85), dtype=tf.float32, name=None), name='tf.reshape/Reshape:0', description="created by layer 'tf.reshape'")
input_layer1_shape: layer_id=262: (5,)
tf_layers_dict: KerasTensor(type_spec=TensorSpec(shape=(1, 80, 3, 85, 80), dtype=tf.float32, name=None), name='tf.compat.v1.transpose/transpose:0', description="created by layer 'tf.compat.v1.transpose'")

replace_n.json - first step Rewrite the transposition order of Transpose so that it behaves as expected. The inverted order constant corresponds to the layer number of the name of output minus 1. output name: 263:2 -> layer_id:262 = custom name: Constant_3067
```
{
"format_version": 2,
"layers": [
    {
      "layer_id": "262",
      "type": "Const",
      "replace_mode": "direct",
      "values": [
          0,
          3,
          1,
          2,
          4
      ]
    }
]
}
```

re-convert

openvino2tensorflow \
--model_path yolov5n.xml \
--output_saved_model \
--output_pb \
--non_verbose \
--output_no_quant_float32_tflite \
--weight_replacement_config replace_n.json

Shape Error occurred.

ERROR: axes don't match array
ERROR: model_path  : yolov5n.xml
ERROR: weights_path: yolov5n.bin
ERROR: layer_id    : 328 <<<<<<<<<<<--------------------- Shape Error occured
ERROR: input_layer0 layer_id=326: KerasTensor(type_spec=TensorSpec(shape=(1, 40, 3, 85, 2), dtype=tf.float32, name=None), name='tf.math.multiply_3/Mul:0', description="created by layer 'tf.math.multiply_3'")
ERROR: input_layer1 layer_id=327: Const(ndarray).shape  (1, 3, 40, 40, 2)
array([[[[[ -8.,  -8.],
      [  8.,  -8.],
      [ 24.,  -8.],

The conversion of the unpredictable transposition operation from NCDHW to NCHWD fails at this point.

====================================================================================
layer_type: Reshape
layer_id: 318
input_layer0: layer_id=316: KerasTensor(type_spec=TensorSpec(shape=(1, 40, 40, 255), dtype=tf.float32, name=None), name='tf.math.add_60/Add:0', description="created by layer 'tf.math.add_60'")
input_layer1_shape: layer_id=317: (5,)
tf_layers_dict: KerasTensor(type_spec=TensorSpec(shape=(1, 40, 40, 3, 85), dtype=tf.float32, name=None), name='tf.reshape_2/Reshape:0', description="created by layer 'tf.reshape_2'")
====================================================================================
layer_type: Const
layer_id: 319
tf_layers_dict_shape: (5,)
====================================================================================
layer_type: Transpose
layer_id: 320 <<<<<<<<<<<--------------------- The conversion of the unpredictable transposition operation from NCDHW to NCHWD fails at this point.
input_layer0: layer_id=318: KerasTensor(type_spec=TensorSpec(shape=(1, 40, 40, 3, 85), dtype=tf.float32, name=None), name='tf.reshape_2/Reshape:0', description="created by layer 'tf.reshape_2'")
input_layer1_shape: layer_id=319: (5,)
tf_layers_dict: KerasTensor(type_spec=TensorSpec(shape=(1, 40, 3, 85, 40), dtype=tf.float32, name=None), name='tf.compat.v1.transpose_1/transpose:0', description="created by layer 'tf.compat.v1.transpose_1'")
====================================================================================

replace_n.json - second step

{
"format_version": 2,
"layers": [
    {
      "layer_id": "262",
      "type": "Const",
      "replace_mode": "direct",
      "values": [
          0,
          3,
          1,
          2,
          4
      ]
    },
    {
      "layer_id": "319",
      "type": "Const",
      "replace_mode": "direct",
      "values": [
          0,
          3,
          1,
          2,
          4
      ]
    }
]
}

re-convert

openvino2tensorflow \
--model_path yolov5n.xml \
--output_saved_model \
--output_pb \
--non_verbose \
--output_no_quant_float32_tflite \
--weight_replacement_config replace_n.json

Shape Error occurred.

ERROR: axes don't match array
ERROR: model_path  : yolov5n.xml
ERROR: weights_path: yolov5n.bin
ERROR: layer_id    : 384
ERROR: input_layer0 layer_id=382: KerasTensor(type_spec=TensorSpec(shape=(1, 20, 3, 85, 2), dtype=tf.float32, name=None), name='tf.math.multiply_6/Mul:0', description="created by layer 'tf.math.multiply_6'")
ERROR: input_layer1 layer_id=383: Const(ndarray).shape  (1, 3, 20, 20, 2)
array([[[[[-16., -16.],
      [ 16., -16.],

The conversion of the unpredictable transposition operation from NCDHW to NCHWD fails at this point.

====================================================================================
layer_type: Reshape
layer_id: 374
input_layer0: layer_id=372: KerasTensor(type_spec=TensorSpec(shape=(1, 20, 20, 255), dtype=tf.float32, name=None), name='tf.math.add_68/Add:0', description="created by layer 'tf.math.add_68'")
input_layer1_shape: layer_id=373: (5,)
tf_layers_dict: KerasTensor(type_spec=TensorSpec(shape=(1, 20, 20, 3, 85), dtype=tf.float32, name=None), name='tf.reshape_4/Reshape:0', description="created by layer 'tf.reshape_4'")
====================================================================================
layer_type: Const
layer_id: 375
tf_layers_dict_shape: (5,)
====================================================================================
layer_type: Transpose
layer_id: 376 <<<<<<<<<<<--------------------- The conversion of the unpredictable transposition operation from NCDHW to NCHWD fails at this point.
input_layer0: layer_id=374: KerasTensor(type_spec=TensorSpec(shape=(1, 20, 20, 3, 85), dtype=tf.float32, name=None), name='tf.reshape_4/Reshape:0', description="created by layer 'tf.reshape_4'")
input_layer1_shape: layer_id=375: (5,)
tf_layers_dict: KerasTensor(type_spec=TensorSpec(shape=(1, 20, 3, 85, 20), dtype=tf.float32, name=None), name='tf.compat.v1.transpose_2/transpose:0', description="created by layer 'tf.compat.v1.transpose_2'")
====================================================================================

replace_n.json - second step

{
"format_version": 2,
"layers": [
    {
      "layer_id": "262",
      "type": "Const",
      "replace_mode": "direct",
      "values": [
          0,
          3,
          1,
          2,
          4
      ]
    },
    {
      "layer_id": "319",
      "type": "Const",
      "replace_mode": "direct",
      "values": [
          0,
          3,
          1,
          2,
          4
      ]
    },
    {
      "layer_id": "375",
      "type": "Const",
      "replace_mode": "direct",
      "values": [
          0,
          3,
          1,
          2,
          4
      ]
    }
]
}

re-convert

openvino2tensorflow \
--model_path yolov5n.xml \
--output_saved_model \
--output_pb \
--non_verbose \
--output_no_quant_float32_tflite \
--weight_replacement_config replace_n.json

success

====================================================================================
layer_type: Result
layer_id: 394
input_layer0: layer_id=393: KerasTensor(type_spec=TensorSpec(shape=(1, 25200, 85), dtype=tf.float32, name=None), name='tf.concat_16/concat:0', description="created by layer 'tf.concat_16'")
tf_layers_dict: KerasTensor(type_spec=TensorSpec(shape=(1, 25200, 85), dtype=tf.float32, name=None), name='tf.identity/Identity:0', description="created by layer 'tf.identity'")
====================================================================================
TensorFlow/Keras model building process complete!
saved_model output started ==========================================================
WARNING:absl:Found untraced functions such as _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op while saving (showing 5 of 60). These functions will not be directly callable after loading.
saved_model output complete!
.pb output started ==================================================================
.pb output complete! - saved_model/model_float32.pb
tflite Float32 convertion started ===================================================
WARNING:absl:Found untraced functions such as _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op while saving (showing 5 of 60). These functions will not be directly callable after loading.
Estimated count of arithmetic ops: 5.393 G  ops, equivalently 2.697 G  MACs
tflite Float32 convertion complete! - saved_model/model_float32.tflite
All the conversion process is finished! =============================================

PINTO0309 commented 1 year ago

PINTO0309 commented 1 year ago

Btw, If you want to rename an input/output layer, you can easily rewrite it manually using this tool. https://github.com/PINTO0309/tflite2json2tflite

glenn-jocher commented 1 year ago

@PINTO0309 super interesting, thanks for the detailed response!

The 3 transpose ops are occurring 1 in each of the output grids P3/8, P4/16, P5/32. The permute op is undesirable as it leaves the output not contiguous, so it requires a .contiguous() method afterward.

I'll revisit the idea of removing the permutation completely from YOLOv5. It's a legacy from YOLOv3 onward, but maybe there's a better way I can restructure this module.

PINTO0309 commented 1 year ago

Closed due to no activity for over a week.

PINTO0309 commented 1 year ago

https://github.com/PINTO0309/onnx2tf

Straight conversion sample
YOLOv7-tiny with Post-Process (NMS) ONNX to TFLite Float32

PINTO0309 / openvino2tensorflow