PINTO0309 / PINTO_model_zoo

A repository for storing models that have been inter-converted between various frameworks. Supported frameworks are TensorFlow, PyTorch, ONNX, OpenVINO, TFJS, TFTRT, TensorFlowLite (Float32/16/INT8), EdgeTPU, CoreML.
https://qiita.com/PINTO
MIT License
3.49k stars 566 forks source link

Error while create postprocess for onnx yolov7 #358

Closed HoangTienDuc closed 12 months ago

HoangTienDuc commented 1 year ago

Issue Type

Bug

OS

Ubuntu

OS architecture

x86_64

Programming Language

Python

Framework

TensorFlow, TensorFlowLite

Model name and Weights/Checkpoints URL

Model name: yolov7 https://github.com/PINTO0309/PINTO_model_zoo/tree/18e75913cb97eefa2d8fe9f87b86841407b4c826/307_YOLOv7/post_process_gen_tools

Description

Hi PINTO, your work is awesome. I try to replicate your work to post-process my custon yolov7 model. First, i tried to follow the instruction from your model zoo yolov7 post-process without any change, but with your original postprocess docker (1.1.5, 1.1.15), i cannot "update_model_dims.update_inputs_outputs_dims". Tried with diffirent dockers, tf2onnx, tensorflow version, but it is still not work. Can you pls give me some advice? Thanks

Relevant Log Output

2023-08-19 04:45:55,743 - INFO - Using tensorflow=2.13.0, onnx=1.14.0, tf2onnx=1.12.1/b6d590
2023-08-19 04:45:55,743 - INFO - Using opset <onnx, 11>
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
2023-08-19 04:45:55,764 - WARNING - Error loading model into tflite interpreter: _get_tensor_details() missing 1 required positional argument: 'subgraph_index'
2023-08-19 04:45:55,770 - INFO - Optimizing ONNX model
2023-08-19 04:45:55,784 - INFO - After optimization: Cast -1 (1->0), Const -2 (6->4), Identity -1 (1->0)
2023-08-19 04:45:55,785 - INFO - 
2023-08-19 04:45:55,785 - INFO - Successfully converted TensorFlow model saved_model_postprocess/nms_score_gather_nd.tflite to ONNX
2023-08-19 04:45:55,785 - INFO - Model inputs: ['serving_default_input_1:0', 'serving_default_input_2:0']
2023-08-19 04:45:55,785 - INFO - Model outputs: ['PartitionedCall:0']
2023-08-19 04:45:55,785 - INFO - ONNX model is saved at nms_score_gather_nd.onnx
INFO: Finish!
INFO: Finish!
INFO: Finish!
INFO: Finish!
OUTPUT_MODEL_PATH:  nms_score_gather_nd.onnx
model.graph.input:  [name: "serving_default_input_1:0"
type {
  tensor_type {
    elem_type: 1
    shape {
      dim {
        dim_value: 1
      }
      dim {
        dim_value: 80
      }
      dim {
        dim_value: 5040
      }
    }
  }
}
, name: "serving_default_input_2:0"
type {
  tensor_type {
    elem_type: 7
    shape {
      dim {
        dim_param: "unk__10"
      }
      dim {
        dim_value: 3
      }
    }
  }
}
]
input_dicts:  {'gn_scores': ['1', '80', '5040'], 'gn_selected_indices': ['N', '3']}
output_dicts:  {'final_scores': ['N', '1']}
input_dims:  {'gn_scores': ['1', '80', '5040'], 'gn_selected_indices': ['N', '3']}
Traceback (most recent call last):
  File "make_input_output_shape_update.py", line 75, in <module>
    updated_model = update_model_dims.update_inputs_outputs_dims(
  File "/home/user/.local/lib/python3.8/site-packages/onnx/tools/update_model_dims.py", line 88, in update_inputs_outputs_dims
    input_dim_arr = input_dims[input_name]
KeyError: 'serving_default_input_1:0'

URL or source code for simple inference testing code

No response

HoangTienDuc commented 1 year ago

Hi Katsuya, Thanks for your reply. I only run the sample "convert_script.txt" from 307_yolov7/post_process_gen_tools

Reproduce:

OPSET=11
BATCHES=1
BOXES=5040
CLASSES=80

################################################### Boxes + Scores
python make_boxes_scores.py -o ${OPSET} -b ${BATCHES} -x ${BOXES} -c
${CLASSES}
python make_cxcywh_y1x1y2x2.py -o ${OPSET} -b ${BATCHES} -x ${BOXES}

snc4onnx \
--input_onnx_file_paths boxes_scores_${BOXES}.onnx
cxcywh_y1x1y2x2_${BOXES}.onnx \
--srcop_destop boxes_cxcywh cxcywh \
--output_onnx_file_path boxes_y1x1y2x2_scores_${BOXES}.onnx

################################################### NonMaxSuppression
sog4onnx \
--op_type Constant \
--opset ${OPSET} \
--op_name max_output_boxes_per_class_const \
--output_variables max_output_boxes_per_class int64 [1] \
--attributes value int64 [20] \
--output_onnx_file_path Constant_max_output_boxes_per_class.onnx

sog4onnx \
--op_type Constant \
--opset ${OPSET} \
--op_name iou_threshold_const \
--output_variables iou_threshold float32 [1] \
--attributes value float32 [0.5] \
--output_onnx_file_path Constant_iou_threshold.onnx

sog4onnx \
--op_type Constant \
--opset ${OPSET} \
--op_name score_threshold_const \
--output_variables score_threshold float32 [1] \
--attributes value float32 [0.4] \
--output_onnx_file_path Constant_score_threshold.onnx

OP=NonMaxSuppression
LOWEROP=${OP,,}
sog4onnx \
--op_type ${OP} \
--opset ${OPSET} \
--op_name ${LOWEROP}${OPSET} \
--input_variables boxes_var float32 [${BATCHES},${BOXES},4] \
--input_variables scores_var float32 [${BATCHES},${CLASSES},${BOXES}] \
--input_variables max_output_boxes_per_class_var int64 [1] \
--input_variables iou_threshold_var float32 [1] \
--input_variables score_threshold_var float32 [1] \
--output_variables selected_indices int64 [\'N\',3] \
--attributes center_point_box int64 0 \
--output_onnx_file_path ${OP}${OPSET}.onnx

snc4onnx \
--input_onnx_file_paths Constant_max_output_boxes_per_class.onnx
NonMaxSuppression11.onnx \
--srcop_destop max_output_boxes_per_class max_output_boxes_per_class_var \
--output_onnx_file_path NonMaxSuppression${OPSET}.onnx

snc4onnx \
--input_onnx_file_paths Constant_iou_threshold.onnx
NonMaxSuppression${OPSET}.onnx \
--srcop_destop iou_threshold iou_threshold_var \
--output_onnx_file_path NonMaxSuppression${OPSET}.onnx

snc4onnx \
--input_onnx_file_paths Constant_score_threshold.onnx
NonMaxSuppression${OPSET}.onnx \
--srcop_destop score_threshold score_threshold_var \
--output_onnx_file_path NonMaxSuppression${OPSET}.onnx

soc4onnx \
--input_onnx_file_path NonMaxSuppression${OPSET}.onnx \
--output_onnx_file_path NonMaxSuppression${OPSET}.onnx \
--opset ${OPSET}

################################################### Boxes + Scores +
NonMaxSuppression
snc4onnx \
--input_onnx_file_paths boxes_y1x1y2x2_scores_${BOXES}.onnx
NonMaxSuppression${OPSET}.onnx \
--srcop_destop scores scores_var y1x1y2x2 boxes_var \
--output_onnx_file_path nms_yolov7_${BOXES}.onnx

################################################### Myriad workaround Mul
OP=Mul
LOWEROP=${OP,,}
OPSET=${OPSET}
sog4onnx \
--op_type ${OP} \
--opset ${OPSET} \
--op_name ${LOWEROP}${OPSET} \
--input_variables workaround_mul_a int64 [\'N\',3] \
--input_variables workaround_mul_b int64 [1] \
--output_variables workaround_mul_out int64 [\'N\',3] \
--output_onnx_file_path ${OP}${OPSET}_workaround.onnx

############ Myriad workaround Constant
sog4onnx \
--op_type Constant \
--opset ${OPSET} \
--op_name workaround_mul_const_op \
--output_variables workaround_mul_const int64 [1] \
--attributes value int64 [1] \
--output_onnx_file_path Constant_workaround_mul.onnx

############ Myriad workaround Mul + Myriad workaround Constant
snc4onnx \
--input_onnx_file_paths Constant_workaround_mul.onnx
Mul${OPSET}_workaround.onnx \
--srcop_destop workaround_mul_const workaround_mul_b \
--output_onnx_file_path Mul${OPSET}_workaround.onnx

################################################### NonMaxSuppression +
Myriad workaround Mul
snc4onnx \
--input_onnx_file_paths nms_yolov7_${BOXES}.onnx
Mul${OPSET}_workaround.onnx \
--srcop_destop selected_indices workaround_mul_a \
--output_onnx_file_path nms_yolov7_${BOXES}.onnx

################################################### Score GatherND
python make_score_gather_nd.py -b ${BATCHES} -x ${BOXES} -c ${CLASSES}

python -m tf2onnx.convert \
--opset ${OPSET} \
--tflite saved_model_postprocess/nms_score_gather_nd.tflite \
--output nms_score_gather_nd.onnx

sor4onnx \
--input_onnx_file_path nms_score_gather_nd.onnx \
--old_new ":0" "" \
--output_onnx_file_path nms_score_gather_nd.onnx

sor4onnx \
--input_onnx_file_path nms_score_gather_nd.onnx \
--old_new "serving_default_input_1" "gn_scores" \
--output_onnx_file_path nms_score_gather_nd.onnx \
--mode inputs

sor4onnx \
--input_onnx_file_path nms_score_gather_nd.onnx \
--old_new "serving_default_input_2" "gn_selected_indices" \
--output_onnx_file_path nms_score_gather_nd.onnx \
--mode inputs

sor4onnx \
--input_onnx_file_path nms_score_gather_nd.onnx \
--old_new "PartitionedCall" "final_scores" \
--output_onnx_file_path nms_score_gather_nd.onnx \
--mode outputs

python make_input_output_shape_update.py \
--input_onnx_file_path nms_score_gather_nd.onnx \
--output_onnx_file_path nms_score_gather_nd.onnx \
--input_names gn_scores \
--input_names gn_selected_indices \
--input_shapes ${BATCHES} ${CLASSES} ${BOXES} \
--input_shapes N 3 \
--output_names final_scores \
--output_shapes N 1

On Sun, Aug 20, 2023 at 12:28 PM Katsuya Hyodo @.***> wrote:

KeyError: 'serving_default_input_1:0' is displayed. That's all there is to it. How would I know what model you are using?

— Reply to this email directly, view it on GitHub https://github.com/PINTO0309/PINTO_model_zoo/issues/358#issuecomment-1685181593, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMWDRMRIZ42A7M5WQBQLSO3XWGOADANCNFSM6AAAAAA3WKOJDY . You are receiving this because you authored the thread.Message ID: @.***>

PINTO0309 commented 1 year ago

An error occurred during the conversion because the specification of sor4onnx, a homebrew tool, was changed. Committed to fix convert_script.txt.

This script is still buggy as it has never been maintained since it was first created on a trial basis.

The final output is Y1X1Y2X2.

image

PINTO0309 commented 1 year ago

In addition, the binding bug of final_box_nums was adjusted.

image

PINTO0309 commented 1 year ago

x1y1x2y2 -> y1x1y2x2 image

PINTO0309 commented 1 year ago

In the Concat just before NMS, we need to generate both the Y1X1Y2X2 tensor for NMS and the X1Y1X2Y2 for the final output, but it is a year old script and I have not modified it because it is too much trouble.

PINTO0309 commented 1 year ago

One more point. If you don't add Sqrt here, the score should be oddly small, less than half.

image

HoangTienDuc commented 1 year ago

Thank you, I will check it and let you know later

On Sun, Aug 20, 2023 at 1:26 PM Katsuya Hyodo @.***> wrote:

One more point. If you don't add Sqrt here, the score should be oddly small, less than half.

[image: image] https://user-images.githubusercontent.com/33194443/261835292-0d170c79-da66-4e52-9253-3b878c23d273.png

— Reply to this email directly, view it on GitHub https://github.com/PINTO0309/PINTO_model_zoo/issues/358#issuecomment-1685199748, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMWDRMWJF3HNYUUATQZKWCTXWGU27ANCNFSM6AAAAAA3WKOJDY . You are receiving this because you authored the thread.Message ID: @.***>

PINTO0309 commented 1 year ago

All corrected. image

HoangTienDuc commented 1 year ago

Awesome, it works. Thank you very much. Your work is incredible. I hope in the future you will have more projects to help in the community.

On Sun, Aug 20, 2023 at 5:34 PM Katsuya Hyodo @.***> wrote:

All corrected. [image: image] https://user-images.githubusercontent.com/33194443/261846206-ea8ceef2-1f6a-46e3-8986-c5024b1b3315.png

— Reply to this email directly, view it on GitHub https://github.com/PINTO0309/PINTO_model_zoo/issues/358#issuecomment-1685249783, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMWDRMT3DO67D675M2UX27DXWHR4VANCNFSM6AAAAAA3WKOJDY . You are receiving this because you authored the thread.Message ID: @.***>

HoangTienDuc commented 12 months ago

Duplicate result from an image to other images

Hi @PINTO0309 , I convert the model from onnx to tensorrt, then use tensorrt code to make prediction.

For example:


image_path /data/data_video/24/pgie/images/0_10-8-2023-17-46_11_109380.png, batch_pred: [[array([149,  66, 207, 156], dtype=int32), array([175, 290, 213, 320], dtype=int32), array([110, 291, 141, 319], dtype=int32)], [0.9104384, 0.91341376, 0.894918], [0, 0, 0]]

**************************************************

image_path /data/data_video/24/pgie/images/0_10-8-2023-17-46_17_72020.png, batch_pred: [[array([154, 136, 201, 189], dtype=int32), array([175, 290, 213, 320], dtype=int32), array([110, 291, 141, 319], dtype=int32)], [0.98416233, 0.91341376, 0.894918], [0, 0, 0]]

In the two images above, both detected two objects with a confidence score of 0.91341376, 0.894918 in the image. However, in reality, the two images above have no such object, this result is the result of a few previous images.

My sample code is saved at gitlab, please take a look

PINTO0309 commented 12 months ago

I don't understand what you are talking about.

I will not look into it unless you provide me with test images and models.

HoangTienDuc commented 12 months ago

Sorry for that. Please allow me to send the model and data to you privately

Model: yolov7, num_classes = 5, input_size 320x256 best_672.pt https://drive.google.com/file/d/1py84NNymqSFHKb2QOFJQcXMuAJ1M1mer/view?usp=drive_web Step to convert model:

  1. From WongKinYiu/yolov7 export .pt model to onnx: python3 export.py --weights best_672.pt --simplify --grid --img-size 256 320 --dynamic-batch
  2. Then use PINTO yolov7 post processing tool to get yolov7_post.onnx model
  3. Convert yolov7_post.onnx model to tensorrt

/usr/src/tensorrt/bin/trtexec --onnx=src_model.onnx --saveEngine=dst_model.trt --fp16 --explicitBatch --minShapes=images:1x3x256x320 --maxShapes=images:4x3x256x320 --optShapes=images:2x3x256x320 --workspace=2048

Data is attached in this email

On Thu, Sep 7, 2023 at 11:49 PM Katsuya Hyodo @.***> wrote:

I don't understand what you are talking about.

I will not look into it unless you provide me with test images and models.

— Reply to this email directly, view it on GitHub https://github.com/PINTO0309/PINTO_model_zoo/issues/358#issuecomment-1710484243, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMWDRMSHWFLB3WQIWFGDEBLXZH3KNANCNFSM6AAAAAA3WKOJDY . You are receiving this because you modified the open/close state.Message ID: @.***>

PINTO0309 commented 12 months ago

It is no surprise that merging single-batch post-processing into a multi-batch main model will corrupt the output.

image