The issue of exporting the onnx model and conducting inference

[XianYang2547]

When I added the statement model.export() in predict.py, I obtained the ONNX model and found some errors in displaying and saving videos during the inference process. I adjusted it myself，It's the error with im0, which is a list that stores the detection image and two segmented images separately，By rendering it once, it can be displayed and saved correctly。Thank you for your work

[libaos]

When I added the statement model.export() in predict.py, I obtained the ONNX model and found some errors in displaying and saving videos during the inference process. I adjusted it myself，It's the error with im0, which is a list that stores the detection image and two segmented images separately，By rendering it once, it can be displayed and saved correctly。Thank you for your work

hey，man!how to solve it. I meet the same problem!

[XianYang2547]

When I added the statement model.export() in predict.py, I obtained the ONNX model and found some errors in displaying and saving videos during the inference process. I adjusted it myself，It's the error with im0, which is a list that stores the detection image and two segmented images separately，By rendering it once, it can be displayed and saved correctly。Thank you for your work

hey，man!how to solve it. I meet the same problem!

export onnx model ？or display and save ？

[libaos]

When I added the statement model.export() in predict.py, I obtained the ONNX model and found some errors in displaying and saving videos during the inference process. I adjusted it myself，It's the error with im0, which is a list that stores the detection image and two segmented images separately，By rendering it once, it can be displayed and saved correctly。Thank you for your work

hey，man!how to solve it. I meet the same problem!

export onnx model ？or display and save ？

export onnx model

[1623021453]

Hi, how did you export onnx model? When I use model.export(), I meet the errors: Traceback (most recent call last): File "predict.py", line 20, in model.export(format='onnx') File "/media/user/新加卷/zxc_ubuntu/code/YOLOv8-multi-task/ultralytics/yolo/engine/model.py", line 356, in export return Exporter(overrides=args, _callbacks=self.callbacks)(model=self.model) File "/home/user/anaconda3/envs/YOLOv8-multi-task/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(*args, **kwargs) File "/media/user/新加卷/zxc_ubuntu/code/YOLOv8-multi-task/ultralytics/yolo/engine/exporter.py", line 167, in call self.imgsz = check_imgsz(self.args.imgsz, stride=model.stride, min_dim=2) # check image size File "/media/user/新加卷/zxc_ubuntu/code/YOLOv8-multi-task/ultralytics/yolo/utils/checks.py", line 59, in check_imgsz stride = int(stride.max() if isinstance(stride, torch.Tensor) else stride) TypeError: int() argument must be a string, a bytes-like object or a number, not 'list'

[XianYang2547]

Hi, how did you export onnx model? When I use model.export(), I meet the errors: Traceback (most recent call last): File "predict.py", line 20, in model.export(format='onnx') File "/media/user/新加卷/zxc_ubuntu/code/YOLOv8-multi-task/ultralytics/yolo/engine/model.py", line 356, in export return Exporter(overrides=args, _callbacks=self.callbacks)(model=self.model) File "/home/user/anaconda3/envs/YOLOv8-multi-task/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(*args, kwargs) File "/media/user/新加卷/zxc_ubuntu/code/YOLOv8-multi-task/ultralytics/yolo/engine/exporter.py", line 167, in call** self.imgsz = check_imgsz(self.args.imgsz, stride=model.stride, min_dim=2) # check image size File "/media/user/新加卷/zxc_ubuntu/code/YOLOv8-multi-task/ultralytics/yolo/utils/checks.py", line 59, in check_imgsz stride = int(stride.max() if isinstance(stride, torch.Tensor) else stride) TypeError: int() argument must be a string, a bytes-like object or a number, not 'list'

In Exporter class,change the stride to 32,debug model,step by step

[adeirman46]

Hi, how did you export onnx model? When I use model.export(), I meet the errors: Traceback (most recent call last): File "predict.py", line 20, in model.export(format='onnx') File "/media/user/新加卷/zxc_ubuntu/code/YOLOv8-multi-task/ultralytics/yolo/engine/model.py", line 356, in export return Exporter(overrides=args, _callbacks=self.callbacks)(model=self.model) File "/home/user/anaconda3/envs/YOLOv8-multi-task/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(*args, kwargs) File "/media/user/新加卷/zxc_ubuntu/code/YOLOv8-multi-task/ultralytics/yolo/engine/exporter.py", line 167, in call** self.imgsz = check_imgsz(self.args.imgsz, stride=model.stride, min_dim=2) # check image size File "/media/user/新加卷/zxc_ubuntu/code/YOLOv8-multi-task/ultralytics/yolo/utils/checks.py", line 59, in check_imgsz stride = int(stride.max() if isinstance(stride, torch.Tensor) else stride) TypeError: int() argument must be a string, a bytes-like object or a number, not 'list'

In Exporter class,change the stride to 32,debug model,step by step

Could you please elaborate on your steps? I followed all the steps you mentioned but still encountered an error.

[XianYang2547]

ultralytics.zip run python 1.py

[adeirman46]

ultralytics.zip run python 1.py

Thanks, I appreciate your work. Finally, I got the .engine and .onnx files, but I encountered an error when I tried to predict using them. The CUDA won't work (succeeded running on CPU) when I used .onnx, and when I used .engine, I got this error:

onnx_exporter/ultralytics/yolo/utils/ops.py", line 191, in non_max_suppression xc = prediction[:, 4:mi].amax(1) > conf_thres # candidates IndexError: amax(): Expected reduction dim 1 to have non-zero size.

Have you encountered these problems? If yes, would you like to share the solutions?

Anyone who needs the .onnx and .engine files for the baseline model v4s.pt and v4n.pt can download those models from this link: https://drive.google.com/drive/folders/1vUVsc41FC4Jo0g7NYq3_m0cdjUNou0LK?usp=sharing

[JiayuanWang-JW]

ultralytics.zip run python 1.py

Hi @XianYang2547,

Thanks for your excellent work, would you spend some time creating a pull request for this function? It is really useful for our repo. It will help more potential researchers or users.

Thanks

[XianYang2547]

In my onnx model ，I removed a model output.Give me your contact information such as wechat.

发自我的iPhone

在 2024年7月17日，08:39，Jiayuan Wang @.***> 写道：

 ultralytics.zip run python 1.py

Hi @XianYang2547,

Thanks for your excellent work, would you spend some time creating a pull request for this function? It is really useful for our repo. It will help more potential researchers or users.

Thanks

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

[XianYang2547]

ultralytics.zip run python 1.py

Thanks, I appreciate your work. Finally, I got the .engine and .onnx files, but I encountered an error when I tried to predict using them. The CUDA won't work (succeeded running on CPU) when I used .onnx, and when I used .engine, I got this error:

onnx_exporter/ultralytics/yolo/utils/ops.py", line 191, in non_max_suppression xc = prediction[:, 4:mi].amax(1) > conf_thres # candidates IndexError: amax(): Expected reduction dim 1 to have non-zero size.

Have you encountered these problems? If yes, would you like to share the solutions?

Anyone who needs the .onnx and .engine files for the baseline model v4s.pt and v4n.pt can download those models from this link: https://drive.google.com/drive/folders/1vUVsc41FC4Jo0g7NYq3_m0cdjUNou0LK?usp=sharing

First,In my onnx model ，I removed a model output.Give me your contact information such as wechat.Finally,you'd better use it such as image pre -->model infer-->postprocess> by yourself,refuse to using ultralytics YOLO,some error will happen in it.

[loganwu0526]

@XianYang2547 do you have the TensorRT inference script for your model?

[adeirman46]

@XianYang2547 do you have the TensorRT inference script for your model?

If you have it and are willing to give it, it would be a huge help for my team. @XianYang2547

[XianYang2547]

@XianYang2547 do you have the TensorRT inference script for your model?

If you have it and are willing to give it, it would be a huge help for my team. @XianYang2547

I'll look for it tomorrow._

[XianYang2547]

@XianYang2547 do you have the TensorRT inference script for your model?

If you have it and are willing to give it, it would be a huge help for my team. @XianYang2547

I'll look for it tomorrow._

Trt_infers.zip

I find it...

[loganwu0526]

@XianYang2547 how do you convert .onnx to .engine? I use your 1.py to convert my .pt to .onnx, and I try three approaches to convert .onnx to .engine

1. your get_trt.py, success
2. your get_TRT_model.py, fail, error message Traceback (most recent call last): File "get_TRT_model.py", line 27, in f.write(serialized_engine) TypeError: a bytes-like object is required, not 'NoneType'
3. /usr/src/tensorrt/bin/trtexec, success

but when I use you inference script by using the engines obtained from approach 1 and 3, I get the same error message for both cases, the error message is given as follows

Traceback (most recent call last): File "TRT_infer_image_video.py", line 77, in Models = TRT_infer() File "TRT_infer_image_video.py", line 21, in init self.generate() File "TRT_infer_image_video.py", line 31, in generate size = engine.get_tensor_shape(binding) AttributeError: 'tensorrt.tensorrt.ICudaEngine' object has no attribute 'get_tensor_shape'

do you have any idea how to correct it? thanks

[XianYang2547]

I don't know how to troubleshoot this, because it works fine for me.(I need detection and lane seg) first,unzip ultralytics.zip,use pycharm open ultralytics dir,run 1.py,My onnx only has 2 outputs, and I run get_trt.py to get the engine tensorrt=8.6.1. finally,If it is convenient for you, I can use remote control software to check your.

[loganwu0526]

@XianYang2547 can you mention what files you have modified? Because I have two segmentations, drivable area and lane, so maybe I do not modify some files for my case.

[1623021453]

@XianYang2547 I run the engine model on the NVIDIA GeForce RTX 4080,the inference time is 0.2s/frame, why is it so slow?

[XianYang2547]

@XianYang2547 can you mention what files you have modified? Because I have two segmentations, drivable area and lane, so maybe I do not modify some files for my case.

It's been so long, I can't remember. two segmentation...，you can premeditate using yolo v8 rather than multi-task

[XianYang2547]

@XianYang2547 I run the engine model on the NVIDIA GeForce RTX 4080,the inference time is 0.2s/frame, why is it so slow?

You can write your own inference script, my script is just for testing. To improve the speed, you can start from these aspects: use pytorch's tensor, use cupy, rebuild the trt inference engine such as memory copy release, or use execute_async_v3. Then, when you get the output of the model, use efficient post-processing methods to parse the output to get the results you want. Such as TRT_infer-->detect_image-->show method, modify it. Visualization generally consumes performance. If you don't calculate the time taken by show, one frame takes about 4ms.

[loganwu0526]

@XianYang2547 I do not quite understand the function postprocess_seg you defined in the my_utils.py, can you explain it more? on the other hand, if there are two segmentation parts- drivable area and lane, how should I modify it? thanks btw, I use your inference script, and find the overhead is quite large, if it is normal? or if there is any parameter I can try to adjust to reduce the overhead? I deploy my model in jetson nano.

[loganwu0526]

@XianYang2547 moreover, the performance when I deploy the model in jetson nano and use your inference script is much worse than the case in PC. But it is not case for the model with only OD.

[XianYang2547]

@XianYang2547我不太理解你在 my_utils.py 中定义的函数 postprocess_seg，你能进一步解释一下吗？另一方面，如果有两个分割部分 - 可行驶区域和车道，我应该如何修改它？顺便说一句，谢谢 ，我使用了你的推理脚本，发现开销很大，这是否正常？或者是否有任何参数我可以尝试调整以减少开销？我在 jetson nano 中部署了我的模型。

In the output of the model, 0 is detection, 1 is segmentation, and 2 is also segmentation. I distinguish them just to show them with different colors. If you have 2 segmentation branches, postprocess_seg should be called twice. If there is no detection branch, postprocess_det is not needed. It is normal that the overhead is large. I did not optimize the post-processing part. I simply superimposed the segmentation map on the original image to see the effect. Visualization is not important. What is important is to get the required information from the model output.

[loganwu0526]

@XianYang2547 do you have any idea how to optimize the segment part to reduce the latency (especially the overhead)? I need to deply a lightweight a multitask model with 4 classes object detection and two segmentations (drivable area and lane). On the other hand, when I apply your inference script on the jetson nano, the performance drops significantly, some classes are almost undetected. Is it normal orI may miss something? thanks