search

PrinceP / tensorrt-cpp-for-onnx

Tensorrt codebase to inference in c++ for all major neural arch using onnx
GNU General Public License v3.0
16 stars 0 forks source link
rt-detr yolo-world yolo-world-v2 yolov10 yolov5-face yolov5-face-landmark yolov8 yolov8-classification yolov8-detection yolov8-obb yolov8-pose yolov8-seg yolov9

readme

TENSORRT CPP FOR ONNX

Tensorrt codebase in c++ to inference for all major neural arch using onnx and dynamic batching

NVIDIA Driver


wget https://developer.download.nvidia.com/compute/cuda/12.4.0/local_installers/cuda_12.4.0_550.54.14_linux.run

sudo sh cuda_12.4.0_550.54.14_linux.run

Docker


sudo docker build -t trt_24.02_opencv  .

sudo docker run --rm --network="host" -v $(pwd):/app -it --runtime nvidia trt_24.02_opencv bash

Models

YOLO-World

Model Conversion url = https://github.com/AILab-CVC/YOLO-World - Clone the YOLO-World ```bash git clone https://github.com/AILab-CVC/YOLO-World Follow steps for installation https://github.com/AILab-CVC/YOLO-World?tab=readme-ov-file#1-installation Define custom classes(Any name can be defined) echo '[["helmet"], ["head"],["sunglasses"]]' > custom_class.json PYTHONPATH=./ python3 deploy/export_onnx.py configs/pretrain/yolo_world_v2_s_vlpan_bn_2e-3_100e_4x8gpus_obj365v1_goldg_train_lvis_minival.py yolo_world_v2_s_obj365v1_goldg_pretrain-55b943ea.pth --custom-text custom_class.json --opset 12 --without-nms After the fix https://github.com/AILab-CVC/YOLO-World/pull/416 python3 deploy/onnx_demo.py ./work_dirs/yolo_world_v2_s_obj365v1_goldg_pretrain-55b943ea.onnx ~/disk1/uncanny/projects/tensorrt-cpp-for-onnx/data/ custom_class.json --onnx-nms git clone https://github.com/PrinceP/tensorrt-cpp-for-onnx Adjust settings in the ./examples/yolo-world/main.cpp 0.019/*score_threshold*/, 0.7/*iou_threshold*/, 300/*max_detections*/ // Move .onnx file to 'examples/yolo-world' cp cp ./work_dirs/.onnx /app/examples/yolo-world mkdir build cd build cmake .. make -j4 ./yolo-world /app/examples/yolo-world/.onnx /app/data/ // Check the results folder ```
Results **Results [yolo_world_v2_s_obj365v1_goldg_pretrain, Batchsize = 1, Model size = 640x640]**

RT-DETR

Model Conversion url = https://github.com/lyuwenyu/RT-DETR.git - Clone the RT-DETR ```bash git clone https://github.com/lyuwenyu/RT-DETR.git Follow steps from here Version1 https://github.com/lyuwenyu/RT-DETR/blob/main/rtdetr_pytorch/README.md#todo Version2 https://github.com/lyuwenyu/RT-DETR/tree/main/rtdetrv2_pytorch#quick-start Any of above can run. git clone https://github.com/PrinceP/tensorrt-cpp-for-onnx // Move .onnx file to 'examples/rt-detr' cp .onnx /app/examples/rt-detr mkdir build cd build cmake .. make -j4 ./rt-detr /app/examples/rt-detr/.onnx /app/data/ // Check the results folder ```
Results **Results [RT-DETRv2-S, Batchsize = 2, Model size = 640x640]**

YOLOV10

Model Conversion url = https://github.com/THU-MIG/yolov10 - Clone the yolov10 ```bash git clone https://github.com/THU-MIG/yolov10 yolo export model=yolov10n/s/m/b/l/x.pt format=onnx opset=13 simplify dynamic git clone https://github.com/PrinceP/tensorrt-cpp-for-onnx // Move .onnx file to 'examples/yolov10' cp .onnx /app/examples/yolov10 mkdir build cd build cmake .. make -j4 ./yolov10 /app/examples/yolov10/.onnx /app/data/ // Check the results folder ```
Results **Results [YOLOv10m, Batchsize = 2, Model size = 640x640]**

YOLOV9

Model Conversion url = https://github.com/WongKinYiu/yolov9.git commit 380284cb66817e9ffa30a80cad4c1b110897b2fb - Clone the yolov9 ```bash git clone https://github.com/WongKinYiu/yolov9 python3 export.py --weights .pt --include onnx_end2end git clone https://github.com/PrinceP/tensorrt-cpp-for-onnx // Move -end2end.onnx file to 'examples/yolov9' cp -end2end.onnx /app/examples/yolov9 mkdir build cd build cmake .. make -j4 ./yolov9 /app/examples/yolov9/-end2end.onnx /app/data/ // Check the results folder ```
Results **Results [YOLOv9-C, Batchsize = 2, Model size = 640x640]**

YOLOV8-Detect

Model Conversion url = https://github.com/ultralytics/ultralytics ultralytics==8.1.24 - Install ultralytics package in python ```python from ultralytics import YOLO model = YOLO('yolov8s.pt') model.export(format='onnx', dynamic=True) ``` ```bash git clone https://github.com/PrinceP/tensorrt-cpp-for-onnx // Move .onnx file to 'examples/yolov8' cp .onnx /app/examples/yolov8 mkdir build cd build cmake .. make -j4 ./yolov8-detect /app/examples/yolov8/.onnx /app/data/ // Check the results folder ```
Results **Results [YOLOv8s, Batchsize = 2, Model size = 640x640]**

YOLOV8-Segment

Model Conversion url = https://github.com/ultralytics/ultralytics ultralytics==8.1.24 - Install ultralytics package in python ```python from ultralytics import YOLO # Load a model model = YOLO('yolov8n-seg.pt') # Export the model model.export(format='onnx', dynamic=True) ``` ```bash git clone https://github.com/PrinceP/tensorrt-cpp-for-onnx // Move .onnx file to 'examples/yolov8' cp .onnx /app/examples/yolov8 mkdir build cd build cmake .. make -j4 ./yolov8-segment /app/examples/yolov8/.onnx /app/data/ // Check the results folder ```
Results **Results [YOLOv8n, Batchsize = 2, Model size = 640x640]**

YOLOV8-Pose

Model Conversion url = https://github.com/ultralytics/ultralytics ultralytics==8.1.24 - Install ultralytics package in python ```python from ultralytics import YOLO # Load a model model = YOLO('yolov8n-pose.pt') # Export the model model.export(format='onnx', dynamic=True) ``` ```bash git clone https://github.com/PrinceP/tensorrt-cpp-for-onnx // Move .onnx file to 'examples/yolov8' cp .onnx /app/examples/yolov8 mkdir build cd build cmake .. make -j4 ./yolov8-pose /app/examples/yolov8/.onnx /app/data/ // Check the results folder ```
Results **Results [YOLOv8n, Batchsize = 2, Model size = 640x640]**

YOLOV8-OBB

Model Conversion url = https://github.com/ultralytics/ultralytics ultralytics==8.1.24 - Install ultralytics package in python ```python from ultralytics import YOLO # Load a model model = YOLO('yolov8n-obb.pt') # Export the model model.export(format='onnx', dynamic=True) ``` ```bash git clone https://github.com/PrinceP/tensorrt-cpp-for-onnx // Move .onnx file to 'examples/yolov8' cp .onnx /app/examples/yolov8 mkdir build cd build cmake .. make -j4 ./yolov8-obb /app/examples/yolov8/.onnx /app/data/obb/ // Check the results folder ```
Results **Results [YOLOv8n, Batchsize = 2, Model size = 640x640]**

YOLOV8-Classify

Model Conversion url = https://github.com/ultralytics/ultralytics ultralytics==8.1.24 - Install ultralytics package in python ```python from ultralytics import YOLO # Load a model model = YOLO('yolov8n-cls.pt') # Export the model model.export(format='onnx', dynamic=True) ``` ```bash git clone https://github.com/PrinceP/tensorrt-cpp-for-onnx // Move .onnx file to 'examples/yolov8' cp .onnx /app/examples/yolov8 mkdir build cd build cmake .. make -j4 ./yolov8-classify /app/examples/yolov8/.onnx /app/data/classify/ // Check the results folder ```
Results **Results [YOLOv8n, Batchsize = 2, Model size = 224x224]**

YOLOV5-Face

Model Conversion url = https://github.com/deepcam-cn/yolov5-face - Install onnx==1.16.2, tqdm, thop, seaborn, torch==1.9, torchvision==0.10 - Changes for dynamic shape ```txt diff --git a/export.py b/export.py index 1aa7dae..2502e23 100644 --- a/export.py +++ b/export.py @@ -75,14 +75,22 @@ if __name__ == '__main__': print('\nStarting ONNX export with onnx %s...' % onnx.__version__) f = opt.weights.replace('.pt', '.onnx') # filename model.fuse() # only for ONNX - input_names=['input'] - output_names=['output'] - torch.onnx.export(model, img, f, verbose=False, opset_version=12, + + # Dynamic batching support + input_names = [ "input" ] + output_names = [ "output" ] + dynamic_axes={'input' : {0 : 'batch_size'}, 'output' : {0 : 'batch_size'}} + dynamic_axes['input'][2] = 'height' + dynamic_axes['input'][3] = 'width' + + torch.onnx.export(model, img, f, verbose=True, opset_version=12, input_names=input_names, output_names=output_names, + # dynamic_axes = dynamic_axes dynamic_axes = {'input': {0: 'batch'}, 'output': {0: 'batch'} } if opt.dynamic else None) + # ) # Checks onnx_model = onnx.load(f) # load onnx model ``` - Convert pytorch to onnx ```bash python3 export.py --weights ./yolov5s-face.pt --dynamic ``` ```bash git clone https://github.com/PrinceP/tensorrt-cpp-for-onnx // Move .onnx file to 'examples/yolov5-face' cp .onnx /app/examples/yolov5-face mkdir build cd build cmake .. make -j4 ./yolov5-face /app/examples/yolov5-face/.onnx /app/data/yolov5-face/ // Check the results folder ```
Results **Results [YOLOv5s-face, Batchsize = 2, Model size = 640x640]**

NOTES

Issues - Dynamic batching is supported. The batchsize and image sizes can be updated in the codebase. - Dynamic batch issue resolved for yolov10: https://github.com/THU-MIG/yolov10/issues/27 - Dynamic batch not present for Yolo-World - If size issue happens while building. Increase the workspaceSize ```bash Internal error: plugin node /end2end/EfficientNMS_TRT requires XXX bytes of scratch space, but only XXX is available. Try increasing the workspace size with IBuilderConfig::setMemoryPoolLimit(). ``` ```cpp config->setMaxWorkspaceSize(1U << 26) //The current memory is 2^26 bytes ```