Is it possible to use TensorRT?

[manbehindthemadness]

Is it possible to use this model over torch2trt? https://github.com/NVIDIA-AI-IOT/torch2trt How would one go about accomplishing that? It seems that the matting network exists externally from the NN.

[manbehindthemadness]

In an alternative (failed) attempt I tried to pull the model from onnx into TensorRT:

onnx2trt rvm_resnet50_fp16.onnx -o rvm_resnet50_fp16.trt -d 16 -v

Input filename: rvm_resnet50_fp16.onnx ONNX IR version: 0.0.6 Opset version: 12 Producer name: pytorch Producer version: 1.9 Domain: Model version: 0 Doc string:

[2023-01-31 19:52:57 INFO] [MemUsageChange] Init CUDA: CPU +220, GPU +0, now: CPU 294, GPU 9414 (MiB) [2023-01-31 19:53:00 INFO] [MemUsageChange] Init builder kernel library: CPU +351, GPU +660, now: CPU 664, GPU 10000 (MiB) Parsing model [2023-01-31 19:53:00 ERROR] [graphShapeAnalyzer.cpp::analyzeShapes::1237] Error Code 4: Internal Error (downsample_ratio: network input that is shape tensor must have type Int32) While parsing node number 3 [Resize -> "399"]: --- Begin node --- input: "src" input: "396" input: "398" output: "399" name: "Resize_3" op_type: "Resize" attribute { name: "coordinate_transformation_mode" s: "pytorch_half_pixel" type: STRING } attribute { name: "cubic_coeff_a" f: -0.75 type: FLOAT } attribute { name: "mode" s: "linear" type: STRING } attribute { name: "nearest_mode" s: "floor" type: STRING }

--- End node --- ERROR: ModelImporter.cpp:180 In function parseGraph: [6] Invalid Node - Resize_3 [graphShapeAnalyzer.cpp::analyzeShapes::1237] Error Code 4: Internal Error (downsample_ratio: network input that is shape tensor must have type Int32)

[manbehindthemadness]

This most definitely relates: https://github.com/PeterL1n/RobustVideoMatting/issues/202 However, the problem addressed in this issue was in the process of converting into onnx, not from.

[manbehindthemadness]

So the solution is a combination of the two: 1: make the alterations in https://github.com/PeterL1n/RobustVideoMatting/issues/202 2: python export_onnx.py --model-variant resnet50 --precision float16 --model-refiner deep_guided_filter --checkpoint rvm_resnet50.pth --opset 12 --device cuda --output onnx_f16_resnet50_op12_deep.onnx 3: onnx2trt onnx_f16_resnet50_op12_deep.onnx -o rvm_resnet50_fp16deep.trt -d 16 -v paydirt

This is a workaround, but the desired result seems to have been achieved (it's still compiling the model)

[sunfan1997]

这个链接里有比较详细的实现：https://www.cnblogs.com/gocodinginmyway/p/15795663.html，用C++进行推理的，我试了一下mobilenetv3的模型，没有onnx快。

[manbehindthemadness]

@sunfan1997 This is wonderfully helpful, thanks so much!

[manbehindthemadness]

@sunfan1997 I am having trouble with your example, everything works perfect until trtexec fails to build reporting a core dump with no specific error:

[02/10/2023-18:09:40] [V] [TRT] Parsing node: Resize_3 [Resize]
[02/10/2023-18:09:40] [V] [TRT] Searching for input: src
[02/10/2023-18:09:40] [V] [TRT] Searching for input: 386
[02/10/2023-18:09:40] [V] [TRT] Searching for input: 388
[02/10/2023-18:09:40] [V] [TRT] Resize_3 [Resize] inputs: [src -> (1, 3, 1440, 2560)[FLOAT]], [386 -> (0)[FLOAT]], [388 -> (4)[FLOAT]],
[02/10/2023-18:09:40] [V] [TRT] Registering layer: Resize_3 for ONNX node: Resize_3
./convert.sh: line 30: 68615 Segmentation fault      (core dumped)

My conversion and build build script: convert.sh.txt

Full error: error.txt

Any idea?

[sunfan1997]

@sunfan1997 I am having trouble with your example, everything works perfect until trtexec fails to build reporting a core dump with no specific error:

[02/10/2023-18:09:40] [V] [TRT] Parsing node: Resize_3 [Resize]
[02/10/2023-18:09:40] [V] [TRT] Searching for input: src
[02/10/2023-18:09:40] [V] [TRT] Searching for input: 386
[02/10/2023-18:09:40] [V] [TRT] Searching for input: 388
[02/10/2023-18:09:40] [V] [TRT] Resize_3 [Resize] inputs: [src -> (1, 3, 1440, 2560)[FLOAT]], [386 -> (0)[FLOAT]], [388 -> (4)[FLOAT]],
[02/10/2023-18:09:40] [V] [TRT] Registering layer: Resize_3 for ONNX node: Resize_3
./convert.sh: line 30: 68615 Segmentation fault      (core dumped)

My conversion and build build script: convert.sh.txt

Full error: error.txt

Any idea?

Sorry, i have no idea. I fixed the ‘downsample_ratio’ parameter using the method in 202，then use the onnx file convert to trt file.

[manbehindthemadness]

No problem, thanks so much, I'll speak with the Nvidia guys about the problem and post any answers here for reference.

[manbehindthemadness]

I realized that this problem wasn't quite the issue I believed, The first segmentation fault was due to a cross linking problem that was resolved by a simple recompile of the TensorRT library. A second segmentation fault would then occur in the post-deserialization process: https://forums.developer.nvidia.com/t/orin-trtexec-results-in-segmentation-fault/242724/3 Despite this error, the engine file was indeed created and it does function per the example code in the write-up you supplied. This appears to be an issue in the post-compile operations of TensorRT and not with the example found here: https://www.cnblogs.com/gocodinginmyway/p/15795663.html