YoloV7 TRT export from onnx fails

[JanKrcek]

Hello,

I am using the default pretrained yolov7.pt model and followed steps on this repo to export into onnx file (using your export fork) and then attempt to run the compiled exporter code in yolov7. Unfortunately that results in following:

[07/15/2022-15:47:55] [I] [TRT] [MemUsageChange] Init CUDA: CPU +329, GPU +0, now: CPU 335, GPU 1625 (MiB)
[07/15/2022-15:47:55] [I] [TRT] ----------------------------------------------------------------
[07/15/2022-15:47:55] [I] [TRT] Input filename:   ../yolov7.onnx
[07/15/2022-15:47:55] [I] [TRT] ONNX IR version:  0.0.6
[07/15/2022-15:47:55] [I] [TRT] Opset version:    12
[07/15/2022-15:47:55] [I] [TRT] Producer name:    pytorch
[07/15/2022-15:47:55] [I] [TRT] Producer version: 1.10
[07/15/2022-15:47:55] [I] [TRT] Domain:           
[07/15/2022-15:47:55] [I] [TRT] Model version:    0
[07/15/2022-15:47:55] [I] [TRT] Doc string:       
[07/15/2022-15:47:55] [I] [TRT] ----------------------------------------------------------------
[07/15/2022-15:47:56] [W] [TRT] onnx2trt_utils.cpp:364: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[07/15/2022-15:47:56] [W] [TRT] onnx2trt_utils.cpp:390: One or more weights outside the range of INT32 was clamped
[07/15/2022-15:47:56] [I] [TRT] No importer registered for op: ScatterND. Attempting to import as plugin.
[07/15/2022-15:47:56] [I] [TRT] Searching for plugin: ScatterND, plugin_version: 1, plugin_namespace: 
[07/15/2022-15:47:56] [I] [TRT] Successfully created plugin: ScatterND
[07/15/2022-15:47:56] [E] [TRT] [graphShapeAnalyzer.cpp::throwIfError::1306] Error Code 9: Internal Error (Mul_291: broadcast dimensions must be conformable
)
[07/15/2022-15:47:56] [E] [TRT] ModelImporter.cpp:720: While parsing node number 291 [Mul -> "518"]:
[07/15/2022-15:47:56] [E] [TRT] ModelImporter.cpp:721: --- Begin node ---
[07/15/2022-15:47:56] [E] [TRT] ModelImporter.cpp:722: input: "515"
input: "998"
output: "518"
name: "Mul_291"
op_type: "Mul"

I am using TenorRT 8.0.1 + Cuda 11.4 locally. But I get the same error in the PyTorch docker used for training/inference as well. Docker: nvcr.io/nvidia/pytorch:21.08-py3

Any ideas what am I doing wrong?

[linghu8812]

Hello,

I am using the default pretrained yolov7.pt model and followed steps on this repo to export into onnx file (using your export fork) and then attempt to run the compiled exporter code in yolov7. Unfortunately that results in following:

[07/15/2022-15:47:55] [I] [TRT] [MemUsageChange] Init CUDA: CPU +329, GPU +0, now: CPU 335, GPU 1625 (MiB)
[07/15/2022-15:47:55] [I] [TRT] ----------------------------------------------------------------
[07/15/2022-15:47:55] [I] [TRT] Input filename:   ../yolov7.onnx
[07/15/2022-15:47:55] [I] [TRT] ONNX IR version:  0.0.6
[07/15/2022-15:47:55] [I] [TRT] Opset version:    12
[07/15/2022-15:47:55] [I] [TRT] Producer name:    pytorch
[07/15/2022-15:47:55] [I] [TRT] Producer version: 1.10
[07/15/2022-15:47:55] [I] [TRT] Domain:           
[07/15/2022-15:47:55] [I] [TRT] Model version:    0
[07/15/2022-15:47:55] [I] [TRT] Doc string:       
[07/15/2022-15:47:55] [I] [TRT] ----------------------------------------------------------------
[07/15/2022-15:47:56] [W] [TRT] onnx2trt_utils.cpp:364: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[07/15/2022-15:47:56] [W] [TRT] onnx2trt_utils.cpp:390: One or more weights outside the range of INT32 was clamped
[07/15/2022-15:47:56] [I] [TRT] No importer registered for op: ScatterND. Attempting to import as plugin.
[07/15/2022-15:47:56] [I] [TRT] Searching for plugin: ScatterND, plugin_version: 1, plugin_namespace: 
[07/15/2022-15:47:56] [I] [TRT] Successfully created plugin: ScatterND
[07/15/2022-15:47:56] [E] [TRT] [graphShapeAnalyzer.cpp::throwIfError::1306] Error Code 9: Internal Error (Mul_291: broadcast dimensions must be conformable
)
[07/15/2022-15:47:56] [E] [TRT] ModelImporter.cpp:720: While parsing node number 291 [Mul -> "518"]:
[07/15/2022-15:47:56] [E] [TRT] ModelImporter.cpp:721: --- Begin node ---
[07/15/2022-15:47:56] [E] [TRT] ModelImporter.cpp:722: input: "515"
input: "998"
output: "518"
name: "Mul_291"
op_type: "Mul"

I am using TenorRT 8.0.1 + Cuda 11.4 locally. But I get the same error in the PyTorch docker used for training/inference as well. Docker: nvcr.io/nvidia/pytorch:21.08-py3

Any ideas what am I doing wrong?

first, please try TensorRT 7.1.3.4 https://github.com/linghu8812/tensorrt_inference/blob/master/INSTALL.md#tensort-7134, or check the anchor shape of the onnx model

[jia0511]

HELLO，

I tried TensorRT 7.1.3.4,but still have problem!

`lov7/build$ ./yolov7_trt ../../../configs/yolov7/config.yaml ../../../samples/ [07/30/2022-12:33:48] [I] [TRT] [MemUsageChange] Init CUDA: CPU +332, GPU +0, now: CPU 338, GPU 640 (MiB) [07/30/2022-12:33:48] [I] [TRT] ---------------------------------------------------------------- [07/30/2022-12:33:48] [I] [TRT] Input filename: ../yolov7.onnx [07/30/2022-12:33:48] [I] [TRT] ONNX IR version: 0.0.7 [07/30/2022-12:33:48] [I] [TRT] Opset version: 12 [07/30/2022-12:33:48] [I] [TRT] Producer name: pytorch [07/30/2022-12:33:48] [I] [TRT] Producer version: 1.10 [07/30/2022-12:33:48] [I] [TRT] Domain:
[07/30/2022-12:33:48] [I] [TRT] Model version: 0 [07/30/2022-12:33:48] [I] [TRT] Doc string:
[07/30/2022-12:33:48] [I] [TRT] ---------------------------------------------------------------- [07/30/2022-12:33:48] [W] [TRT] onnx2trt_utils.cpp:364: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [07/30/2022-12:33:48] [E] [TRT] [graphShapeAnalyzer.cpp::throwIfError::1306] Error Code 9: Internal Error (Mul_322: broadcast dimensions must be conformable ) [07/30/2022-12:33:48] [E] [TRT] ModelImporter.cpp:720: While parsing node number 322 [Mul -> "528"]: [07/30/2022-12:33:48] [E] [TRT] ModelImporter.cpp:721: --- Begin node --- [07/30/2022-12:33:48] [E] [TRT] ModelImporter.cpp:722: input: "525" input: "657" output: "528" name: "Mul_322" op_type: "Mul"

[07/30/2022-12:33:48] [E] [TRT] ModelImporter.cpp:723: --- End node --- [07/30/2022-12:33:48] [E] [TRT] ModelImporter.cpp:726: ERROR: ModelImporter.cpp:179 In function parseGraph: [6] Invalid Node - Mul_322 [graphShapeAnalyzer.cpp::throwIfError::1306] Error Code 9: Internal Error (Mul_322: broadcast dimensions must be conformable ) [07/30/2022-12:33:48] [E] Failure while parsing ONNX file start building engine [07/30/2022-12:33:48] [E] [TRT] 4: [network.cpp::validate::2411] Error Code 4: Internal Error (Network must have at least one output) build engine done yolov7_trt: /home/jack/data1/gitcode/tensorRT/tensorrt_inference/code/src/model.cpp:46: void Model::OnnxToTRTModel(): Assertion engine' failed. 已放弃 (核心已转储)

[JanKrcek]

I suggest trying to do the Onnx export again, using the official YoloV7 repo. This bug was recently fixed there.

Edit: Also using newest pytorch 1.12.0 + Cuda 11.3/4 and corresponding TRT seems to help a lot with Onnx related issues.

[thuongnct]

Hi @jia0511, did you solve the issue? If it worked, could you show the combination of cuda, torch, tensorrt, and onnx?