search

zhudongwork / SAM_TensorRT

This is the code to implement Segment Anything (SAM) using TensorRT(C++).
MIT License
26 stars 3 forks source link

TensorRT 8.6 change onnx to trt error #1

Open avBuffer opened 1 year ago

avBuffer commented 1 year ago

1) /usr/local/TensorRT-8.6.1.6//bin/trtexec --onnx=./models/mobile_sam_encoder.onnx --saveEngine=./models/mobile_sam_encoder.engine --memPoolSize=workspace:10240

2) TensorRT 8.6 change onnx to trt error log: &&&& RUNNING TensorRT.trtexec [TensorRT v8601] # /usr/local/TensorRT-8.6.1.6//bin/trtexec --onnx=./models/mobile_sam_encoder.onnx --saveEngine=./models/mobile_sam_encoder.engine --memPoolSize=workspace:10240 [11/03/2023-16:35:20] [I] === Model Options === [11/03/2023-16:35:20] [I] Format: ONNX [11/03/2023-16:35:20] [I] Model: ./models/mobile_sam_encoder.onnx [11/03/2023-16:35:20] [I] Output: [11/03/2023-16:35:20] [I] === Build Options === [11/03/2023-16:35:20] [I] Max batch: explicit batch [11/03/2023-16:35:20] [I] Memory Pools: workspace: 10240 MiB, dlaSRAM: default, dlaLocalDRAM: default, dlaGlobalDRAM: default [11/03/2023-16:35:20] [I] minTiming: 1 [11/03/2023-16:35:20] [I] avgTiming: 8 [11/03/2023-16:35:20] [I] Precision: FP32 [11/03/2023-16:35:20] [I] LayerPrecisions: [11/03/2023-16:35:20] [I] Layer Device Types: [11/03/2023-16:35:20] [I] Calibration: [11/03/2023-16:35:20] [I] Refit: Disabled [11/03/2023-16:35:20] [I] Version Compatible: Disabled [11/03/2023-16:35:20] [I] TensorRT runtime: full [11/03/2023-16:35:20] [I] Lean DLL Path: [11/03/2023-16:35:20] [I] Tempfile Controls: { in_memory: allow, temporary: allow } [11/03/2023-16:35:20] [I] Exclude Lean Runtime: Disabled [11/03/2023-16:35:20] [I] Sparsity: Disabled [11/03/2023-16:35:20] [I] Safe mode: Disabled [11/03/2023-16:35:20] [I] Build DLA standalone loadable: Disabled [11/03/2023-16:35:20] [I] Allow GPU fallback for DLA: Disabled [11/03/2023-16:35:20] [I] DirectIO mode: Disabled [11/03/2023-16:35:20] [I] Restricted mode: Disabled [11/03/2023-16:35:20] [I] Skip inference: Disabled [11/03/2023-16:35:20] [I] Save engine: ./models/mobile_sam_encoder.engine [11/03/2023-16:35:20] [I] Load engine: [11/03/2023-16:35:20] [I] Profiling verbosity: 0 [11/03/2023-16:35:20] [I] Tactic sources: Using default tactic sources [11/03/2023-16:35:20] [I] timingCacheMode: local [11/03/2023-16:35:20] [I] timingCacheFile: [11/03/2023-16:35:20] [I] Heuristic: Disabled [11/03/2023-16:35:20] [I] Preview Features: Use default preview flags. [11/03/2023-16:35:20] [I] MaxAuxStreams: -1 [11/03/2023-16:35:20] [I] BuilderOptimizationLevel: -1 [11/03/2023-16:35:20] [I] Input(s)s format: fp32:CHW [11/03/2023-16:35:20] [I] Output(s)s format: fp32:CHW [11/03/2023-16:35:20] [I] Input build shapes: model [11/03/2023-16:35:20] [I] Input calibration shapes: model [11/03/2023-16:35:20] [I] === System Options === [11/03/2023-16:35:20] [I] Device: 0 [11/03/2023-16:35:20] [I] DLACore: [11/03/2023-16:35:20] [I] Plugins: [11/03/2023-16:35:20] [I] setPluginsToSerialize: [11/03/2023-16:35:20] [I] dynamicPlugins: [11/03/2023-16:35:20] [I] ignoreParsedPluginLibs: 0 [11/03/2023-16:35:20] [I] [11/03/2023-16:35:20] [I] === Inference Options === [11/03/2023-16:35:20] [I] Batch: Explicit [11/03/2023-16:35:20] [I] Input inference shapes: model [11/03/2023-16:35:20] [I] Iterations: 10 [11/03/2023-16:35:20] [I] Duration: 3s (+ 200ms warm up) [11/03/2023-16:35:20] [I] Sleep time: 0ms [11/03/2023-16:35:20] [I] Idle time: 0ms [11/03/2023-16:35:20] [I] Inference Streams: 1 [11/03/2023-16:35:20] [I] ExposeDMA: Disabled [11/03/2023-16:35:20] [I] Data transfers: Enabled [11/03/2023-16:35:20] [I] Spin-wait: Disabled [11/03/2023-16:35:20] [I] Multithreading: Disabled [11/03/2023-16:35:20] [I] CUDA Graph: Disabled [11/03/2023-16:35:20] [I] Separate profiling: Disabled [11/03/2023-16:35:20] [I] Time Deserialize: Disabled [11/03/2023-16:35:20] [I] Time Refit: Disabled [11/03/2023-16:35:20] [I] NVTX verbosity: 0 [11/03/2023-16:35:20] [I] Persistent Cache Ratio: 0 [11/03/2023-16:35:20] [I] Inputs: [11/03/2023-16:35:20] [I] === Reporting Options === [11/03/2023-16:35:20] [I] Verbose: Disabled [11/03/2023-16:35:20] [I] Averages: 10 inferences [11/03/2023-16:35:20] [I] Percentiles: 90,95,99 [11/03/2023-16:35:20] [I] Dump refittable layers:Disabled [11/03/2023-16:35:20] [I] Dump output: Disabled [11/03/2023-16:35:20] [I] Profile: Disabled [11/03/2023-16:35:20] [I] Export timing to JSON file: [11/03/2023-16:35:20] [I] Export output to JSON file: [11/03/2023-16:35:20] [I] Export profile to JSON file: [11/03/2023-16:35:20] [I] [11/03/2023-16:35:20] [I] === Device Information === [11/03/2023-16:35:20] [I] Selected Device: NVIDIA GeForce GTX 1080 [11/03/2023-16:35:20] [I] Compute Capability: 6.1 [11/03/2023-16:35:20] [I] SMs: 20 [11/03/2023-16:35:20] [I] Device Global Memory: 8112 MiB [11/03/2023-16:35:20] [I] Shared Memory per SM: 96 KiB [11/03/2023-16:35:20] [I] Memory Bus Width: 256 bits (ECC disabled) [11/03/2023-16:35:20] [I] Application Compute Clock Rate: 1.7715 GHz [11/03/2023-16:35:20] [I] Application Memory Clock Rate: 5.005 GHz [11/03/2023-16:35:20] [I] [11/03/2023-16:35:20] [I] Note: The application clock rates do not reflect the actual clock rates that the GPU is currently running at. [11/03/2023-16:35:20] [I] [11/03/2023-16:35:20] [I] TensorRT version: 8.6.1 [11/03/2023-16:35:20] [I] Loading standard plugins [11/03/2023-16:35:21] [I] [TRT] [MemUsageChange] Init CUDA: CPU +196, GPU +0, now: CPU 204, GPU 5740 (MiB) [11/03/2023-16:35:22] [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +7, GPU +0, now: CPU 230, GPU 5740 (MiB) [11/03/2023-16:35:22] [I] Start parsing network model. [11/03/2023-16:35:23] [I] [TRT] ---------------------------------------------------------------- [11/03/2023-16:35:23] [I] [TRT] Input filename: ./models/mobile_sam_encoder.onnx [11/03/2023-16:35:23] [I] [TRT] ONNX IR version: 0.0.7 [11/03/2023-16:35:23] [I] [TRT] Opset version: 12 [11/03/2023-16:35:23] [I] [TRT] Producer name: pytorch [11/03/2023-16:35:23] [I] [TRT] Producer version: 1.11.0 [11/03/2023-16:35:23] [I] [TRT] Domain:
[11/03/2023-16:35:23] [I] [TRT] Model version: 0 [11/03/2023-16:35:23] [I] [TRT] Doc string:
[11/03/2023-16:35:23] [I] [TRT] ---------------------------------------------------------------- [11/03/2023-16:35:23] [W] [TRT] onnx2trt_utils.cpp:369: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [11/03/2023-16:35:23] [W] [TRT] onnx2trt_utils.cpp:395: One or more weights outside the range of INT32 was clamped [11/03/2023-16:35:23] [E] Error[4]: [shuffleNode.cpp::symbolicExecute::392] Error Code 4: Internal Error (Reshape_98: IShuffleLayer applied to shape tensor must have 0 or 1 reshape dimensions: dimensions were [-1,2]) [11/03/2023-16:35:23] [E] [TRT] ModelImporter.cpp:773: While parsing node number 109 [Pad -> "onnx::Reshape_442"]: [11/03/2023-16:35:23] [E] [TRT] ModelImporter.cpp:774: --- Begin node --- [11/03/2023-16:35:23] [E] [TRT] ModelImporter.cpp:775: input: "x" input: "onnx::Pad_440" input: "onnx::Pad_441" output: "onnx::Reshape_442" name: "Pad_109" op_type: "Pad" attribute { name: "mode" s: "constant" type: STRING }

[11/03/2023-16:35:23] [E] [TRT] ModelImporter.cpp:776: --- End node --- [11/03/2023-16:35:23] [E] [TRT] ModelImporter.cpp:779: ERROR: ModelImporter.cpp:180 In function parseGraph: [6] Invalid Node - Pad_109 [shuffleNode.cpp::symbolicExecute::392] Error Code 4: Internal Error (Reshape_98: IShuffleLayer applied to shape tensor must have 0 or 1 reshape dimensions: dimensions were [-1,2]) [11/03/2023-16:35:23] [E] Failed to parse onnx file [11/03/2023-16:35:23] [I] Finished parsing network model. Parse time: 0.133705 [11/03/2023-16:35:23] [E] Parsing model failed [11/03/2023-16:35:23] [E] Failed to create engine from model or file. [11/03/2023-16:35:23] [E] Engine set up failed &&&& FAILED TensorRT.trtexec [TensorRT v8601] # /usr/local/TensorRT-8.6.1.6//bin/trtexec --onnx=./models/mobile_sam_encoder.onnx --saveEngine=./models/mobile_sam_encoder.engine --memPoolSize=workspace:10240

zhudongwork commented 12 months ago

Please try TensorRT-8.5

avBuffer commented 12 months ago

1> Changing TensorRT-8.5 is also same error! /usr/local/TensorRT-8.5.3.1/bin/trtexec --onnx=./models/mobile_sam_encoder.onnx --saveEngine=./models/mobile_sam_encoder.engine --memPoolSize=workspace:10240

2> error log: &&&& RUNNING TensorRT.trtexec [TensorRT v8503] # /usr/local/TensorRT-8.5.3.1/bin/trtexec --onnx=./models/mobile_sam_encoder.onnx --saveEngine=./models/mobile_sam_encoder.engine --memPoolSize=workspace:10240 [11/06/2023-16:33:17] [I] === Model Options === [11/06/2023-16:33:17] [I] Format: ONNX [11/06/2023-16:33:17] [I] Model: ./models/mobile_sam_encoder.onnx [11/06/2023-16:33:17] [I] Output: [11/06/2023-16:33:17] [I] === Build Options === [11/06/2023-16:33:17] [I] Max batch: explicit batch [11/06/2023-16:33:17] [I] Memory Pools: workspace: 10240 MiB, dlaSRAM: default, dlaLocalDRAM: default, dlaGlobalDRAM: default [11/06/2023-16:33:17] [I] minTiming: 1 [11/06/2023-16:33:17] [I] avgTiming: 8 [11/06/2023-16:33:17] [I] Precision: FP32 [11/06/2023-16:33:17] [I] LayerPrecisions: [11/06/2023-16:33:17] [I] Calibration: [11/06/2023-16:33:17] [I] Refit: Disabled [11/06/2023-16:33:17] [I] Sparsity: Disabled [11/06/2023-16:33:17] [I] Safe mode: Disabled [11/06/2023-16:33:17] [I] DirectIO mode: Disabled [11/06/2023-16:33:17] [I] Restricted mode: Disabled [11/06/2023-16:33:17] [I] Build only: Disabled [11/06/2023-16:33:17] [I] Save engine: ./models/mobile_sam_encoder.engine [11/06/2023-16:33:17] [I] Load engine: [11/06/2023-16:33:17] [I] Profiling verbosity: 0 [11/06/2023-16:33:17] [I] Tactic sources: Using default tactic sources [11/06/2023-16:33:17] [I] timingCacheMode: local [11/06/2023-16:33:17] [I] timingCacheFile: [11/06/2023-16:33:17] [I] Heuristic: Disabled [11/06/2023-16:33:17] [I] Preview Features: Use default preview flags. [11/06/2023-16:33:17] [I] Input(s)s format: fp32:CHW [11/06/2023-16:33:17] [I] Output(s)s format: fp32:CHW [11/06/2023-16:33:17] [I] Input build shapes: model [11/06/2023-16:33:17] [I] Input calibration shapes: model [11/06/2023-16:33:17] [I] === System Options === [11/06/2023-16:33:17] [I] Device: 0 [11/06/2023-16:33:17] [I] DLACore: [11/06/2023-16:33:17] [I] Plugins: [11/06/2023-16:33:17] [I] === Inference Options === [11/06/2023-16:33:17] [I] Batch: Explicit [11/06/2023-16:33:17] [I] Input inference shapes: model [11/06/2023-16:33:17] [I] Iterations: 10 [11/06/2023-16:33:17] [I] Duration: 3s (+ 200ms warm up) [11/06/2023-16:33:17] [I] Sleep time: 0ms [11/06/2023-16:33:17] [I] Idle time: 0ms [11/06/2023-16:33:17] [I] Streams: 1 [11/06/2023-16:33:17] [I] ExposeDMA: Disabled [11/06/2023-16:33:17] [I] Data transfers: Enabled [11/06/2023-16:33:17] [I] Spin-wait: Disabled [11/06/2023-16:33:17] [I] Multithreading: Disabled [11/06/2023-16:33:17] [I] CUDA Graph: Disabled [11/06/2023-16:33:17] [I] Separate profiling: Disabled [11/06/2023-16:33:17] [I] Time Deserialize: Disabled [11/06/2023-16:33:17] [I] Time Refit: Disabled [11/06/2023-16:33:17] [I] NVTX verbosity: 0 [11/06/2023-16:33:17] [I] Persistent Cache Ratio: 0 [11/06/2023-16:33:17] [I] Inputs: [11/06/2023-16:33:17] [I] === Reporting Options === [11/06/2023-16:33:17] [I] Verbose: Disabled [11/06/2023-16:33:17] [I] Averages: 10 inferences [11/06/2023-16:33:17] [I] Percentiles: 90,95,99 [11/06/2023-16:33:17] [I] Dump refittable layers:Disabled [11/06/2023-16:33:17] [I] Dump output: Disabled [11/06/2023-16:33:17] [I] Profile: Disabled [11/06/2023-16:33:17] [I] Export timing to JSON file: [11/06/2023-16:33:17] [I] Export output to JSON file: [11/06/2023-16:33:17] [I] Export profile to JSON file: [11/06/2023-16:33:17] [I] [11/06/2023-16:33:17] [I] === Device Information === [11/06/2023-16:33:17] [I] Selected Device: NVIDIA GeForce GTX 1080 [11/06/2023-16:33:17] [I] Compute Capability: 6.1 [11/06/2023-16:33:17] [I] SMs: 20 [11/06/2023-16:33:17] [I] Compute Clock Rate: 1.7715 GHz [11/06/2023-16:33:17] [I] Device Global Memory: 8112 MiB [11/06/2023-16:33:17] [I] Shared Memory per SM: 96 KiB [11/06/2023-16:33:17] [I] Memory Bus Width: 256 bits (ECC disabled) [11/06/2023-16:33:17] [I] Memory Clock Rate: 5.005 GHz [11/06/2023-16:33:17] [I] [11/06/2023-16:33:17] [I] TensorRT version: 8.5.3 [11/06/2023-16:33:18] [I] [TRT] [MemUsageChange] Init CUDA: CPU +196, GPU +0, now: CPU 204, GPU 904 (MiB) [11/06/2023-16:33:19] [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +7, GPU +0, now: CPU 230, GPU 895 (MiB) [11/06/2023-16:33:19] [I] Start parsing network model [11/06/2023-16:33:19] [I] [TRT] ---------------------------------------------------------------- [11/06/2023-16:33:19] [I] [TRT] Input filename: ./models/mobile_sam_encoder.onnx [11/06/2023-16:33:19] [I] [TRT] ONNX IR version: 0.0.7 [11/06/2023-16:33:19] [I] [TRT] Opset version: 12 [11/06/2023-16:33:19] [I] [TRT] Producer name: pytorch [11/06/2023-16:33:19] [I] [TRT] Producer version: 1.11.0 [11/06/2023-16:33:19] [I] [TRT] Domain:
[11/06/2023-16:33:19] [I] [TRT] Model version: 0 [11/06/2023-16:33:19] [I] [TRT] Doc string:
[11/06/2023-16:33:19] [I] [TRT] ---------------------------------------------------------------- [11/06/2023-16:33:19] [W] [TRT] onnx2trt_utils.cpp:369: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [11/06/2023-16:33:19] [W] [TRT] onnx2trt_utils.cpp:395: One or more weights outside the range of INT32 was clamped [11/06/2023-16:33:19] [E] Error[4]: [shuffleNode.cpp::symbolicExecute::392] Error Code 4: Internal Error (Reshape_98: IShuffleLayer applied to shape tensor must have 0 or 1 reshape dimensions: dimensions were [-1,2]) [11/06/2023-16:33:19] [E] [TRT] ModelImporter.cpp:773: While parsing node number 109 [Pad -> "onnx::Reshape_442"]: [11/06/2023-16:33:19] [E] [TRT] ModelImporter.cpp:774: --- Begin node --- [11/06/2023-16:33:19] [E] [TRT] ModelImporter.cpp:775: input: "x" input: "onnx::Pad_440" input: "onnx::Pad_441" output: "onnx::Reshape_442" name: "Pad_109" op_type: "Pad" attribute { name: "mode" s: "constant" type: STRING }

[11/06/2023-16:33:19] [E] [TRT] ModelImporter.cpp:776: --- End node --- [11/06/2023-16:33:19] [E] [TRT] ModelImporter.cpp:779: ERROR: ModelImporter.cpp:180 In function parseGraph: [6] Invalid Node - Pad_109 [shuffleNode.cpp::symbolicExecute::392] Error Code 4: Internal Error (Reshape_98: IShuffleLayer applied to shape tensor must have 0 or 1 reshape dimensions: dimensions were [-1,2]) [11/06/2023-16:33:19] [E] Failed to parse onnx file [11/06/2023-16:33:19] [I] Finish parsing network model [11/06/2023-16:33:19] [E] Parsing model failed [11/06/2023-16:33:19] [E] Failed to create engine from model or file. [11/06/2023-16:33:19] [E] Engine set up failed &&&& FAILED TensorRT.trtexec [TensorRT v8503] # /usr/local/TensorRT-8.5.3.1/bin/trtexec --onnx=./models/mobile_sam_encoder.onnx --saveEngine=./models/mobile_sam_encoder.engine --memPoolSize=workspace:10240

zhudongwork commented 11 months ago

This is confusing, I tried it on both TensorRT-8.6.1.6 and 8.5.3.1 and it works fine. This may be an environmental problem or hardware issue with your local tensorrt.

dijiupianhai9 commented 3 months ago

Hello, have you solved this problem？

qingwang-usc commented 2 months ago

Maybe you can try const-folding your ONNX file: https://github.com/NVIDIA/TensorRT/tree/main/tools/Polygraphy/examples/cli/surgeon/02_folding_constants ? @dijiupianhai9 @avBuffer