search

iwatake2222 / play_with_tensorrt

Sample projects for TensorRT in C++
Apache License 2.0
187 stars 34 forks source link

Calibration failed with virtualMemoryBuffer.cpp::nvinfer1::StdVirtualMemoryBufferImpl #27

Closed ghost closed 2 years ago

ghost commented 2 years ago

Environment (Hardware)

Cuda 11.3 Cudnn 8.4 TensorRT 8.4

Project Name

Int8 calibration

Issue Details

When i set all the important parameters for Calibration like following:

define CAL_DIR "ppmSamples"

define CAL_LIST_FILE "list.txt"

define CAL_BATCH_SIZE 10

define CAL_NB_BATCHES 2430

define CAL_IMAGE_C 3

define CAL_IMAGE_H 256

define CAL_IMAGE_W 256

/ 0 ~ 1.0 / // #define CAL_SCALE (1.0 / 255.0) // #define CAL_BIAS (0.0) / -2.25 ~ 2.25 /

define CAL_SCALE (1.0 / (255.0 * 0.225))

define CAL_BIAS (0.45 / 0.225)

and use all the preparation steps for the calibration it doesnt work. First of all i used 2430 images for calibration which i put into the directly ppmSamples and the list is also filled with the image names without the file extension. When compiling the programm i get following console output and i cant figure out what im doing wrong or if there maybe is a bug

Error Log

[InferenceHelper][119] Use TensorRT
[08/27/2022-19:04:43] [I] [TRT] [MemUsageChange] Init CUDA: CPU +444, GPU +0, now: CPU 4908, GPU 1226 (MiB)
[08/27/2022-19:04:43] [I] [TRT] [MemUsageChange] Init CUDA: CPU +0, GPU +0, now: CPU 4952, GPU 1226 (MiB)
[08/27/2022-19:04:45] [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +357, GPU +104, now: CPU 5441, GPU 1330 (MiB)
[08/27/2022-19:04:45] [I] [TRT] ----------------------------------------------------------------
[08/27/2022-19:04:45] [I] [TRT] Input filename:   resource/model/yolov7_256x256.onnx
[08/27/2022-19:04:45] [I] [TRT] ONNX IR version:  0.0.6
[08/27/2022-19:04:45] [I] [TRT] Opset version:    11
[08/27/2022-19:04:45] [I] [TRT] Producer name:    pytorch
[08/27/2022-19:04:45] [I] [TRT] Producer version: 1.11.0
[08/27/2022-19:04:45] [I] [TRT] Domain:
[08/27/2022-19:04:45] [I] [TRT] Model version:    0
[08/27/2022-19:04:45] [I] [TRT] Doc string:
[08/27/2022-19:04:45] [I] [TRT] ----------------------------------------------------------------
[08/27/2022-19:04:45] [W] [TRT] onnx2trt_utils.cpp:369: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/27/2022-19:04:45] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +796, GPU +314, now: CPU 6294, GPU 1652 (MiB)
[08/27/2022-19:04:45] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +139, GPU +58, now: CPU 6433, GPU 1710 (MiB)
[08/27/2022-19:04:45] [I] [TRT] Timing cache disabled. Turning it on will improve builder speed.
[08/27/2022-19:04:47] [I] [TRT] Detected 1 inputs and 4 output network tensors.
[08/27/2022-19:04:47] [I] [TRT] Total Host Persistent Memory: 26592
[08/27/2022-19:04:47] [I] [TRT] Total Device Persistent Memory: 0
[08/27/2022-19:04:47] [I] [TRT] Total Scratch Memory: 0
[08/27/2022-19:04:47] [I] [TRT] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 0 MiB, GPU 140 MiB
[08/27/2022-19:04:47] [I] [TRT] [BlockAssignment] Algorithm ShiftNTopDown took 127.918ms to assign 10 blocks to 336 nodes requiring 29229056 bytes.
[08/27/2022-19:04:47] [I] [TRT] Total Activation Memory: 29229056
[08/27/2022-19:04:47] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 7814, GPU 2392 (MiB)
[08/27/2022-19:04:47] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +0, GPU +10, now: CPU 7812, GPU 2376 (MiB)
[08/27/2022-19:04:47] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +27, now: CPU 0, GPU 167 (MiB)
[08/27/2022-19:04:47] [I] [TRT] Starting Calibration.
[08/27/2022-19:04:47] [I] Batch #0
.ppm27/2022-19:04:47] [I] Calibrating with file 0
.ppm27/2022-19:04:47] [I] Calibrating with file 1
.ppm27/2022-19:04:47] [I] Calibrating with file 2
.ppm27/2022-19:04:47] [I] Calibrating with file 3
.ppm27/2022-19:04:47] [I] Calibrating with file 4
.ppm27/2022-19:04:47] [I] Calibrating with file 5
.ppm27/2022-19:04:47] [I] Calibrating with file 6
.ppm27/2022-19:04:47] [I] Calibrating with file 7
.ppm27/2022-19:04:47] [I] Calibrating with file 8
.ppm27/2022-19:04:47] [I] Calibrating with file 9
.ppm in data directories:
        ppmSamples
&&&& FAILED
[08/27/2022-19:04:47] [E] [TRT] 1: [virtualMemoryBuffer.cpp::nvinfer1::StdVirtualMemoryBufferImpl::~StdVirtualMemoryBufferImpl::104] Error Code 1: Cuda Runtime (driver shutting down)