search

isarsoft / yolov4-triton-tensorrt

This repository deploys YOLOv4 as an optimized TensorRT engine to Triton Inference Server
http://www.isarsoft.com
Other
276 stars 63 forks source link

How can I generate batch=5 engine? #66

Closed miseon119 closed 2 years ago

miseon119 commented 2 years ago

I changed main.cpp file, and built it. Screenshot from 2022-04-20 14-46-14

But I got this after running 

./trtexec --loadEngine=/yolov4-triton-tensorrt/build/yolov4.engine --plugins=/yolov4-triton-tensorrt/build/liblayerplugin.so
&&&& RUNNING TensorRT.trtexec [TensorRT v8003] # ./trtexec --loadEngine=/yolov4-triton-tensorrt/build/yolov4.engine --plugins=/yolov4-triton-tensorrt/build/liblayerplugin.so
[04/20/2022-05:39:12] [I] === Model Options ===
[04/20/2022-05:39:12] [I] Format: *
[04/20/2022-05:39:12] [I] Model: 
[04/20/2022-05:39:12] [I] Output:
[04/20/2022-05:39:12] [I] === Build Options ===
[04/20/2022-05:39:12] [I] Max batch: 1
[04/20/2022-05:39:12] [I] Workspace: 16 MiB
[04/20/2022-05:39:12] [I] minTiming: 1
[04/20/2022-05:39:12] [I] avgTiming: 8
[04/20/2022-05:39:12] [I] Precision: FP32
[04/20/2022-05:39:12] [I] Calibration: 
[04/20/2022-05:39:12] [I] Refit: Disabled
[04/20/2022-05:39:12] [I] Sparsity: Disabled
[04/20/2022-05:39:12] [I] Safe mode: Disabled
[04/20/2022-05:39:12] [I] Restricted mode: Disabled
[04/20/2022-05:39:12] [I] Save engine: 
[04/20/2022-05:39:12] [I] Load engine: /yolov4-triton-tensorrt/build/yolov4.engine
[04/20/2022-05:39:12] [I] NVTX verbosity: 0
[04/20/2022-05:39:12] [I] Tactic sources: Using default tactic sources
[04/20/2022-05:39:12] [I] timingCacheMode: local
[04/20/2022-05:39:12] [I] timingCacheFile: 
[04/20/2022-05:39:12] [I] Input(s)s format: fp32:CHW
[04/20/2022-05:39:12] [I] Output(s)s format: fp32:CHW
[04/20/2022-05:39:12] [I] Input build shapes: model
[04/20/2022-05:39:12] [I] Input calibration shapes: model
[04/20/2022-05:39:12] [I] === System Options ===
[04/20/2022-05:39:12] [I] Device: 0
[04/20/2022-05:39:12] [I] DLACore: 
[04/20/2022-05:39:12] [I] Plugins: /yolov4-triton-tensorrt/build/liblayerplugin.so
[04/20/2022-05:39:12] [I] === Inference Options ===

Any suggestion?

philipp-schmidt commented 2 years ago

What trtexec reports is not about your engine but what trtexec was set so build if you woild use it to create an engine. Check the flags of trtexec for more info.

The batch size is probably visible in your output dimension tensor or at a different part of trtexec. Sou can also load it into triton and check there.