TensorRT and inference benchmarking

[changtimwu]

official resnet50

• https://forums.developer.nvidia.com/t/jetson-agx-xavier-deep-learning-inference-benchmarks/67844
• https://developer.nvidia.com/embedded/jetson-agx-xavier-dl-inference-benchmarks

yolov3

• https://forums.developer.nvidia.com/t/yolov3-fps-on-tensorrt/109982/21
• https://forums.developer.nvidia.com/t/how-to-use-gpu-2-dla-can-be-100fps-for-yolov3-on-xavier/113205

[changtimwu]

Benchmark result

fp16

bin/trtexec --avgRuns=100 --deploy=resnet50.prototxt --fp16 --batch=8 --iterations=10000 --output=prob --useSpinWait

t4bm_fp16_resnet50.txt

int8

bin/trtexec --avgRuns=100 --deploy=resnet50.prototxt --int8 --batch=8 --iterations=10000 --output=prob --useSpinWait

t4bm_int8_resnet50.txt

[changtimwu]

not sure if qps is equivlent to fps. The internal variable is latencyThroughtput. q means query. https://github.com/NVIDIA/TensorRT/blob/master/samples/common/sampleReporting.cpp#L153

[changtimwu]

worth to read the How Do I Measure Performance? section. The overall system performance can be measured by the latency and throughput of the entire processing pipeline. Because the pre and post-processing steps depend so strongly on the particular application, in this section, we will mostly consider the latency and throughput of the network inference, excluding the data pre and post-processing overhead.

• trtexec complete usage: https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/trtexec

[changtimwu]

check tensorrt examples.

dpkg -L libnvinfer-samples 

• to build automl/efficientdet

  pip install -U Cython 'git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI'

checkpoint to PB

# Step 1: export model
!python model_inspect.py --runmode=saved_model \
  --model_name=efficientdet-d0 --ckpt_path=efficientdet-d0 \
  --saved_model_dir=/tmp/saved_model

root@55b6f8ad558e:~/workspace/automl/efficientdet# ls -ahR /tmp/saved_model/
/tmp/saved_model/:
.  ..  saved_model.pb  variables

/tmp/saved_model/variables:
.  ..  variables.data-00000-of-00001  variables.index

[changtimwu]

let's evaluate fps in this way

https://github.com/jkjung-avt/tensorrt_demos#ssd

[changtimwu]

Pretrained model inference with TensorRT

• via onnix https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch/issues/29#issuecomment-613775950
• directly https://github.com/google/automl/issues/77
• https://docs.nvidia.com/deeplearning/frameworks/tf-trt-user-guide/index.html

[changtimwu]

https://zhuanlan.zhihu.com/p/88318324

[changtimwu]

tensort acceleration

GTC 2020

• https://developer.download.nvidia.com/video/gputechconf/gtc/2020/presentations/s22676-developing-real-time-neural-networks-for-jetson.pdf

Blog

• https://devblogs.nvidia.com/speeding-up-deep-learning-inference-using-tensorrt/
• https://devblogs.nvidia.com/speeding-up-deep-learning-inference-using-tensorflow-onnx-and-tensorrt/