Low latency than expected

[paduck86]

Hello Mr.Jung, Tanks to you, I was able to test faster rcnn with tensorrt.

But, the latency is lower than I expected on my machine. The response time is as follows,

• architecture : faster rcnn resnet101
• machine : V100
• version : tensorflow 1.10, tensorrt4-ga, cuda9.0, cudnn7.0
• tensorflow native : 0.17s per image
• tensrrt fp 32 : 0.52s per image
• tensrrt fp 16 : 0.51s per image
• tensrrt int 8 : 0.51s per image

In addition, the memory usage is not much different.

My code is as follows,

trt_graph = trt.create_inference_graph( input_graph_def=frozen_graph, outputs=output_names, max_batch_size=1, max_workspace_size_bytes=1 << 25, precision_mode='INT8' #'FP32' / 'FP16', minimum_segment_size=50 )

Did I do something wrong? I would really appreciate it if you answer for me.

[jkjung-avt]

I put my latest code in my own GitHub repository: https://github.com/jkjung-avt/tf_trt_models. Feel free to check it out.

Meanwhile, I'm not completely sure what your question is. Are you trying to say that TF-TRT fails to optimize 'faster_rcnn_resnet101' at all?