Open bobbilichandu opened 3 years ago
I am using ScaledYolov4 with tensorrt version 7.2.2 and T4 GPUs. When I am increasing batchsize, inference time is increasing linearly. Is this behaviour expected? FOR BATCH SIZE - 1:
Inference take: 48.5283 ms. Inference take: 48.518 ms. Inference take: 40.1897 ms. Inference take: 40.0713 ms. Inference take: 38.54 ms. Inference take: 38.7829 ms. Inference take: 38.6083 ms. Inference take: 38.6635 ms. Inference take: 38.1827 ms. Inference take: 38.1016 ms
FOR BATCH SIZE - 2:
Inference take: 76.3045 ms. Inference take: 74.9346 ms. Inference take: 73.3341 ms. Inference take: 73.9554 ms. Inference take: 73.4185 ms. Inference take: 75.4546 ms. Inference take: 77.7809 ms. Inference take: 78.3289 ms. Inference take: 79.5533 ms. Inference take: 79.0556 ms. Inference take: 79.2939 ms. Inference take: 77.214 ms.
FOR BATCH SIZE - 4:
Inference take: 158.327 ms. Inference take: 157.001 ms. Inference take: 157.107 ms. Inference take: 154.237 ms. Inference take: 155.899 ms. Inference take: 157.408 ms. Inference take: 155.758 ms. Inference take: 155.906 ms.
I expected batch size not to have this proportionality. Can anything done to improve the inference time using batching? TIY.
I am using ScaledYolov4 with tensorrt version 7.2.2 and T4 GPUs. When I am increasing batchsize, inference time is increasing linearly. Is this behaviour expected? FOR BATCH SIZE - 1:
FOR BATCH SIZE - 2:
FOR BATCH SIZE - 4:
I expected batch size not to have this proportionality. Can anything done to improve the inference time using batching? TIY.