rmccorm4
commented
1 year ago Hi @jike-algorithm-zhangxiao ,
On trtexec side:
it just cost 1+ms to infer
I believe you're only looking at the GPU compute time from the trtexec results but not the other latencies involved in the end-to-end process. I think this line may be a bit more of an apples-to-apples comparison:
[12/15/2022-12:04:27] [I] Average on 10 runs - GPU latency: 1.98892 ms - Host latency: 6.94839 ms (enqueue 1.55723 ms)which looks to be about ~9-11 ms (not sure if enqueue is included in Host or not).
On tritonserver side:
It cost 13+ ms to exec a decoder inference
t = time.time()
d_rep = d_req.exec()
logging.error((time.time() - t) * 1000)Off the top of my head I don't know exactly what else might be involved in the BLS pipeline other than model execution. I'm not sure if any extra copies/gathers would be involved if keeping memory in-GPU. @Tabrizian could you comment?
zxOnVacation
Tabrizian
yuanquderzi
nlin5
I wanna use triton+tensorrt to deploy whisper, a transformer-like arc asr model.
I wanna use kv-cache to accelerate inference speed, so i use python-backend and dlpack to do this when i build a decoder to tensorrt, use trtexec to measure the decoder performance as below
it just cost 1+ms to infer But when i use python-backend to infer as below:
It cost 13+ ms to exec a decoder inference I think dlpack is zero-copy, so what's the extra cost for python-backend inference? and how to fix my python-backend code to achieve the ~1ms speed inference?