Set specific Cuda stream ID in Cpp implimentation code.

wang-xinyu / tensorrtx

Implementation of popular deep learning networks with TensorRT network definition API

MIT License

6.95k stars 1.77k forks source link

Set specific Cuda stream ID in Cpp implimentation code. #1565

Open phamthanhdat270198 opened 2 months ago

phamthanhdat270198 commented 2 months ago

Env

Agx Orin 64GB.
Ubuntu20.04
Cuda version 11.4
TensorRT version 8.5.2

Question

Hi, I want to set specific cudastream ID on Cpp code yolo like --streams=0 or --streams=1, ... in trtexec Many thanks!!!!!

wang-xinyu commented 2 months ago

You need to adapt the inference code, create multiple streams and pass to infer() function.

phamthanhdat270198 commented 2 months ago

When I create multiple streams It just decrease a little bit infer time. So Does it working in parallel. give me an example about multi streams and multi contexts. Many thanks!!! Here is my impliment multi streams ![Uploading image.png…]()