search

cyrusbehr / tensorrt-cpp-api

TensorRT C++ API Tutorial
MIT License
577 stars 72 forks source link

multi-stream inference #28

Closed lizexu123 closed 10 months ago

lizexu123 commented 10 months ago

Hello, does enqueueV3 support multi-stream inference? Is this part included in your code?

cyrusbehr commented 10 months ago

According to this it should be possible, but you'd need to create a IExecutionContext for each stream. In the case of my code, you'd basically just create an instance of the Engine class for each model you want to run in parallel. However, do realize that you will have GPU contention if running models in parallel and not in serial.

lizexu123 commented 10 months ago

据此，这应该是可能的，但是您需要 IExecutionContext为每个流创建一个。就我的代码而言，您基本上只需Engine为要并行运行的每个模型创建该类的实例。但是，请注意，如果并行而不是串行运行模型，将会出现 GPU 争用。

Can multiple cuda streams be provided to achieve multi-stream execution?