search

NVIDIA / trt-samples-for-hackathon-cn

Simple samples for TensorRT programming
Apache License 2.0
1.51k stars 342 forks source link

多模型并发 #5

Closed dengxbin closed 2 years ago

dengxbin commented 2 years ago

Hi, 看到官网上说 In general TensorRT objects are not thread-safe. The expected runtime concurrency model is that different threads will operate on different execution contexts. The context contains the state of the network (activation values etc) during execution, so using a context concurrently in different threads results in undefined behavior. 同时进行多个模型的并行推理，使用多个engine进行推理，是不是不安全？

shining365 commented 2 years ago

在不同的线程上使用相同的execution context：错误，文档说这样导致undefined behavior 在不同的线程上使用相同engine的不同的execution context：正确 在不同的线程上使用不同的engine：正确