create_execution_context() doesn't release GIL (TensorRT 8.2)

NVIDIA / TensorRT

NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.

https://developer.nvidia.com/tensorrt

Apache License 2.0

10.88k stars 2.14k forks source link

create_execution_context() doesn't release GIL (TensorRT 8.2) #2945

Closed Fred-Erik closed 1 year ago

Fred-Erik commented 1 year ago

Description

create_execution_context() doesn't release the Python GIL, see the pybind code here. Would it be possible to add a py::call_guard<py::gil_scoped_release>{} there? On a Jetson TX2, for my model, it takes about 7.5 seconds to create the execution context. This makes my semi-realtime program unresponsive during this period, and it would be nicer if the Python interpreter kept functioning.

zerollzeng commented 1 year ago

@pranavm-nvidia ^ ^

pranavm-nvidia commented 1 year ago

I'll fix this on our internal development branch

ttyio commented 1 year ago

This is fixed in 9.0, see https://github.com/NVIDIA/TensorRT/blob/release/9.0/python/src/infer/pyCore.cpp#L1041 closing and thanks all!