[Performance] TensorRT builder instantiation is very slow

[gedoensmax]

Describe the issue

Acquiring a TRT builder object is very slow and is actually unnecessary work if an engine is already present on disk. But even if the engine would have to be build the builder instance could be reused if the full graph is applicable for TRT. Not sure about using the same builder instance for multiple compilations though.

To reproduce

I ran a lama like model using onnxruntime_perf_test. The command line used is:

I believe this issue reproduces with any model I also tried with a simple resnet where the impact vs actual load time is even worse. I have not tested if this issue is Windows specific, but I would assume that maybe it is less impactful on Linux but should also have significant performance impact. Below is a screenshot of Superluminal (profiler) with the highlighted sections showing the instantiation call for trt_builder.

Urgency

No response

Platform

Windows

OS Version

11

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

61ddb898eb1e1c921680af849dd7ae15e855937a

ONNX Runtime API

C++

Architecture

X64

Execution Provider

TensorRT

Execution Provider Library Version

TRT 8.6, CUDA 12

Model File

No response

Is this a quantized model?

Unknown

[gedoensmax]

@chilo-ms

[chilo-ms]

I had a draft PR to address this issue. Feel free to provide feedback there. What I did is to make the TRT builder only being instantiated once, currently we can't skip the very first builder instantization but can significantly reduce the time for subsequent access to the builder instance.

Profiling 1: The current TRT EP will spend around 80% of time instantiating TRT builder in TRT EP's Compile() even though the builder instance already being instantiated before in GetCapability()

Profiling 2: As you can see, the time spend in TRT builder in TRT EP's Compile() to get the builder instance is negligible since we get the reference of the builder instance directly.

Will try to profile/verify the case that the models (FasterRCNN, MaskRCNN...) will be partitioned into multiple TRT EP subgraphs and CUDA EP kernels. I assume there should be more time reduction since more builder instance needed for that case.

[gedoensmax]

Ha, another superluminal user :D The improvements look great.

[github-actions[bot]]

This issue has been automatically marked as stale due to inactivity and will be closed in 7 days if no further activity occurs. If further support is needed, please provide an update and/or more details.

[gedoensmax]

This improved a lot due to #18100 and #18217 will enable completely bypassing the builder. I am fine with closing this. @jywu-msft what do you think ?