search

occ-ai / obs-backgroundremoval

An OBS plugin for removing background in portrait images (video), making it easy to replace the background when recording or streaming.
GNU General Public License v2.0
2.68k stars 185 forks source link

Cache TensorRT modules #549

Closed sjuxax closed 2 months ago

sjuxax commented 4 months ago

Is your feature request related to a problem? Please describe.

Every time I open the OBS with this plugin installed, there's a 60 sec+ delay while the TensorRT modules are compiled.

Describe the solution you'd like

Most applications use a cache to prevent this from occurring on each startup. That would be great.

Describe alternatives you've considered

Waiting a long time each time I start OBS, which is annoying.

royshil commented 4 months ago

do you have any details on precompilation of tensorRT modules?

sjuxax commented 4 months ago

Both ONNX Runtime and Tensorflow provide mechanisms for caching TensorRT engines.

I don't have a whole lot of other information, unfortunately, other than it's worked well on the onnxruntime applications I've used.

krakowski commented 2 months ago

I forked this repository and added CUDA as an option which loads instantly. I could also look into the TensorRT option mentioned by @sjuxax.

@royshil Should I open a pull request with my changes?

I have also made a small change to the build script, allowing me to set the ONNX Runtime version using variables.

EDIT

The latest commit enables the TensorRT cache (placed in ~/.cache/obs-backgroundremoval/tensorrt). I can confirm that using the cache reduced startup time from ~50 seconds to ~4 seconds, in my case.

umireon commented 2 months ago

@krakowski Feel free to open a pull requests! We appreciate any contributions!

krakowski commented 2 months ago

@umireon Done, please see #575 :slightly_smiling_face: