GPU TensorRT on kubuntu 22.04 issue

[Zahrun]

Describe the bug

I installed obs-backgroundremoval v0.5.17 deb from the github release page. OBS is installed from the ppa in version 29.0.2. I can't seem to use the new GPU tensorRT feature.

To Reproduce

Steps to reproduce the behavior:

1. Go to Filters
2. Click on Background Removal
3. Pick Inference device as "GPU - TensorRT"
4. Initially it seems the CPU is still in use
5. Close OBS
6. Start OBS
7. The filter gives a black image

Expected behavior

Be able to use the GPU mode

Log and Crash Report

2023-04-21 14-18-57.txt

14:18:58.187: os_dlopen(/usr//lib/x86_64-linux-gnu/obs-plugins/libonnxruntime_providers_cuda.so->/usr//lib/x86_64-linux-gnu/obs-plugins/libonnxruntime_providers_cuda.so): libcublasLt.so.11: cannot open shared object file: No such file or directory
14:18:58.187: 
14:18:58.190: os_dlopen(/usr//lib/x86_64-linux-gnu/obs-plugins/libonnxruntime_providers_cuda.so->/usr//lib/x86_64-linux-gnu/obs-plugins/libonnxruntime_providers_cuda.so): libcublasLt.so.11: cannot open shared object file: No such file or directory
14:18:58.190: 
14:18:58.190: Module '/usr//lib/x86_64-linux-gnu/obs-plugins/libonnxruntime_providers_cuda.so' not loaded
14:18:58.198: os_dlopen(/usr//lib/x86_64-linux-gnu/obs-plugins/libonnxruntime_providers_tensorrt.so->/usr//lib/x86_64-linux-gnu/obs-plugins/libonnxruntime_providers_tensorrt.so): libcudnn.so.8: cannot open shared object file: No such file or directory
14:18:58.198: 
14:18:58.201: os_dlopen(/usr//lib/x86_64-linux-gnu/obs-plugins/libonnxruntime_providers_tensorrt.so->/usr//lib/x86_64-linux-gnu/obs-plugins/libonnxruntime_providers_tensorrt.so): libcudnn.so.8: cannot open shared object file: No such file or directory
14:18:58.201: 
14:18:58.201: Module '/usr//lib/x86_64-linux-gnu/obs-plugins/libonnxruntime_providers_tensorrt.so' not loaded

Desktop (please complete the following information):

• OS: kubuntu 22.04
• Browser: Firefox
• Plugin Version: 0.5.17
• OBS Version: 29.0.2

[Zahrun]

If I install nvidia-cudnn, then it should work. Should the deb depend on that?

PS: https://www.debian.org/doc/manuals/debian-faq/pkg-basics.en.html#depends The Debian package system has a range of package "dependencies" which are designed to indicate (in a single flag) the level at which Program A can operate independently of the existence of Program B on a given system:

Package A depends on Package B if B absolutely must be installed in order to run A. In some cases, A depends not only on B, but on a version of B. In this case, the version dependency is usually a lower limit, in the sense that A depends on any version of B more recent than some specified version.

Package A recommends Package B, if the package maintainer judges that most users would not want A without also having the functionality provided by B.

Package A suggests Package B if B contains files that are related to (and usually enhance) the functionality of A. 

[Zahrun]

After installing nvidia-cudnn, obs does not start anymore:

debug: Found portal inhibitor
debug: Attempted path: share/obs/obs-studio/locale/en-US.ini
debug: Attempted path: /usr/share/obs/obs-studio/locale/en-US.ini
debug: Attempted path: share/obs/obs-studio/locale.ini
debug: Attempted path: /usr/share/obs/obs-studio/locale.ini
debug: Attempted path: share/obs/obs-studio/locale/en-GB.ini
debug: Attempted path: /usr/share/obs/obs-studio/locale/en-GB.ini
info: Using preferred locale 'en-GB'
debug: Attempted path: share/obs/obs-studio/themes/Yami.qss
debug: Attempted path: /usr/share/obs/obs-studio/themes/Yami.qss
info: Using EGL/X11
info: CPU Name: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
info: CPU Speed: 4139.090MHz
info: Physical Cores: 6, Logical Cores: 12
info: Physical Memory: 31745MB Total, 901MB Free
info: Kernel Version: Linux 6.1.0-1009-tuxedo
info: Distribution: "TUXEDO OS" "22.04"
info: Session Type: x11
info: Window System: X11.0, Vendor: The X.Org Foundation, Version: 1.21.1
info: Qt Version: 6.5.0 (runtime), 6.2.4 (compiled)
info: Portable mode: false
qt.core.qmetaobject.connectslotsbyname: QMetaObject::connectSlotsByName: No matching signal for on_tbar_position_valueChanged(int)
info: OBS 29.0.2 (linux)
info: ---------------------------------
info: ---------------------------------
info: audio settings reset:
        samples per sec: 48000
        speakers:        2
        max buffering:   960 milliseconds
        buffering type:  dynamically increasing
info: ---------------------------------
info: Initializing OpenGL...
info: Loading up OpenGL on adapter Intel Mesa Intel(R) UHD Graphics 630 (CFL GT2)
info: OpenGL loaded successfully, version 4.6 (Core Profile) Mesa 22.3.5, shading language 4.60
info: ---------------------------------
info: video settings reset:
        base resolution:   1920x1080
        output resolution: 1280x720
        downscale filter:  Bicubic
        fps:               30/1
        format:            NV12
        YUV mode:          Rec. 709/Partial
info: NV12 texture support not available
info: P010 texture support not available
info: Audio monitoring device:
        name: Default
        id: default
info: ---------------------------------
warning: Failed to load 'en-GB' text for module: 'aja-output-ui.so'
warning: No AJA devices found, skipping loading AJA UI plugin
warning: Failed to initialize module 'aja-output-ui.so'
warning: Failed to load 'en-GB' text for module: 'aja.so'
warning: No AJA devices found, skipping loading AJA plugin
warning: Failed to initialize module 'aja.so'
warning: Failed to load 'en-US' text for module: 'decklink-captions.so'
warning: Failed to load 'en-US' text for module: 'decklink-output-ui.so'
libDeckLinkAPI.so: cannot open shared object file: No such file or directory
warning: A DeckLink iterator could not be created.  The DeckLink drivers may not be installed
warning: Failed to initialize module 'decklink.so'
warning: Failed to load 'en-GB' text for module: 'frontend-tools.so'
obs: symbol lookup error: /usr//lib/x86_64-linux-gnu/obs-plugins/libonnxruntime_providers_cuda.so: undefined symbol: Provider_GetHost
obs: symbol lookup error: /usr//lib/x86_64-linux-gnu/obs-plugins/libonnxruntime_providers_cuda.so: undefined symbol: Provider_GetHost

[umireon]

@Zahrun Can you try this build? https://github.com/royshil/obs-backgroundremoval/suites/12403050442/artifacts/659505292

[umireon]

If you cannot get it run, please execute the following command.

sudo rm /usr//lib/x86_64-linux-gnu/obs-plugins/libonnxruntime*

[Zahrun]

Thank you @umireon Now obs launches, I see one frame from the webcam and then only black. This time the error seems to be

error: [obs-backgroundremoval] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1106 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_tensorrt.so with error: libnvinfer.so.8: cannot open shared object file: No such file or directory

So I think I have to install cuda repo as in https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#ubuntu

sudo apt-key del 7fa2af80
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.0-1_all.deb
sudo dpkg -i cuda-keyring_1.0-1_all.deb
sudo apt update
sudo apt install libnvinfer-bin

This, I guess, makes the previous installation of nvidia-cudnn unnecessary since libcublas and libcudnn are dependencies of libnvinfer.

And... it works. The model seems to take quite some time to load, like 40 seconds, which delays the opening of obs and also it takes this time to change from one model to the other. Then it uses about 20-30% CPU, 2% GPU and 824 MiB of VRAM on MediaPipe (instead of 50-60% cpu) 20-30% CPU, 8% GPU and 840 MiB of VRAM on PPHumanSeg (instead of 50-60% cpu) The other models have similar resource usage but lesser output quality imo. RobustVideoMapping fails (it doesn't on CPU) with:

2023-04-23 17:06:10.104934993 [E:onnxruntime:background-removal-inference, tensorrt_execution_provider.h:61 log] [2023-04-23 11:36:10   ERROR] [graphShapeAnalyzer.cpp::checkCalculationStatusSanity::1504] Error Code 2: Internal Error (Assertion !isPartialWork(p.second.symbolicRep) failed. )
2023-04-23 17:06:10.118547421 [E:onnxruntime:, inference_session.cc:1532 operator()] Exception during initialization: /onnxruntime_src/onnxruntime/core/providers/tensorrt/tensorrt_execution_provider.cc:897 SubGraphCollection_t onnxruntime::TensorrtExecutionProvider::GetSupportedList(SubGraphCollection_t, int, int, const onnxruntime::GraphViewer&, bool*) const [ONNXRuntimeError] : 1 : FAIL : TensorRT input: 389 has no shape specified. Please run shape inference on the onnx model first. Details can be found in https://onnxruntime.ai/docs/execution-providers/TensorRT-ExecutionProvider.html#shape-inference-for-tensorrt-subgraphs

error: [obs-backgroundremoval] Exception during initialization: /onnxruntime_src/onnxruntime/core/providers/tensorrt/tensorrt_execution_provider.cc:897 SubGraphCollection_t onnxruntime::TensorrtExecutionProvider::GetSupportedList(SubGraphCollection_t, int, int, const onnxruntime::GraphViewer&, bool*) const [ONNXRuntimeError] : 1 : FAIL : TensorRT input: 389 has no shape specified. Please run shape inference on the onnx model first. Details can be found in https://onnxruntime.ai/docs/execution-providers/TensorRT-ExecutionProvider.html#shape-inference-for-tensorrt-subgraphs

Segmentation fault

I think I will leave it on GPU, MediaPipe, calculate mask every 3 frames to reduce further CPU load. Thank you for adding GPU support to Linux, great improvement 🎉 I will try to package all that in pacstall for ubuntu and in nix. Just one doubt, will this work on all hardware or is it only compatble with NVIDIA chips?

[umireon]

Thank you for adding GPU support to Linux, great improvement 🎉

I'm very happy that we can deliver the long-demanded GPU support for Linux!

This plugin requires TensorRT 8.5, CUDA 11.6, and cuDNN 8.2.4 so you are in the right way.

The model seems to take quite some time to load, like 40 seconds, which delays the opening of obs and also it takes this time to change from one model to the other.

We might tackle this problem later.

I will try to package all that in pacstall for ubuntu and in nix.

I appreciate your efforts very much!

Just one doubt, will this work on all hardware or is it only compatble with NVIDIA chips?

Only for NVIDIA chips. We need to add other execution providers to support other chips but we have no resources to add them.

[Zahrun]

@Zahrun Can you try this build? https://github.com/royshil/obs-backgroundremoval/suites/12403050442/artifacts/659505292

Will this fix come in a release?

[umireon]

This fix will be included in v0.5.18.

[Zahrun]

Ok thank you

[umireon]

This is fixed by #257

[poleguy]

I can confirm that GPU TensorRT works on linux. Thanks for all your work! (Ubuntu 22.04, OBS 30.0.0, obs-backgroundremoval 1.1.9).

I can also confirm that startup time for OBS is very long (many seconds). I will not be using GPU because of this delay. I bought a new GPU this week. One of my goals was to try out this obs-backgroundremoval gpu mode. It's not bad. But until the latency is hidden from OBS startup, it's not worth it for me. It's fine if it takes 40 seconds for the filter to start working, but it's not okay for me to slow the load of OBS to wait for this. Can the latency be put in a different thread or something?

It would also be nice to add the list of dependencies to the front page. I am not a CUDA developer, so it took a long time to figure out that I needed to look in the log messages for an error. It took a long time to figure out I might need to install CUDA, and how to do it. Then I couldn't get it to work with CUDA 12... With no list of version requirements on the github page, I just started installing the latest version of everything I could think of.

Can we edit the readme on the top page to add a line like this?

This plugin requires TensorRT 8.5, CUDA 11.6, and cuDNN 8.2.4

I actually have CUDA 11.8 and 12.3 installed, TensorRT 8.6.1, and cudnn 8.9.7.29 So I have no clear idea what exactly the requirements are to make a suggestion.

[royshil]

@poleguy this is excellent feedback!

i can certainly try to make the errors more visible in the Filter view if any error occur they can popup there in red.

what kind of errors did you run into that you had to look in the logs?