Video effects as OBS effect for window capture on Linux

[Corben78]

What is the requested feature? Adding the possibility to apply the Video Stabilizer (and Adaptive De-Blocker, Lens Correction, Calibration Tool) to window capture sources in OBS.

Why do you want this feature? Presenting VR games to viewers will be more pleasant if the window isn't shaking so much due to head movement. Same as #3 but for Linux

As shown on youtube, a stabilized video is of higher quality and also helps with encoding the video.

[Crowsinc]

Hi, this is will be released as part of the next update.

Its actually been a feature for a while, but its been plagued with Linux driver errors so has been disabled until now!

[Corben78]

That are fantastic news! Thanks a lot! I've already thought about workarounds, like using NDI or Teleport, but this will make it much easier (getting opencv compiled is already tricky enough) Looking forward to the update, can you share any ETA yet?

[Crowsinc]

I can't provide an ETA because I'm quite busy outside of the project, but its in the final stages.

[Crowsinc]

This has now been released as part of update 1.2.1.

Please let me know if you encounter any issues!

[Corben78]

Unfortunately 1.2.1 seems to have GLIBC_2.32 as requirement, while Ubuntu 20.04 LTS ships GLIBC_2.30. Looks like 1.2.0 was still working with GLIBC_2.30 as it did load for me. As I need xrdesktop for VR, which hasn't been updated for Ubuntu 22.04 LTS yet I cannot upgrade. Would it be possible to compile LiveVisionKit against GLIBC_2.30?

[Crowsinc]

Ill see about re-compiling it next time I'm on Linux and ill let you know!

[Corben78]

Looks like glibc 2.31 is installed on Ubuntu 20.04 LTS. I was just looking for the GLIBC_2.3 string (strings /lib/x86_64-linux-gnu/libc.so.6 | grep -i GLIBC_2.3) which didn't show 2.31. But when I look a bit less strict:

$ strings /lib/x86_64-linux-gnu/libc.so.6 | grep -i glibc
...
GLIBC_2.29
GLIBC_2.30
GLIBC_PRIVATE
glibc 2.31
GNU C Library (Ubuntu GLIBC 2.31-0ubuntu9.9) stable release version 2.31.
...

Maybe this info helps.

[Crowsinc]

None of my current Linux distributions support glibc 2.30, I've swapped them out since compiling version 1.2. But I'm happy to bring back an Ubuntu 20.04 LTS based distro to ensure future compatibility. It will most likely be done within the upcoming week.

[Crowsinc]

Here is a copy of the OBS28 version compiled for Ubuntu 20.04, please let me know how it goes, or if you need an OBS27 version instead.

LiveVisionKit-1.2.1-Linux.zip

[Corben78]

Awesome! Yeah, I would need a version for OBS 27.

[Crowsinc]

Here you go

OBS27-LiveVisionKit-1.2.1-Linux.zip

[Corben78]

Thanks a lot! It does work!

Unfortunately it seems very performance heavy on the CPU side. OBS 27 is usually using about 130% CPU for me, which is normally not an issue. With LiveVisionKit it jumps to about 500% CPU usage. I'm wondering if something went wrong with my manually compiled opencv version. I does have the compile option -D WITH_OPENCL=ON, but it doesn't seem to use my nVidia GPU for computation. As mentioned in #13 I wasn't able to get it compiled with CUDA.

This results in a rubber-band-like effect, where the captured window seems to slow down and then speed up again. Visible here: https://youtu.be/pfjYDVmJhEQ?t=2276

Any hints on that?

[Crowsinc]

Yes, it is strange that it is lagging and has such high CPU usage given your computer hardware. For reference, the core of the project was made and optimized on an i7 4790k and RTX 2070S.

Could you please supply me with your stabilizer settings, an OBS log, and the output of running opencv_version --opencl in your terminal?

All of the heavy processing is done via OpenCL for cross-compatibility, so it is possible that perhaps OpenCV is using your CPU as the main OpenCL processing device instead of the GPU.

Also given that you are running an Nvidia GPU, I would suggest compiling OpenCV with the additional -D WITH_OPENGL=ON and -D WITH_QT=ON flags. You may need to install the base qt5 development libraries. On some systems, this can enable some large performance optimizations in LiveVisionKit.

[Corben78]

Thanks for sticking with me. I'm doing VR on my notebook, which has a nVidia 2080 instead of a 3070, but is also an i9 9900 CPU.

opencv_version --opencl
4.6.0-dev
OpenCL Platforms:
    NVIDIA CUDA
        dGPU: NVIDIA GeForce RTX 2080 (OpenCL 3.0 CUDA)
Current OpenCL device:
    Type = dGPU
    Name = NVIDIA GeForce RTX 2080
    Version = OpenCL 3.0 CUDA
    Driver version = 515.65.01
    Address bits = 64
    Compute units = 46
    Max work group size = 1024
    Local memory size = 48 KB
    Max memory allocation size = 1 GB 969 MB 528 KB
    Double support = Yes
    Half support = No
    Host unified memory = No
    Device extensions:
        cl_khr_global_int32_base_atomics
        cl_khr_global_int32_extended_atomics
        cl_khr_local_int32_base_atomics
        cl_khr_local_int32_extended_atomics
        cl_khr_fp64
        cl_khr_3d_image_writes
        cl_khr_byte_addressable_store
        cl_khr_icd
        cl_khr_gl_sharing
        cl_nv_compiler_options
        cl_nv_device_attribute_query
        cl_nv_pragma_unroll
        cl_nv_copy_opts
        cl_nv_create_buffer
        cl_khr_int64_base_atomics
        cl_khr_int64_extended_atomics
        cl_khr_device_uuid
        cl_khr_pci_bus_info
        cl_khr_external_semaphore
        cl_khr_external_memory
        cl_khr_external_semaphore_opaque_fd
        cl_khr_external_memory_opaque_fd
    Has AMD Blas = No
    Has AMD Fft = No
    Preferred vector width char = 1
    Preferred vector width short = 1
    Preferred vector width int = 1
    Preferred vector width long = 1
    Preferred vector width float = 1
    Preferred vector width double = 1
    Preferred vector width half = 0

OBS Log attached: 2022-11-09 19-32-33.txt

edit: oh... this line in the log seems problematic: [LiveVisionKit] The OpenCL interop context failed to initialize and has been disabled! (bad drivers?)

Do you know the needed packages to install for Ubuntu 20.04 to compile opencv with qt5 (would just save me some time).

[Crowsinc]

I will have to thank you as well for sticking with it because the project is still in its early stages, so feedback like this is extremely valuable!

I believe the correct qt5 package is qt5-default, and I'd also suggest running git checkout 4.6.0 on the opencv project to make sure you're compiling the release copy of OpenCV 4.6.0. The interop context error you are seeing is expected as you don't have the OpenGL and Qt flags enabled for the compilation. However, the drivers for the interop are very shaky on Linux in general so its almost standard that it won't work, which does lead to 2-3ms worse performance on most filters.

Looking at the output from opencv_version --opencl, it seems that your graphics card is correctly identified as the OpenCL processor. I'm not sure why there is such a high load on the CPU, however I can tell you that the stabilizer currently struggles to deal with scenes that have a lot of parallax layers. In the Skyrim gameplay you linked, the scenery has a lot of depth to it (the foreground, the far trees, the background mountains, the sky etc.) which all move at different speeds to its hard to stabilize. It is possible that all these layers are adding a lot of computational load to the CPU, causing the rubber banding effect. There are already plans to fix this for the next major update (although the ultimate plan for VR is to just grab the movement data directly from the headset and stabilize that, as the current stabilizer is really built for IRL content.)

In your experience, was the rubber banding consistent, or was it just in certain areas?

I also see a lot of instances of the following message in the log.

xcompcap: cleanup glXReleaseTexImageEXT failed: GLXBadPixmap
xcompcap: [window-capture: 'VR View 2'] update settings:
title: VR View
    class: vrmonitor
    Bit depth: 24
    Found proper GLXFBConfig (in 9): yes

Are you aware of anything on your end which is causing this error? It is sometimes followed by a [LiveVisionKit] '(LVK) Video Stabilizer' frame queue is outdated, restarting..., message. Basically, if it has been over a second since the last frame, the stabilizer will restart, creating a short delay.

[Corben78]

Will try to compile a new opencv and see if that's having a positive effect on performance.

From watching the VoD it seems like the rubber band effect was only outside the city in Skyrim, as in e.g. the Inn before nobody was complaining about it, and I can't really see any rubber banding. So the explanation about the higher depth outside seems very reasonable. Even though I'd imagine a IRL cam capture when moving a bit faster would cause the same issue? I was only realizing it through the feedback from viewers. In VR itself there was no rubber banding at all.

Sounds great if it would be possible to use the VR motion data to smooth the VR view.

What I am doing here for capturing the 1st person view, is to enable the VR mirror view in Steam VR, as from my experience all game windows are only one eye and sometimes even only a half screen window, mostly with a narrow FoV. I even enable "both eyes" in the mirror view, which results in an even better FoV for the viewers, but introduces a little overlapping area, in my case on the right side of the window. It's barely visible, mostly when there is text traversing that intersection area (e.g. that chat overlay window in VR).

And that VR mirror window capture might cause those errors, e.g. if I restart OBS the capture of that window is mostly black, until I reopen the window. Might be better with OBS 28, as the Linux devs of OBS 28 said they've put a lot of effort into the window capture. I'm still using nvfbc (the zero copy capture from the nVidia frame buffer), which doesn't work anymore with OBS 28, as they switched to EGL and nVidia doesn't support nvfbc with EGL (yet).

Also I have switched desktops a few times (I got multiple workspaces in Gnome configured), and things tend to break on switching sometimes, e.g. xrdesktop crashes, or even Steam VR crashes. During that session also Sykrim dropped to desktop twice. Once things are working, it can be stable for a few hours. Nevertheless the xcompcap errors might come from those desktop/workspace switches.

[Crowsinc]

If it was mainly happening outside, then I am fairly confident that it is the parallax issue. It does occur on IRL content as well but it is less common and less pronounced, simply because real life cameras have very different properties then a rendered frame of a game.

I already have solutions planned for the stabilizer which tackles this issue and others in update 1.3. Unfortunately, this will likely take a few months as there is a lot of research and engineering to be done. So I am sorry on that regard, but I am confident that there will be a fairly robust stabilization solution for VR in update 1.3, whether through the improved stabilizer or by taking the data directly from the headset.