NVIDIA Optimus laptop fails to capture vkcube

[MisterMonstr]

Fedora KDE 36 on ThinkPad P1 Gen 3 with NVIDIA Quadro T2000. NVIDIA Driver 520.56.06. Running in X11 with external display. Used this script I edited to compile and install for both 64-bit and 32-bit: obs-vkcapture_64+32.sh.txt

Started capture of vkcube with OBS_VKCAPTURE_LINEAR=1 prime-run obs-gamecapture vkcube Started OBS with prime-run obs

In the terminal, I see this in the OBS output:

[linux-vkcapture] Creating texture from dmabuf 500x500 modifier:0
[linux-vkcapture]  [0] fd:67 stride:2016 offset:0
glEGLImageTargetTexture2DOES failed, glGetError returned GL_INVALID_OPERATION(0x502)
[linux-vkcapture] Could not create texture from dmabuf source

... and this in the vkcube output:

[obs-vkcapture] Init Vulkan 1.2.1
MESA-INTEL: warning: Performance support disabled, consider sysctl dev.i915.perf_stream_paranoid=0
Selected GPU 1: Quadro T2000 with Max-Q Design, type: 2
[obs-vkcapture] Texture VK_FORMAT_B8G8R8A8_UNORM 500x500
[obs-vkcapture] ------------------ vulkan capture started ------------------
[obs-vkcapture] ------------------- vulkan capture freed -------------------

Running both on the Intel iGPU (obs , VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/intel_icd.x86_64.json obs-gamecapture vkcube) works just fine.

This plugin looks really promising; I hope this is fixable!

OBS Log: 2022-10-30 22-42-27.txt

[nowrep]

I can't really help you, sorry. You can try to report this to nvidia directly.

[MisterMonstr]

I think I got this to work now. I ensured that early modesetting was enabled with the help of this guide: https://www.if-not-true-then-false.com/2015/fedora-nvidia-guide/2/:

echo "options nvidia_drm modeset=1" >> /etc/modprobe.d/nvidia.conf

/etc/dracut.conf.d/nvidia.conf:

add_drivers+=" nvidia nvidia_modeset nvidia_uvm nvidia_drm "

install_items+=" /etc/modprobe.d/nvidia.conf "

sudo dracut -f

I also start games/vkcube with env OBS_VKCAPTURE=1 obs-gamecapture %command%. (I should have tried testing with this before filing this issue...) Some games don't work, but most do. I'll have to do a bit more testing to see if I can figure out why.