Made it work without setting nvidia-drm.modeset=1 on a 10yr old Dell laptop

[burglarhobbit]

Configuration and results:

• Dell Inspiron i5-3542
• NVIDIA GPU: GeForce 840M
• Ubuntu 22.04.2 LTS
• Installed nvidia-vaapi-driver from the latest master branch (as of Sep 12, 2023)
• Nvidia driver version: 535.86.05 (Older version was 470.xx)
• GPU utilization verified using mpv and checking nvidia-smi output. My VLC still hangs on a 4K 60fps HEVC video shot on an iPhone, it's probably my ancient hardware haha.
• Updated NVIDIA drivers using this gist. Basically removes all NVIDIA libraries before installing newer drivers. Additionally, I also purged libnvidia* libraries.

Initial problems:

vainfo and vdpauinfo gave me errors. Tried searching but without any avail. Here were the errors I was getting:

$ vainfo
[vaapi] libva: /usr/lib/x86_64-linux-gnu/dri/nvidia_drv_video.so init failed
vaInitialize failed with error code 1 (operation failed),exit
-----
$ vdpauinfo
display: :1   screen: 0
GPU at BusId 0x8 doesn't have a supported video decoder
Error creating VDPAU device: 1

I went into the rabbithole of gpus support by driver version. I came across some vdpau links which helped me realized I needed newer NVIDIA drivers (535.xx) rather than my old one (470.xx). It's definitely the NVDEC support in newer drivers for my GPU that made it work.

Setting nvidia-drm.modeset=1 actually made my nvidia-smi stop working and gave an error message. Removing drm modeset setting and a reboot after works. Thank god for me not effing up any screen rendering programs.

Successful vainfo output:

libva info: VA-API version 1.14.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/nvidia_drv_video.so
libva info: Found init function __vaDriverInit_1_0
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.14 (libva 2.12.0)
vainfo: Driver version: VA-API NVDEC driver [egl backend]
vainfo: Supported profile and entrypoints

Unfortunately vdpauinfo still gives me an error but it's a non-issue.

Hope this helps!

[elFarto]

Setting nvidia-drm.modeset=1 is required by this driver (it's the only way DMA-BUF works). It should not break nvidia-smi, it doesn't on my machine.

What error does nvidia-smi produce with it set?

[burglarhobbit]

I got this error last time I set nvidia-drm.modeset=1:

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

I didn't try turning the setting on/off again.

[elFarto]

That error indicates that the NVIDIA driver isn't even loaded. Is the nouveau properly blacklisted? Could you paste your dmesg output from boot with modeset enabled?