glxtest: VA-API test failed: failed to initialise VAAPI connection.

[ZhangTianrong]

I have read a few issues regarding this and tried almost every solution mentioned but am still stuck with this error.

$ NVD_LOG=1 NVD_BACKEND=direct firefox
Gtk-Message: 01:15:06.179: Not loading module "atk-bridge": The functionality is provided by GTK natively. Please try to not load it.
[GFX1-]: glxtest: VA-API test failed: failed to initialise VAAPI connection.
ATTENTION: default value of option mesa_glthread overridden by environment.
ATTENTION: default value of option mesa_glthread overridden by environment.
ALSA lib conf.c:4120:(snd_config_update_r) Cannot access file /usr/share/alsa/alsa.conf
ALSA lib seq.c:935:(snd_seq_open_noupdate) Unknown SEQ default
[2023-08-10T05:15:58Z ERROR mp4parse] Found 2 nul bytes in "\0\0"
[2023-08-10T05:15:58Z ERROR mp4parse] Found 2 nul bytes in "\0\0"
[2023-08-10T05:15:58Z ERROR mp4parse] Found 2 nul bytes in "\0\0"
[2023-08-10T05:15:58Z ERROR mp4parse] Found 2 nul bytes in "\0\0"

As to the environment variables, I have

#/etc/environment
PATH="/usr/local/cuda-12.2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
LD_LIBRARY_PATH="/usr/local/lib"
MOZ_DISABLE_RDD_SANDBOX="1"
LIBVA_DRIVER_NAME="nvidia"

The output of vainfo:

$ NVD_LOG=1 NVD_BACKEND=direct vainfo
      2510.296016238 [9225-9225] ../src/vabackend.c:2171       __vaDriverInit_1_0 Initialising NVIDIA VA-API Driver: 10
      2510.296037165 [9225-9225] ../src/vabackend.c:2180       __vaDriverInit_1_0 Now have 0 (0 max) instances
      2510.296040046 [9225-9225] ../src/vabackend.c:2206       __vaDriverInit_1_0 Selecting Direct backend
      2510.305753815 [9225-9225] ../src/direct/direct-export-buf.c:  85      direct_initExporter Found NVIDIA GPU 0 at /dev/dri/renderD128
      2510.305763955 [9225-9225] ../src/direct/nv-driver.c: 223            init_nvdriver Initing nvdriver...
      2510.305766799 [9225-9225] ../src/direct/nv-driver.c: 228            init_nvdriver Got dev info: 7 1 2 6
      2510.305787674 [9225-9225] ../src/direct/nv-driver.c: 246            init_nvdriver NVIDIA kernel driver version: 535.86.10, major version: 535
vainfo: VA-API version: 1.14 (libva 2.12.0)
vainfo: Driver version: VA-API NVDEC driver [direct backend]
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointVLD
      VAProfileVP8Version0_3          : VAEntrypointVLD
      VAProfileVP9Profile0            : VAEntrypointVLD
      VAProfileHEVCMain10             : VAEntrypointVLD
      VAProfileHEVCMain12             : VAEntrypointVLD
      VAProfileVP9Profile2            : VAEntrypointVLD
      VAProfileHEVCMain444            : VAEntrypointVLD
      2510.359201110 [9225-9225] ../src/vabackend.c:2081              nvTerminate Terminating 0x56237f7a2e50
      2510.359254446 [9225-9225] ../src/vabackend.c:2095              nvTerminate Now have 0 (0 max) instances

The ffmpeg being used is

$ ffmpeg -version
ffmpeg version N-111724-g7a2b587dea Copyright (c) 2000-2023 the FFmpeg developers
built with gcc 11 (Ubuntu 11.4.0-1ubuntu1~22.04)
configuration: --enable-gpl --enable-version3 --enable-static --disable-debug --disable-ffplay --disable-indev=sndio --disable-outdev=sndio --cc=gcc --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gmp --enable-libgme --enable-gray --enable-libaom --enable-libfribidi --enable-libass --enable-libvmaf --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librubberband --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libdav1d --enable-libxvid --enable-libzvbi --enable-libzimg --enable-nonfree --enable-cuda-nvcc --enable-libnpp --extra-cflags=-I/usr/local/cuda/include --extra-ldflags=-L/usr/local/cuda/lib64 --disable-static --enable-shared --enable-vaapi
libavutil      58. 15.100 / 58. 15.100
libavcodec     60. 23.100 / 60. 23.100
libavformat    60. 10.100 / 60. 10.100
libavdevice    60.  2.101 / 60.  2.101
libavfilter     9. 10.100 /  9. 10.100
libswscale      7.  3.100 /  7.  3.100
libswresample   4. 11.100 /  4. 11.100
libpostproc    57.  2.100 / 57.  2.100

The driver version is

$ cat /proc/driver/nvidia/version 
NVRM version: NVIDIA UNIX x86_64 Kernel Module  535.86.10  Wed Jul 26 23:20:03 UTC 2023
GCC version:  gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)

nvidia-smi output with Firefox running:

$ nvidia-smi
Thu Aug 10 01:35:38 2023       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.86.10              Driver Version: 535.86.10    CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 2080 Ti     On  | 00000000:00:07.0  On |                  N/A |
| 40%   45C    P2              57W / 260W |   1051MiB / 22528MiB |     14%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      1048      G   /usr/lib/xorg/Xorg                          302MiB |
|    0   N/A  N/A      1279    C+G   ...libexec/gnome-remote-desktop-daemon      604MiB |
|    0   N/A  N/A      1316      G   /usr/bin/gnome-shell                        129MiB |
|    0   N/A  N/A      2903      G   /usr/bin/nvidia-settings                      0MiB |
|    0   N/A  N/A      7747      G   ...irefox/2356/usr/lib/firefox/firefox        8MiB |
+---------------------------------------------------------------------------------------+

About the kernel parameter:

$ cat /etc/modprobe.d/nvidia-graphics-drivers-kms.conf 
# This file was generated by nvidia-driver-535
# Set value to 0 to disable modesetting
options nvidia-drm modeset=1

[elFarto]

That's a difficult one to suggest a fix for. The issue seems to be more related to Firefox than the driver. You might try running MOZ_LOG="PlatformDecoderModule:5 Dmabuf:5" firefox (without any instances of Firefox running) and see what it outputs.

[ZhangTianrong]

That's a difficult one to suggest a fix for. The issue seems to be more related to Firefox than the driver. You might try running MOZ_LOG="PlatformDecoderModule:5 Dmabuf:5" firefox (without any instances of Firefox running) and see what it outputs.

Thanks for the reply. I ran NVD_LOG=1 NVD_BACKEND=direct MOZ_LOG="PlatformDecoderModule:5 Dmabuf:5" firefox > firefox.log 2>&1, went to YouTube and played a video for a few seconds. The log file is here for Firefox installed with snap and here for Firefox installed with deb. I am not sure if this would be helpful, but here is the content of the about:support page in Firefox.

[jaskij]

I have a similar issue, and following Fedora Wiki checked about:support. Some things stand out:

Reading the issue, it seems that there's a bug in nVidia drivers, and it will be fixed in next release.

[ZhangTianrong]

I just removed everything (including configs, addons, etc.) and then reinstalled Firefox today using apt. I did no other changes, but somehow it manages to get away with the [GFX1-]: glxtest: VA-API test failed: failed to initialise VAAPI connection. error now. (There have been a few reboots since I saw thar error and posted this issue.)

Instead I see Firefox having trouble locating the correct device and falls back to software decoding as is shown below:

[Child 14034: RemVidChild]: D/PlatformDecoderModule RemoteMediaDataDecoder[7f6316ba2c10] ::RemoteMediaDataDecoder: 7f6316ba2c10 is created
[RDD 14140: MediaPDecoder #1]: D/PlatformDecoderModule FFVPX: Initialising VA-API FFmpeg decoder
[RDD 14140: MediaPDecoder #1]: D/PlatformDecoderModule FFVPX:   codec vp9 : Google VP9
[RDD 14140: MediaPDecoder #1]: D/Dmabuf DMABufDevice::Configure()
[RDD 14140: MediaPDecoder #1]: D/Dmabuf Loading DMABuf system library libgbm.so.1 ...
[RDD 14140: MediaPDecoder #1]: D/Dmabuf We're missing DRM render device!
[RDD 14140: MediaPDecoder #1]: D/PlatformDecoderModule FFVPX:   Can't get DRM VA-API display.
[RDD 14140: MediaPDecoder #1]: D/PlatformDecoderModule FFVPX:   Failed to create VA-API device context
[RDD 14140: MediaPDecoder #1]: D/PlatformDecoderModule FFVPX: Initialising FFmpeg decoder
[RDD 14140: MediaPDecoder #1]: D/PlatformDecoderModule FFVPX:   codec vp9 : Google VP9
[RDD 14140: MediaPDecoder #1]: D/PlatformDecoderModule FFVPX: Set get_buffer2 for customized buffer allocation
[RDD 14140: MediaPDecoder #1]: D/PlatformDecoderModule FFVPX:   FFmpeg decoder init successful.

So I just manually specifies the device to use by setting MOZ_DRM_DEVICE=/dev/dri/renderD128. Now things are working just fine and in nvidia-smi, it shows

Sat Aug 26 18:31:24 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.86.10              Driver Version: 535.86.10    CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 2080 Ti     On  | 00000000:00:05.0  On |                  N/A |
| 35%   42C    P2              83W / 260W |   1549MiB / 22528MiB |     37%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      1212      G   /usr/lib/xorg/Xorg                          237MiB |
|    0   N/A  N/A      1366    C+G   ...libexec/gnome-remote-desktop-daemon      604MiB |
|    0   N/A  N/A      1404      G   /usr/bin/gnome-shell                        160MiB |
|    0   N/A  N/A      2359      G   /usr/lib/firefox/firefox                    187MiB |
|    0   N/A  N/A      2711    C+G   /usr/lib/firefox/firefox                    352MiB |
+---------------------------------------------------------------------------------------+

Eventually, I set the following environment variables

#/etc/environment
PATH="/usr/local/cuda-12.2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
LD_LIBRARY_PATH="/usr/local/lib"
MOZ_DISABLE_RDD_SANDBOX="1"
LIBVA_DRIVER_NAME="nvidia"
NVD_BACKEND="direct"
MOZ_DRM_DEVICE="/dev/dri/renderD128"