NVENC does not work on Debian bullseye

[aidalgol]

When I try to record (or stream) using the "NVIDIA NVENC H.264" encoder, I get an error complaining that the required nvenc API version is not met:

Required: 11.1 Found: 11.0

This seems wrong, because I have been able to use NVENC with previous driver versions.

OBS error in terminal

``` info: [NVENC encoder: 'streaming_h264'] settings: rate_control: CBR bitrate: 2500 cqp: 0 keyint: 47 preset: hq profile: high width: 1920 height: 1080 2-pass: false b-frames: 2 psycho-aq: 1 GPU: 0 [h264_nvenc @ 0x559ac46d6a40] Driver does not support the required nvenc API version. Required: 11.1 Found: 11.0 [h264_nvenc @ 0x559ac46d6a40] The minimum required Nvidia driver for nvenc is (unknown) or newer warning: [NVENC encoder: 'streaming_h264'] Failed to open NVENC codec: Function not implemented warning: [NVENC encoder] nvenc_create_internal failed, trying again without Psycho Visual Tuning info: --------------------------------- info: [NVENC encoder: 'streaming_h264'] settings: rate_control: CBR bitrate: 2500 cqp: 0 keyint: 47 preset: hq profile: high width: 1920 height: 1080 2-pass: false b-frames: 2 psycho-aq: 0 GPU: 0 [h264_nvenc @ 0x559ac4aa8940] Driver does not support the required nvenc API version. Required: 11.1 Found: 11.0 [h264_nvenc @ 0x559ac4aa8940] The minimum required Nvidia driver for nvenc is (unknown) or newer warning: [NVENC encoder: 'streaming_h264'] Failed to open NVENC codec: Function not implemented ```

I am running the proprietary nvidia driver, version 460.91.03, with the following flatpak packages:

`flatpak list` output (unrelated applications omitted)

``` OBS Studio com.obsproject.Studio 27.1.3 stable system Freedesktop Platform org.freedesktop.Platform 20.08.16 20.08 system Freedesktop Platform org.freedesktop.Platform 21.08.7 21.08 system Mesa org.freedesktop.Platform.GL.default 21.1.8 20.08 system Mesa org.freedesktop.Platform.GL.default 21.3.1 21.08 system nvidia-460-91-03 org.freedesktop.Platform.GL.nvidia-460-91-03 1.4 system ffmpeg-full org.freedesktop.Platform.ffmpeg-full 21.08 system openh264 org.freedesktop.Platform.openh264 2.1.0 2.0 system Freedesktop SDK org.freedesktop.Sdk 20.08.16 20.08 system ```

[TingPing]

A guess, nv-codec-headers was updated to 11.1 a few months ago: 0b01dfe3f6c33b4d7c825a245c4428fe8c5dc0f2

The 460 driver series is older but it could be reasonable for us to just ship 11.0, I doubt there is a big feature difference but I haven't tested it. I also am assuming newer 11.1 systems can use 11.0 just fine.

[TingPing]

I made a test PR here: https://github.com/flathub/com.obsproject.Studio/pull/173

Once it finishes building can you install it and test it?

[aidalgol]

With that build, I get a generic CUDA error:

info: ---------------------------------
info: [NVENC encoder: 'streaming_h264'] settings:
    rate_control: CBR
    bitrate:      2500
    cqp:          0
    keyint:       47
    preset:       hq
    profile:      high
    width:        1920
    height:       1080
    2-pass:       false
    b-frames:     2
    psycho-aq:    1
    GPU:          0

[h264_nvenc @ 0x559933d65c00] dl_fn->cuda_dl->cuInit(0) failed -> CUDA_ERROR_UNKNOWN: unknown error
warning: [NVENC encoder: 'streaming_h264'] Failed to open NVENC codec: Unknown error occurred
warning: [NVENC encoder] nvenc_create_internal failed, trying again without Psycho Visual Tuning
info: ---------------------------------
info: [NVENC encoder: 'streaming_h264'] settings:
    rate_control: CBR
    bitrate:      2500
    cqp:          0
    keyint:       47
    preset:       hq
    profile:      high
    width:        1920
    height:       1080
    2-pass:       false
    b-frames:     2
    psycho-aq:    0
    GPU:          0

[h264_nvenc @ 0x559933e61200] dl_fn->cuda_dl->cuInit(0) failed -> CUDA_ERROR_UNKNOWN: unknown error
warning: [NVENC encoder: 'streaming_h264'] Failed to open NVENC codec: Unknown error occurred

I definitely have CUDA working on this system, as I regularly use it with other software.

[TingPing]

Well it sounds like the right direction but I'm not familiar enough with nvenc/cuda to know how to further debug it.

[guihkx]

@aidalgol Have you tried running nvidia-modprobe -c0 -uas root to see if the CUDA_ERROR_UNKNOWN error goes away? Source.

[aidalgol]

@aidalgol Have you tried running nvidia-modprobe -c0 -uas root to see if the CUDA_ERROR_UNKNOWN error goes away? Source.

@guihkx Yeah, that did it! I noticed that section earlier, but I checked that /dev/nvidia_uvm existed, and it did, so I did not try running that command.