Nvidia Driver Branch 510 Adds Support for AV1 Decode in VDPAU

[gardotd426]

I don't know quite enough about the code to add the support, but a test branch attempting to add it should be trivial, right?

[marioroy]

@gardotd426, @elFarto is making an attempt AV1 decoding.

[gardotd426]

Yeah, but as has already been discussed on the AUR page for this package, the two packages conflict and will remain that way for a while - especially as the developer of the nvidia-vaapi-driver repo explicitly said that it was nowhere near production-ready - and the fact that either you'd have to choose Firefox or Chromium/Chrome/Brave, or screw around with per-app environment variables, and in this specific instance it pretty much only makes sense to use Firefox for AV1 and Chromium/Chrome/etc for VP9/H264/etc. And until the conflict in the AUR packages is removed, Arch-based distro users will have to manually edit PKGBUILDs and whatnot to use both, along with the aforementioned environment variable nonsense.

I'm glad they're working on it, but it's only for firefox (and only ever will be), and even many of the most ardent Firefox fans are admitting how horrible it's become/becoming as a browser. So it would be nice if this driver could implement AV1, now that it's supported in the drivers.

[marioroy]

By the way, VP9/H.264 works great in Firefox. Though it doesn't use the Video Engine for VP9, it runs on the GPU nevertheless. This vdpau-va-driver-v9 branch makes it possible. See FFmpeg on Clear Linux.

[gardotd426]

Except FFmpeg on Clear Linux is 100% incompatible with Arch, as nothing from Nvidia gets installed in /opt/nvidia, and cuda doesn't get put in /usr/local/lib, but /opt/cuda, and even after modifying the scripts for stuff like hardware decoding in firefox to point to the actual locations, it doesn't work. That FFmpeg on Clear Linux link is for Clear Linux and Clear Linux alone.

[marioroy]

Sadly the project works only in Clear Linux. /usr/local/cuda is a symbolic link to /opt/cuda-11.5.1. It has been a while and looking through /etc/modprobe.d I have modeset enable in nvidia-modeset.conf.

options nvidia-drm modeset=1

[marioroy]

Hi, @gardotd426

FFmpeg also failed for me recently. Previously, the following was false in the documentation at the time you tested. It must remain enabled (default) or hardware acceleration will not work.

media.rdd-process.enabled                      true

[xuanruiqi]

OTOH, only a few very new NVIDIA cards have AV1 acceleration, so I doubt this would be a priority at this point.

[gardotd426]

Unfortunately, mine is one of those, lol (RTX 3090).

[ghost]

Honestly I donno why a need for AV1 is even necessary. No one other than YouTube uses AV1 (& that can be switchable to VP9 from its settings). And ya it needs far more cpu resources than vp9 for just few less data savings. Not worth at all, caz vp9 still lives for its entirety.

[gardotd426]

Honestly I donno why a need for AV1 is even necessary. No one other than YouTube uses AV1 (& that can be switchable to VP9 from its settings).

Well for one thing, if something is available on other modern operating systems, it should be available on Linux. Otherwise it's just one more thing that Linux can't do that everyone else can (like HDR). Yeah, we as the community have to do a lot of the stuff ourselves, but that doesn't matter. If we care about Linux adoption, that's what we have to do.

Also, a lot of people in the tech sphere seem to think that AV1 is going to take over. We'll see. But anyway, if it's possible to implement, it should be implemented. If it's not, then fair enough.

[ghost]

It's not just the software availability but most importantly the hardware limitation. AV1 supports just on the 'select NVIDIA cards'. I understand your concern that the future is very bright for upcoming codecs. That obviously will be a thing only when the AV1 support spreads all over the current hardware. Just saying this is not that time, caz of limited hardware support. Also the fact that developers have elsewhere important other stuff to accomplish such as bug fixes, usability lacking improvements, etc. HDR is already supported by almost all video players on Linux. I think it mostly depends in hardware implementation, not the software. Also Linux adoption isn't about future proofing. It's about improving the present experience.

[gardotd426]

HDR is already supported by almost all video players on Linux. I think it mostly depends in hardware implementation, not the software.

It's got nothing to do with hardware. X11 will never support HDR, and Wayland has zero support for it right now. Until it does, it doesn't matter whether MPV or VLC support it. Also, the vast majority of users that consume HDR content on their PCs consume it through either a browser or via a video game, so not through a video player. And there is no HDR support on Linux for any of that. And it's all software, not hardware.

But that's off-topic and irrelevant to this thread.

Also Linux adoption isn't about future proofing. It's about improving the present experience.

For one, the 7000 series of AMD GPUs seemingly will support AV1 decoding. Intel's Arc supports AV1 decode and encode. Nvidia already supports it. And the 30 series is actually quite popular. It's not some mythical lineup that only 1 or 2% of dGPU users have. It's much more than that. But more importantly, most people don't care whether some random guy says it's "not necessary today." Windows can do it, MacOS can do it, so if Linux can't, that's just another thing that Linux can't do that other modern OSes can.

[ghost]

It's not some mythical lineup that only 1 or 2% of dGPU users have.

You sure the product lineups you just mentioned aren't 1 or 2% ? I think they are 1 or 2% (if not, less than that). And the hardware that majority of users have that support AV1 hw decode, are almost to none (which do change in the future). That's just pure future proofing you're talking about.

Also, this might be phishy but, EU Antitrust regulators recently trying to probe a lawsuit against AOM for AV1 licenses.

Offtopic:

And it's all software, not hardware.

Sure. If everything is software, then people would just get a basic monitor or TV & experience HDR in it. XD

For HDR compatibility, only tone mapping & bit depth mapping are enough for software to get HDR right. Rest of the playback is handled by the HDR display itself (brightness, contrast, color depth, backlight intensity, strobing, etc) Without those, the color looks like washed out & that's the only drawback of HDR support lack.

[marioroy]

There is a recent article on Phoronix.

NVIDIA Lands AV1 VDPAU Hardware Acceleration In FFmpeg

I built FFmpeg master (upcoming 5.1 release) and see av1_vdpau mentioned in the list.

Enabled hwaccels:
av1_nvdec               
av1_vaapi               
av1_vdpau
...             

[gardotd426]

You sure the product lineups you just mentioned aren't 1 or 2% ? I think they are 1 or 2% (if not, less than that). And the hardware that majority of users have that support AV1 hw decode, are almost to none (which do change in the future). That's just pure future proofing you're talking about.

@pc00per Um... I think you need to maybe work on your addition skills. 30 series GPUs make up 16.11% of the GPUs on the Steam hardware survey. I just went through and meticulously did the math myself. Hell, the 3060 Mobile GPU alone is almost 3% by itself. So again, I don't know where on earth you got your "1-2%" figure, but it's wildly wrong, and I recommend you delete/strikethrough your original comment and edit it with the correct information. Actually, considering that it's the most popular GPU lineup on Steam aside from maybe Pascal (and Ampere is gaining on Pascal rapidly), I'd say the fact that Ampere makes up 16% of the GPUs on Steam (and that INCLUDES iGPUs, so when talking about dGPUs it's actually much, much** higher), you should reform your entire stance.

I'm going to give you the chance to show integrity, which requires adjusting your stance when provided with new information. You originally claimed that it was a minimal number of users, 1-2%, and that is objectively not only wrong, but as wrong as it could possibly be. It's actually 800-1600% more than that. That necessitates you reappraising your approach to this issue.

Also, your link is not at all how you calculate the numbers. That link takes you to segmented lists with separate lists for Vulkan GPUs, DX12 GPUs, DX11 GPUs, etc. That's not an accurate way to gauge. But if it were, you would see that just in the top 20 cards or so on the Vulkan list, the Ampere GPUs that have made it that high alone make up almost 25% of all GPUs on the list:

You actually look at the main survey page, and click on "Video Card Description," which is a general list of all video cards, including iGPUs. That is where I got my numbers, and over 16% of all GPUs on Steam are Ampere GPUs.

[gardotd426]

There is a recent article on Phoronix.

NVIDIA Lands AV1 VDPAU Hardware Acceleration In FFmpeg

I built FFmpeg master (upcoming 5.1 release) and see av1_vdpau mentioned in the list.

Enabled hwaccels:
av1_nvdec               
av1_vaapi               
av1_vdpau
...             

@marioroy I'm also on the latest ffmpeg master, just built today, and I don't even get "Enabled hwaccels" in my output for ffmpeg -hwaccels. Is that the command you're running to get that information? Is there a specific flag you need to enable during compilation to have it show up?

Here's my output for ffmpeg -hwaccels:

Hardware acceleration methods:
vdpau
cuda
vaapi
qsv
drm

And I have a 3090, and Nvidia Control Panel obviously shows I have full AV1 decode support.

[marioroy]

@gardotd426 I have the same output for ffmpeg -hwaccels. The Enabled hwaccels: av1_{nvdec,vaapi,vdpau} is taken from the ./configure ... output. Sorry about the confusion. I often capture and search through the configure output to see what changed.

[gardotd426]

@gardotd426 I have the same output for ffmpeg -hwaccels. The Enabled hwaccels: av1_{nvdec,vaapi,vdpau} is taken from the ./configure ... output. Sorry about the confusion. I often capture and search through the configure output to see what changed.

Yeah I had just figured that out a little after posting, thanks for the clarification. Have you tested it? I don't have av1_vdpau as an available decoder, if I try to decode any AV1 video using -hwaccel vdpau -c:v av1_vdpau it says there is no such decoder. Could you give me your output of ffmpeg -decoders | grep -i av1? Also, could you try to decode/transcode an AV1 video and see if it actually uses the GPU (you can check GWE to see if the decoder % goes to something other than 0%) and if you're successful, give me the command you used?

If this actually works, it would seemingly be even more reason to at least attempt to add it to this project, since all future GPUs will support it, and as I stated earlier, Ampere GPUs currently make up 16% of the GPUs on the Steam hardware survey.

[marioroy]

Here is the output for ffmpeg -hide_banner -decoders | grep -i av1.

 V..... libdav1d             dav1d AV1 decoder by VideoLAN (codec av1)
 V....D libaom-av1           libaom AV1 (codec av1)
 V....D av1                  Alliance for Open Media AV1
 V..... av1_cuvid            Nvidia CUVID AV1 decoder (codec av1)
 V....D av1_qsv              AV1 video (Intel Quick Sync Video acceleration) (codec av1)
 A....D wmav1                Windows Media Audio 1

I tried decoding AV1 video using -hwaccel vdpau -c:v av1_cuvid but that segfaults. The following are the options that I tried.

1. -hwaccel vdpau -c:v av1         19.327s  GPU%  5  DEC% 48
2. -hwaccel nvdec -c:v av1          8.671s  GPU% 12  DEC% 31
3. -hwaccel vdpau -c:v av1_cuvid  segfault
4. -hwaccel nvdec -c:v av1_cuvid    6.080s  GPU% 18  DEC% 45

This is the command I use to monitor the GPU.

nvidia-smi pmon

[gardotd426]

So it does seem that the av1 support in ffmpeg is not functional. I get a segfault as well with vdpau, obviously nvdec works fine.

If you want to file a bug report with ffmpeg (especially since it's off topic for this thread but also if the bug gets fixed it could help this project as well), I'll gladly hop on there and report having the same results so they know it's multiple people with the issue.

[marioroy]

The same is true (segfaults) for -hwaccel vdpau -c:v vp9_cuvid. So, I tried -hwaccel vdpau -i input.webm without -c:v vp9_cuvid for VP9 input and that works to include processing on the GPU using the video engine. Unfortunately, not decoding on the GPU for AV1 input -hwaccel vdpau -i input.mp4.

ffmpeg -y -hwaccel vdpau -c:v av1_cuvid -i input_av1.mp4 output.ts   segfaults
ffmpeg -y -hwaccel vdpau -c:v vp9_cuvid -i input_vp9.webm output.ts  segfaults

ffmpeg -y -hwaccel vdpau -i input_av1.mp4 output.ts   decodes on the CPU
ffmpeg -y -hwaccel vdpau -i input_vp9.webm output.ts  decodes on the GPU

ffmpeg -y -hwaccel nvdec -c:v av1_cuvid -i input_av1.mp4 output.ts
ffmpeg -y -hwaccel nvdec -c:v vp9_cuvid -i input_vp9.webm output.ts

ffmpeg -y -c:v av1_cuvid -i input_av1.mp4 output.ts
ffmpeg -y -c:v vp9_cuvid -i input_vp9.webm output.ts

Yesterday, I simply posted the news about AV1. I sent an email to the ffmpeg-user mailing list.

[gardotd426]

Well a bug definitely needs to be filed, unfortunately I'm a member of the Regolith Organization and I am the sole developer and maintainer of the Regolith DE port to Arch Linux, and recently Regolith v2.0 released, which changed literally everything from the ground up about how it works, so reworking the PKGBUILDs and patches and whatnot is a massive undertaking and I'm already behind. That's why I said I'm happy to hop on a bug report, I'm just not able to file it myself. But if no one else will I guess I'll have to

[marioroy]

Hi @gardotd426. I received an explanation from Haihao at Intel regarding the cuvid decoders. So -c:v av1_cuvid and -c:v vp9_cuvid are not to be used with -hwaccel vaapi and -hwaccel vdpau.

I am not familiar with cuvid decoders, but according to https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/cuviddec.c#L1099-L1110, cuvid decoders use CUDA device. I guess you should use -hwaccel cuvid which is remapped to -hwaccel cuda in FFmpeg, see https://github.com/FFmpeg/FFmpeg/blob/master/fftools/ffmpeg_opt.c#L991-L994.

libavcodec/cuviddec.c L1099-L110

static const AVCodecHWConfigInternal *const cuvid_hw_configs[] = {
    &(const AVCodecHWConfigInternal) {
        .public = {
            .pix_fmt     = AV_PIX_FMT_CUDA,
            .methods     = AV_CODEC_HW_CONFIG_METHOD_HW_DEVICE_CTX |
                           AV_CODEC_HW_CONFIG_METHOD_INTERNAL,
            .device_type = AV_HWDEVICE_TYPE_CUDA
        },
        .hwaccel = NULL,
    },
    NULL
};

fftools/ffmpeg_opt.c L991-L994

            if (hwaccel) {
                // The NVDEC hwaccels use a CUDA device, so remap the name here.
                if (!strcmp(hwaccel, "nvdec") || !strcmp(hwaccel, "cuvid"))
                    hwaccel = "cuda";

The proper usage for decoding av1 (fyi: av1_cuvid works with -hwaccel cuda/nvdec only):

$ ffmpeg -y -hwaccel nvdec -c:v av1 -i input_av1.mp4 -f null -
$ ffmpeg -y -hwaccel nvdec -c:v av1_cuvid -i input_av1.mp4 -f null -  # faster
$ ffmpeg -y -hwaccel vaapi -c:v av1 -i input_av1.mp4 -f null -
$ ffmpeg -y -hwaccel vdpau -c:v av1 -i input_av1.mp4 -f null -

One day FFmpeg may default to av1(native) automatically. Haihao has created a patchset https://github.com/xhaihao/FFmpeg/tree/ffmpeg-choose-dec. Currently, FFmpeg defaults to libdav1d internally if omitting the -c:v av1 option.

Update August 4, 2022: The two patches df5defe and ad67ea9 were committed to FFmpeg master.