FFmpeg CUDA support

[paulcarroty]

⬇️

[marioroy]

Remaining tasks:

• [x] nv-codec-headers (dependency for ffmpeg -hwaccel nvdec and the NVDEC back-end VA-API driver)
• [x] ffmpeg-cuda built with ./configure --enable-nvdec
• [x] mpv-cuda built with ./waf configure --enable-cuda-hwaccel
• [x] nvidia-vaapi-driver (NVDEC back-end VA driver used by Firefox, FFmpeg, Mpv)
• [x] vdpau-va-driver-vp9 (used by Brave and Chrome-like browsers for hardware acceleration: Clear Linux Core provides libvdpau dependency)
• [x] update mass-rebuild
• [x] update bundles

[marioroy]

I have a question. There is "Requires: %{name}-filemap" in the ffmpeg.spec file. Where is ffmpeg-filemap defined or is it external? Well, I created ffmpeg-cuda/ffmpeg-cuda.spec based on your recent 5.1 ffmpeg/ffmpeg.spec.

No write access at the moment.

[paulcarroty]

no write access to ffmpeg-cuda.

Try again.

Where is ffmpeg-filemap defined or is it external?

Fixed. This is for clr-elf-replace, I tried to setup AVX2 builds but had not enough free time to finish it.

[marioroy]

Fixed.

The push succeeded.

[marioroy]

How does one install a 3rd-party bundle containing ffmpeg-cuda?

codecs: ffmpeg, ffmpeg-libs codecs-cuda: ffmpeg-cuda, ffmpeg-cuda-libs

sudo swupd 3rd-party bundle-add codecs (or) sudo swupd 3rd-party bundle-add codecs-cuda

Are the ffmpeg-cuda-*rpm (packages) temporary? I do not know your plan :)

[marioroy]

Please feel free to disregard the following:

I wish to deprecate the scripts folder. Being able to install codecs-cuda will immensely elevate the experience for folks using NVIDIA graphics.

Completed: nv-codec-headers that provides ffmpeg -hwaccel nvdec capability.

Remaining tasks:

1. libvdpau (dependency for the VDPAU back-end VA-API driver)
2. vdpau-va-driver-vp9 (used by Brave and Chrome-like browsers for hardware acceleration)
3. vaapi-nvidia (NVDEC back-end VA driver used by Firefox, FFmpeg, Mpv)

Simulation for folks using NVIDIA graphics.

sudo swupd 3rd-party bundle-add codecs-cuda

Brave and Chrome-like browsers

export LIBVA_DRIVERS_PATH=/opt/3rd-party/lib64/dri  (or)
export LIBVA_DRIVERS_PATH=/opt/3rd-party/bundles/clearfraction/usr/lib64/dri
export LIBVA_DRIVER_NAME=vdpau

Firefox, FFmpeg -hwaccel vaapi, and Mpv (OBS too)

export LIBVA_DRIVERS_PATH=/opt/3rd-party/lib64/dri  (or)
export LIBVA_DRIVERS_PATH=/opt/3rd-party/bundles/clearfraction/usr/lib64/dri
export LIBVA_DRIVER_NAME=nvdec (or nvidia linked to nvdec)

FFmpeg

ffmpeg -hwaccel vaapi -c:v {vp9, av1}
ffmpeg -hwaccel nvdec -c:v {vp9, vp9_cuvid, av1, av1_cuvid}
Since there is libvdpau, I can add --enable-vdpau option to configure
ffmpeg -hwaccel vdpau -c:v {vp9, av1} 

[marioroy]

Plus somewhere in the Clear Fraction site or documentation area, a guide on how to configure the Firefox configuration file ~/.config/firefox.conf so that mindful of Xorg and Wayland. Also mindful of Intel Graphics, NVIDIA graphics, etc.

Ditto for Brave and Chrome-like browsers. See the brave and firefox folders in HWAccel.

That's all! Again, please feel free to disregard this.

[marioroy]

Info: The next FFmpeg release 5.2 will default to running on the GPU if the capability exist in hardware. Currently, decoding AV1 defaults to -c:v libdav1d when omitting the -c:v option.

See FFmpeg commits df5defe and ad67ea9.

[marioroy]

If helpful and when the time comes, I can have the install-driver script create symbolic links for:

 /usr/lib64/dri/{nvdec,nvidia,vdpau}_drv_video.so

to:

 /opt/3rd-party/bundles/clearfraction/usr/lib64/dri/{nvdec,nvidia,vdpau}_*

Folks will then not need to configure LIBVA_DRIVERS_PATH. For swupd safety, the pre-install-driver script creates /etc/swupd/config.

/usr/lib64/dri/(nvdec|nvidia|vdpau)_drv_video\.so

[paulcarroty]

Ohhh.

How does one install a 3rd-party bundle containing ffmpeg-cuda?

All bundles are defined here.

Are the ffmpeg-cuda-*rpm (packages) temporary

Yes, mass-rebuild cares about releases; GitHub artifacts will be wiped after 90 days.

Environment variables are defined in /distibution/README.md, /docs required. I'll create the repos^ above if you wanna maintain them and add minimal docs for the scripts.

UPD: nuked second post & inserted task-list. You need the mpv-cuda too, maybe.

[marioroy]

I completed testing the SPEC file libvdpau.spec for libvdpau. The libvdpau package is a dependency for building the VDPAU back-end VA-API driver which is next on my list. In the meantime, I tested VDPAU capability in FFmpeg by updating the ffmpeg-cuda.spec file locally. libvdpau 1.5 works great.

Requires:       libvdpau
BuildRequires:  libvdpau-dev

configure --enable-vdpau

Note: The next FFmpeg 5.2 release will default to -c:v av1 automatically on capable GPUs. Currently, AV1 renders via libdav1d when omitting the -c:v av1 option.

$ ffmpeg -hwaccels
Hardware acceleration methods:
vdpau
cuda
vaapi
qsv
drm
vulkan

$ ffmpeg -hwaccel vdpau -c:v vp9 -i input_vp9.webm -f null -  # GPU
$ ffmpeg -hwaccel vdpau -c:v av1 -i input_av1.mp4 -f null -   # GPU

From FFmpeg HWAccelIntro, "Several decoders are currently supported through VDPAU in libavcodec, in particular H.264, MPEG-1/2/4, and VC-1, AV1."

See also https://www.phoronix.com/news/NVIDIA-AV1-VDPAU-FFmpeg

[paulcarroty]

Check invite to libvdpau.

AV1 renders via libdav1d

'Cause Clear provide only the one AV1 decoder AFAIR.

[marioroy]

inserted task-list

Thank you! Pushed libvdpau.spec.

[marioroy]

All bundles are defined here.

Should I see ffmpeg-cuda in the list? I will remove 3rd-party and try again.

$ sudo swupd 3rd-party update
$ sudo swupd 3rd-party bundle-list -a
_____________________________________
 3rd-Party Repository: clearfraction
_____________________________________

All available bundles:
 - alacritty
 - brave
 - btop
 - codecs
 - fluffychat
 - foliate
 - imv
 - kitty
 - os-core
 - secrets
 - shotwell
 - sway-extras
 - vscodium
 - waybar

Total: 14

[paulcarroty]

Should I see ffmpeg-cuda in the list?

After new release, not yet.

[marioroy]

I pushed a hot fix to add %define abi_package %{nil} in libvdpau.spec. But the build continues to fail. Is this due to stale *.abi files?

error: Installed (but unpackaged) file(s) found:
[530](https://github.com/clearfraction/libvdpau/runs/7775980932?check_suite_focus=true#step:3:531)
   /usr/share/abi/libvdpau.so.1.abi
[531](https://github.com/clearfraction/libvdpau/runs/7775980932?check_suite_focus=true#step:3:532)
   /usr/share/abi/vdpau/libvdpau_trace.so.1.abi
[532](https://github.com/clearfraction/libvdpau/runs/7775980932?check_suite_focus=true#step:3:533)
    Downloading https://gitlab.freedesktop.org/vdpau/libvdpau/-/archive/1.5/libvdpau-1.5.tar.bz2 to /home/libvdpau-1.5.tar.bz2
[533](https://github.com/clearfraction/libvdpau/runs/7775980932?check_suite_focus=true#step:3:534)
    Installed (but unpackaged) file(s) found:
[534](https://github.com/clearfraction/libvdpau/runs/7775980932?check_suite_focus=true#step:3:535)
   /usr/share/abi/libvdpau.so.1.abi
[535](https://github.com/clearfraction/libvdpau/runs/7775980932?check_suite_focus=true#step:3:536)
   /usr/share/abi/vdpau/libvdpau_trace.so.1.abi
[536](https://github.com/clearfraction/libvdpau/runs/7775980932?check_suite_focus=true#step:3:537)
Build failed

[marioroy]

I'm ignoring the abi issue for now so able to move forward. The libvdpau spec file seems fine and has all the proper build requires and requires.

%define _unpackaged_files_terminate_build 0

warning: Installed (but unpackaged) file(s) found:
   /usr/share/abi/libvdpau.so.1.abi
   /usr/share/abi/vdpau/libvdpau_trace.so.1.abi

[marioroy]

I completed mpv-cuda.

Thus far, the workflow isn't installing the build dependencies. I ensured the following exists in the mpv-cuda spec file.

Requires: ffmpeg-cuda-libs
BuildRequires : nv-codec-headers
BuildRequires : ffmpeg-cuda-dev

Testing: The mpv binary utilizes the NVIDIA video engine while playing videos :)

mpv --hwdec={nvdec,cuda}
mpv --hwdec defaults to nvdec

[vd] vp9_cuvid (vp9) - Nvidia CUVID VP9 decoder
[vd] av1_cuvid (av1) - Nvidia CUVID AV1 decoder

[marioroy]

My bad... The repository name vaapi-nvidia was met to be nvidia-vaapi-driver. Can you rename it please. Blessings and grace.

[paulcarroty]

I'm ignoring the abi issue for now so able to move forward. The libvdpau spec file seems fine and has all the proper build requires and requires.

%install
rm -rf %{buildroot}/usr/share/abi

Thus far, the workflow isn't installing the build dependencies

https://github.com/clearfraction/ffmpeg-cuda/commit/9fbe1adcab430f25f635752c1e8675c1f73295fd

[marioroy]

mpv-cuda: It cannot find ffmpeg-cuda or something else.

[marioroy]

Putting back %define _unpackaged_files_terminate_build 0 in libvdpau.spec so that I can complete the VDPAU back-end VA driver. The other NVDEC back-end VA driver completed workflow successfully.

[marioroy]

I finished vdpau-va-driver-vp9, but do not have write access.

[paulcarroty]

mpv-cuda: It cannot find ffmpeg-cuda or something else.

'Cause wrong workflow, should be build.yml, build-core.yml for compilation with Clear dependencies only.

[marioroy]

The nv-codec-headers entry is not needed inside the codecs-cuda bundle, but no problem if you want to keep it there.

I wondered myself and tested to be sure. The binaries FFmpeg, Mpv, and the NVDEC VA driver work without the headers, confirming nv-codec-headers lives in the BuildRequires world and not used otherwise.

[paulcarroty]

Fine, remove the headers.

[paulcarroty]

@marioroy https://github.com/clearfraction/bundles/runs/7821549270?check_suite_focus=true#step:4:232

[marioroy]

@paulcarroty I'm not sure that I understand what's going on. Do we need bundles and bundles-cuda? Because, the files in the ffmpeg-cuda package conflict with ffmpeg. Likewise, the files in the mpv-cuda package conflict with mpv.

If preferring not to have both bundles and bundles-cuda, another path is to have one ffmpeg and mpv supporting cuda. The dependency is simply nv-codec-headers. I can test the binaries on Intel without NVIDIA graphics.

[marioroy]

• [x] Test FFmpeg and Mpv binaries built with nv-codec-headers on an Intel box lacking NVIDIA graphics.

[marioroy]

The ffmpeg and mpv binaries built with nv-codec-headers work on an Intel box lacking NVIDIA graphics. Below is the intel_gpu_top screenshot while playing media using Mpv.

$ mpv --hwdec input_avc1.mp4

Next, is the intel_gpu_top screenshot while decoding the same avc1 media with FFmpeg.

$ ffmpeg -hwaccel vaapi -i input_avc1.mp4 -f null -

Next, is ffmpeg -hwaccels output. Notice that all the same acceleration methods like before. Welcome the new cuda acceleration method as FFmpeg was built with nv-codec-headers.

$ ffmpeg -hwaccels
Hardware acceleration methods:
cuda
vaapi
qsv
drm
vulkan

[marioroy]

I created a repository for testing yt-dlp and mpv. This is how I verified video acceleration working with mpv. The great news is that we only need one set of binaries ffmpeg and mpv, built with nv-codec-headers.

https://github.com/marioroy/youtube-play

Let me know, I can update ffmpeg and mpv to include nv-codec-headers support. Then, ffmpeg-cuda and mpv-cuda are not needed.

[paulcarroty]

I'm against shipping cuda for all 'cause it will fix the case, but not the problem. Need to do a research how Clear handle cases like this with kernels, GCC, LLVM etc.

[paulcarroty]

They're renaming ELFs and library paths: https://github.com/clearlinux-pkgs/gcc11/blob/main/gcc11.spec#L284

Yeah, I know you don't like it 😀

Going to ship this release without CUDA and we'll figure it out later.

[paulcarroty]

Ideas: 1) Play with buildsystems like above with flags --program-suffix 2) Tuning global paths like /opt/3rd-party/bundles/clearfraction/cuda together with $PATH & $LD. Seems easier.

[marioroy]

Great! Trying option 2 { ffmpeg-cuda, mpv-cuda, nvidia-vaapi-driver, vdpau-va-driver-v9 }.

%define cuda_prefix /cuda/usr
%define cuda_bindir /cuda/usr/bin
%define cuda_includedir /cuda/usr/include
%define cuda_libdir /cuda/usr/lib64
%define cuda_datadir /cuda/usr/share

[paulcarroty]

Hold on, only ffmpeg-cuda & mpv-cuda.

[paulcarroty]

Alternative path: /usr/local, already in CF environment.d config, should work out the box.

[marioroy]

Great! Changing to /usr/local.

%define cuda_prefix /usr/local
%define cuda_bindir /usr/local/bin
%define cuda_includedir /usr/local/include
%define cuda_libdir /usr/local/lib64
%define cuda_datadir /usr/local/share

[marioroy]

It appears that rpmbuild (internally) is changing /usr/local to /usr :( There are no macros either to prevent that.

[paulcarroty]

%make_install PREFIX=/usr/local

Do you also change %files?

If this will be also ignored, we can use a hack on bundles generation stage.

[marioroy]

%make_install PREFIX=/usr/local not working either.

https://stackoverflow.com/questions/72203988/rpm-macros-for-usr-local-prefix

no Fedora package can contain files or directories or modify files under ... /usr/local as these directories are not permitted to be used by Distributions in the FHS

[marioroy]

If this will be also ignored, we can use a hack on bundles generation stage.

One installs either codecs or codecs-cuda. But not both -- so no files conflicts. Or in the event an existing codecs installation exist, it requires removing the existing codecs installation before installing the other.

[marioroy]

Simulations if having separate bundles ffmpeg and ffmpeg-cuda.

1. Intel-based notebooks with integrated NVIDIA graphics.
2. Intel-based desktops with dedicated NVIDIA graphics.
3. Two separate installations ffmpeg and ffmpeg-cuda (different paths). Will user need to fiddle with environment variables? Which binary is used when typing ffmpeg, mpv?

I respect your decision to keep them separate. But, what a wonderful experience if one bundle (e.g. one path, no fiddling with env, no concerns installing or updating codecs), no worries if codecs exists that must be removed prior to installing the other or vice versa.

Yikes, IMHO.

[marioroy]

nv-codec-headers is a build dependency and not in the final bundle. So no difference between ffmpeg, ffmpeg-cuda and mpv, mpv-cuda with regards to the list of files. The extra files are nvidia-vaapi-driver, vdpau-va-driver-vp9, and dependencies. Should these be placed in another bundle "libva-nvidia-drivers" or "nvidia-va-drivers"?

[paulcarroty]

I vote for two bundles 'cause more control: we don't need to play with FFmpeg versions when CUDA will stop working, plus easy to drop and easy to pick it up. If you can't build ffmpeg & mpv with /usr/local profile, I hope to do it tomorrow.

Intel-based notebooks with integrated NVIDIA graphics.

Optimus? This crap should die in fire.

So no difference between ffmpeg vs ffmpeg-cuda and mpv vs mpv-cuda with regards to the list of files

ELFs are different, the file names and paths are the same.

"libva-nvidia-drivers" or "nvidia-va-drivers"

All up to you.

[marioroy]

The /usr/local option is working :)

%define cuda_prefix /usr/local-cuda
%define cuda_bindir /usr/local-cuda/bin
%define cuda_includedir /usr/local-cuda/include
%define cuda_libdir /usr/local-cuda/lib64
%define cuda_datadir /usr/local-cuda/share

%install (adding)
mv %{buildroot}%{cuda_prefix} %{buildroot}/usr/local
pushd %{buildroot}/usr
ln -s local local-cuda
popd

%files libs (adding)
/usr/local-cuda

I will set LDFLAGS ensuring not searching for ffmpeg libs elsewhere. e.g. installing both codecs and codecs-cuda. local-cuda is a symbolic link to local in the ffmpeg-cuda-libs package.

export LDFLAGS="-Wl,-rpath=/usr/local-cuda/lib64,-rpath=/opt/3rd-party/bundles/clearfraction/usr/local-cuda/lib64,-rpath=/opt/3rd-party/bundles/clearfraction/usr/lib64"

[marioroy]

Setting LDFLAGS with -Wl,-rpath=... is working. Ensuring the ffmpeg-cuda binaries and libraries search for the ffmpeg-cuda libs first, regardless of LD_LIBRARY_PATH setting. :)

[marioroy]

I modified the LDFLAGS above ensuring simulations passing. Well, here it is.

export LDFLAGS="-Wl,-rpath=/usr/local-cuda/lib64,-rpath=/opt/3rd-party/bundles/clearfraction/usr/local-cuda/lib64,-rpath=/opt/3rd-party/bundles/clearfraction/usr/lib64"

Given Simulations: PASS

1. User compiles FFmpeg from source and installs to /usr/local, libs residing in /usr/local/lib64. Does not affect the user's built FFmpeg or the 3rd-party codecs-cuda. Because, the local-cuda symbolic link resides under the /opt/3rd-party tree, not in /usr.
2. User installs both 3rd-party codecs and 3rd-party codecs-cuda. Does not affect the 3rd-party codecs-cuda as it will search the local-cuda paths first.
3. The 3rd-party environment vars has /opt/3rd-party/.../usr/lib64 first before /opt/3rd-party/.../usr/local/lib64. Does not affect the 3rd-party codecs-cuda as it will search the local-cuda paths first. Same behavior as simulation 2.
4. Setting LD_LIBRARY_PATH explicitly. Does not affect codecs-cuda. Same behavior as simulation 2.
5. Finally, myself building ffmpeg-cuda (/usr/local-cuda is a symbolic link to /usr/local) and searched first before /opt/3rd-party/... Does not affect testing updates to ffmpeg-cuda.
6. There's one more. LD_LIBRARY_PATH='' or unset. Does not affect codecs-cuda due to LDFLAGS -Wl,-rpath...

The mpv-cuda spec file has the same LDFLAGS. Also, PKG_CONFIG_PATH.

export PKG_CONFIG_PATH="/usr/local-cuda/lib64/pkgconfig"

[marioroy]

The updates were pushed for ffmpeg-cuda and mpv-cuda. Each workflow completed successfully. Moreover, ensured (qsv, vaapi, nvdec, and vdpau) hardware acceleration with https://github.com/marioroy/youtube-play

"VDPAU on Radeon R300 and newer GPUs is supported by mesa-vdpau." -- Arch Linux Wiki

The codecs-cuda bundle supports AMD (vaapi,vdpau), Intel (qsv,vaapi), and NVIDIA (nvdec,vaapi,vdpau).

[paulcarroty]

Great job! Parallel installation is the huge achievement.

Now wait a stable Clear release > 36820 and trigger mass-rebuild manually. Two test CL releases this weekend, I wonder what happened.