Rework hardware acceleration decoder selection

[lizardfish0]

Bit to explain here...

tl;dr, I needed a way to force the use of a specific hwaccel/decoder pair.

This PR addresses two problems:

1. Improve decoder selection logic to only use decoders that are included in hwaccel_decoders.

Currently, the script will attempt to use a decoder that matches <input codec>_<hwaccel name>, provided it exists in ffmpeg -decoders. The first and largest issue with this approach is that this codec might not actually be supported by the underlying hardware. For example, if the input codec is av1 and you've provided cuda as a hwaccel, then you'll need Nvidia 30-series or later to run the av1_cuvid decoder, but ffmpeg doesn't know that.

Additionally, the script will currently append the first valid (exists in ffmpeg) hwaccel it finds. In my case, even though my GPU didn't support av1, I'd like to use my CPU's iGPU to perform the decoding. This PR makes that possible.

2. Users might want to select a specific decoder that doesn't fit the <input codec>_<hwaccel name> mold. I added a new setting to override/manually control the process.

I wasn't aware of this, but when trying to solve my use-case I learned that there are internal ffmpeg codecs that support hardware acceleration, and there are external ffmpeg codecs specifically built for a single hardware platform. ffmpeg will implicitly use the internal codec unless you tell it otherwise.

i.e. you can run the implicit hevc decoder with -hwaccel cuda, or run -vcodec hevc_cuvid with -hwaccel cuda. I don't know too much about how these are maintained separately, but I read that sometimes there are differences in implementation that make one more efficient, so perhaps this will be useful to some.

---

Core problem: there is no good way to query ffmpeg to know if a given decoder is actually going to work with the provided hardware. The only one who knows whether it's going to work is the user.

Might be helpful to run through my situation to understand why this might be useful.

Hardware:

• GPU: 1650 Super w/ Turing NVENC/NVDEC
• CPU: i12400

Previously, I had

[Converter]
...
hwaccels = cuda
hwaccel-decoders = h264_cuvid, hevc_cuvid, mjpeg_cuvid, mpeg1_cuvid, mpeg2_cuvid, mpeg4_cuvid, vp8_cuvid, vp9_cuvid
hwdevices = 
hwaccel-output-format = cuda:cuvid
...
[Video]
codec = hevc_nvenc, hevc, h265, x265, h264, x264
...

This worked well until I ran into an AV1 file, which failed transcoding

[av1 @ 0x55e6c68fdd80] Hardware is lacking required capabilities
[av1 @ 0x55e6c68fdd80] Failed setup for format cuda: hwaccel initialisation returned error.
[av1 @ 0x55e6c68fdd80] Your platform doesn't support hardware accelerated AV1 decoding.
[av1 @ 0x55e6c68fdd80] Failed to get pixel format.

My iGPU supports AV1 decoding, so I figured I could wrangle the settings into performing the decode with that. However, the following fails because the script attempts to use cuda.

hwaccels = cuda, vaapi

Even with the change to filter by hwaccel_decoders, the script then searches for <input codec>_<hwaccel name> and there is no av1_vaapi. The solution would be to use -hwaccel vaapi with -vcodec av1.

vcodec would actually be optional due to ffmpeg using it implicitly

Thus the second change.

If you have a cleaner solution I'm happy to work it out, but I think this works pretty well. I explored modifying the way hwaccel_decoders works, where we could detect a listed decoder that wasn't valid according to ffmpeg but corresponded to an internal decoder with hardware acceleration. Adding a new setting to manually specify hwaccel/decoder pairings seemed much cleaner and less confusing to users.

[mdhiggins]

Reviewing this now

Question though, does using -hwaccel vaapi -vcodec av1 actually use your iGPU?

If your ffmpeg build doesn't have a vaapi decoder for av1 isn't it just falling back to software based on those options? Perhaps my understanding isn't correct but that was my assumption

Also, could you share the ffmpeg command generated for this output

[av1 @ 0x55e6c68fdd80] Hardware is lacking required capabilities
[av1 @ 0x55e6c68fdd80] Failed setup for format cuda: hwaccel initialisation returned error.
[av1 @ 0x55e6c68fdd80] Your platform doesn't support hardware accelerated AV1 decoding.
[av1 @ 0x55e6c68fdd80] Failed to get pixel format.

And additionally could you share the command generated with your fork?

Seems like I probably need to allow -hwaccel to be added if the decoder list is empty since some people still use that basic config option to get some generic hwaccel

[lizardfish0]

Question though, does using -hwaccel vaapi -vcodec av1 actually use your iGPU?

It does, confirmed with intel-gpu-top. I assume it would fall back on software but I believe that's the best option. Again it's a bit funky, av1_vaapi doesn't exist but you can run av1 with vaapi hwaccel.

Was about to grab you some ffmpeg outputs but it looks like something just changed with the sonarr-sma alpine container, and SMA_USE_REPO now installs a build of ffmpeg that doesn't have any hardware encoding support.

[mdhiggins]

Had a bug with the startup script on sonarr-sma that I just fixed like 2 minutes ago, might want to just do a fresh pull and try again

[mdhiggins]

Yeah upon reviewing the decoders, it looks like vaapi is the only one that doesn't respect the codec_hwaccel naming convention for the decoders (though it does for the encoders) Could probably hard-code that exception

[lizardfish0]

meh, I'm now an hour down a rabbit hole discovering why I have a half-finished attempt to put ffmpeg behind a thin http api so that we can abstract it from alpine-based images (maybe I'll revisit). My sonarr-sma image was in a weird state. I have no idea how I had a build of ffmpeg that supported qsv/vaapi/cuda/nvenc (I run a pull & up pretty regularly, maybe there was some weirdness in my docker stack left over from using the #build tag).

Regardless, off the top of my head I believe the script without any changes generated:

ffmpeg -hwaccel cuda -i av1_input.mkv -vcodec hevc_nvenc hevc_output.mkv

And now with the changes it generates

ffmpeg -hwaccel vaapi -vcodec av1 -i av1_input.mkv -vcodec hevc_nvenc hevc_output.mkv

Currently, the script will always set -hwaccel to the first entry in hwaccel that exists in ffmpeg. Definitely useful to be able to filter that selection by also requiring the decoder to exist in hwaccel_decoders. Hard-coding an exception to vaapi would solve the name matching issue, but I think there is still value in letting users force a vcodec (if they really wanted to use -hwaccel cuda -vcodec hevc instead of -hwaccel cuda -vcodec hevc_cuvid). Likely very rare that anyone would use it, but it provides a generic way to answer for edge cases.

[lizardfish0]

Thoughts on adding support for https://github.com/jellyfin/jellyfin-ffmpeg? Looks like a great source of pre-compiled ffmpeg binaries w/ hwaccel support.

edit: I mean this specifically in reference to the docker containers, I can open a corresponding PR there if you're interested.

[VampiricAlien]

@lizardfish0 I use those builds myself but when it comes to the SMA mod, they can't be used with Alpine linux.