QSV works with HEVC but not h264 on Ubuntu

[wintervaler]

Don, and the hivemind - thanks first of all for your amazing and instructive work and the time you put into these tools. I am also very sorry in advance for the lengthy nature of this issue - I want to cover my bases. I’ve been wrestling with this for weeks now and so this is a last resort - I realize I may actually be on a fool’s errand but want to check with folks here before I give it up as a lost cause.

The long story short is that while HEVC transcoding on my Ubuntu box using QSV works just fine, the h264 QSV encoder cannot initialize unless it reverts to VAAPI, which I understand to be of lesser quality than proper QSV. For background read on:

I’m running Ubuntu 19.04 on an HP box with an 8th gen Intel CPU. I have ffmpeg built to utilize the Intel Media SDK as prescribed on the SDK’s Github (I’m building ffmpeg from source; but using the Ubuntu repositories to install the media SDK and the libmfx library - building the whole stack from scratch seemed like a good way to break my system, but that may end up being the answer here). I am trying to get other-transcode to properly recognize QSV as opposed to reverting to the open-source VAAPI driver, which has less tuneability and potentially slower throughput as I understand it.

When I try to use other-transcode without passing any encoder options, it automatically detects the VAAPI hardware acceleration mode (though it tries QSV first), even though when I run a list of ffmpeg encoders and decoders h264_qsv is indeed an option. When I pass the --qsv option in other-transcode to try to force it to use that, I get an error:

Transcoding...
[h264_qsv @ 0x558221d52500] Error initializing the encoder: unsupported (-3)
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
/usr/local/bin/other-transcode: transcoding failed: Harry Potter and the Deathly Hallows.mkv

The full ffmpeg command line output with verbose logging is linked here: https://drive.google.com/open?id=1x54RUFbCyBJtAbD27h1bDj6AmieLwi_v

The odd thing is that if I use other-transcode with the --hevc, everything works just fine - it detects the QSV encoder on the first try, without using --qsv, and uses it to complete the task.

After some experimenting, which I did before even knowing that --hevc works just fine, it seems that the h264 encoder will initialize when I run the full ffmpeg string in the command line without look_ahead:v 1 or, alternatively, when I set look-ahead:v to 0 instead of 1 (same thing as not passing it all, if I understand correctly from the documentation). I also took a look at the command line output when I pass --hevc and it looks like it doesn’t use lookahead rate control, instead using QSV’s VBR mode - if I’m reading it right - so that may also explain why HEVC encoding works fine.

So my question is twofold -

1. What the heck is going on that I can’t get look_ahead to work under Ubuntu? This is where I worry I may be on a fool’s errand - as both your documentation AND ffmpeg's documentation says, mapping commands to the Intel driver is very difficult on Linux platforms and some things just may not work. But does anyone have any ideas?

2. If I can’t get this to work - how much quality am I really losing by using VAAPI for my h264 transcodes? Obviously I hope to use --hevc more going forward but some of the devices I’m streaming to are still less than friendly to newer formats, and I've heard that transcoding 10-bit HEVC on the fly gives Plex some trouble with HW acceleration. Alternatively, are there other ffmpeg command line options I could use to approximate look-ahead in quality? Say, specifying a maxrate or Global_Quality?

I’m sure that in my frustration I’ve missed providing some useful information that could possibly help answer my question, so I’m happy to provide more data if that’s helpful… Thanks again.

[lisamelton]

@wintervaler My sincere apologies for taking so long to respond to your issue! But I never received the notification from GitHub on it and it just appeared in my issues list this morning. This has happened once before so it's not something either you or I did but some weird brain fart by GitHub. Anyway...

Thanks so much for using my tools! And I'm glad they're (mostly) working for you. I'm especially glad that HEVC encoding with QSV works.

I don't know why passing -look_ahead:v 1 should be such a problem on Linux but maybe @ttys0, @samhutchins or @martinpickett of the HiveMind™ have some ideas on that. "Help me, Obi-Wan Kenobi. You're my only hope."

As to the quality of using the VAAPI API for H.264 output, I'm told that it's still very good. But because I don't have any samples myself, I can't say for sure.

Passing -look_ahead:v 1 to the h264_qsv encoder selects the "LA (VBR)" ratecontrol system within QSV, which is the highest quality algorithm in that GPU's repertoire.

My guess is that using VAAPI selects the "AVBR" ratecontrol system within QSV because that's the default if you only specify a target bitrate and not a maximum bitrate (which isn't possible using VAAPI anyway).

The good news is that the QSV "AVBR" algorithm is the next highest quality algorithm. And it, again, is still very good.

There's a --vaapi-compression option in other-transcode (found in --help full) if you want to tinker with how that might affect quality. Other than that, VAAPI is pretty much a magical black box.

[skj-dev]

From the FFmpeg wiki, it looks like the libmfx library on Linux only supports a subset of features, so you might be running into that :

https://trac.ffmpeg.org/wiki/Hardware/QuickSync

libmfx on Linux This is a library from Intel which can be installed as part of the Intel Media SDK, and supports a subset of encode and decode cases.

From the little bit of investigation I did a while back with QSV + Linux, it always came back around to something along the lines of "if you value your sanity, just use VAAPI". Others that dug into the whole mess deeper than I might have better explanations.

I realize the "oh, then don't do that" isn't that helpful, and am only including it since that was the verdict I ended up applying for my stuff. 😁

[wintervaler]

Thank you both SO much for your comments and at least it has validated that I'm not completely overlooking something. (My account was flagged with GitHub when I created it which may explain the lack of notifications - I'm glad you saw it eventually).

@ttyS0 I have indeed done that little bit of investigation and more for the past month or so and agree that it is headache inducing, though -- I will say -- with the addition of the libmfx library to Ubuntu repositories things seem to have gotten at least a little easier. But clearly not all the kinks worked out yet! Would definitely welcome thoughts from anyone who has dug deeper, as you say.

Sounds like I'll keep using VAAPI for h264 transcodes - or maybe boot into Windows (ugh) when I need to do them - and just keep trying to push the envelope on HEVC. (Thanks @donmelton for that tip about using --vaapi-compression -- didn't know about that and will be checking it out!)

[lisamelton]

@wintervaler Good to understand the problem with GitHub now. And I'm so glad they unflagged and enabled your account!

I will let @ttyS0 elaborate on any excavations he's done...

[skj-dev]

I don’t have anything else to add. I’m keeping an eye on some of the FFmpeg container chatter. If there is a breakthrough with QSV I would expect to see something pop up there. Till then I’m sticking with VAAPI, and my sanity thanks me. 😁

[lisamelton]

@ttyS0 Thanks! And seems like a good plan.

@wintervaler I'm going to close this for now because I'm not sure how much we can do at this point. Feel free to keep commenting here or open new issues.

[martinpickett]

Sorry, I have completely missed this issue until I noticed @donmelton close it!

In the early (pre-release) days of other-transcode, I successfully built a version of FFmpeg which supported direct QSV transocding on Ubuntu 18.04. The process was slow, difficult and buggy, but it did work. Additionally, my system has a Skylake generation CPU whose HEVC support is dubious. Due to the difficulty of installation, I never suggested to @donmelton that he support direct QSV on Linux and so we ended up with VAAPI.

My memory is a bit hazy, but I do recall some issue around the look_ahead implementation, but I really cannot tell you if it was the same as what @wintervaler has experienced. My recollection is it was death by a thousand paper cuts. A lot of settings which worked with QSV in Windows either did not work in Linux or required lots of extra commands. However, there were features of QSV on Linux which did not exist on Windows, one such feature was hardware accelerated subtitle burning (the overlay filter in FFmpeg parlance). I also seem to remember that QSV was slower than VAAPI, but you might expect that if it produces higher quality results.

Looking forwards, I think the work @ttyS0 has done creating a Docker container for other-transcode is great and if it can be extended to include support for true QSV then that is the way to go. I do not think we can expect people to compile their own version of FFmpeg. However, even that does not resolve the issue of FFmpeg based QSV commands differing from Windows to Linux. For that we either need to wait for parity between the OS's, or other-transcode needs to grow more complex.

[lisamelton]

Thanks, @martinpickett!

[wintervaler]

Thank you all so much for your comments and I'm glad this is also vexing to the folks here. I'd have been embarrassed and ego-bruised otherwise!

I will keep tinkering (because obviously I will) and report back if I have any breakthroughs. If anyone has any recommendations for where to keep an eye/do further research on this kind of thing in either in ffmpeg circles or here, I'm all ears!

[lisamelton]

@wintervaler You are very welcome! Keeping track of this stuff is difficult.