custom ffmpeg VAAPI encoder arguments not properly preconfigured, thus capture fails

[aufkrawall]

Expected Behavior

When selecting h264_vaapi (and any other codec) as encoder for ffmpeg custom output, it should work without further doing, just like it does with libx264.

Current Behavior

When selecting h264_vaapi, recording just fails with "Failed to open video codec: invalid argument."

Steps to Reproduce

Start any capture with the aforementioned encoder config.

Additional information

It would be more attractive to instead stick with the easy to use non-custom "FFMPEG VAAPI" encoder, but its UI unfortunately punishes users to stick with the ancient H.264 codec, unlike integrated NVENC and AMF encoders which offer also HEVC.

[kkartaltepe]

Custom ffmpeg is designed to allow you to do anything you can do in ffmpeg. This includes inputing invalid settings. Many ffmpeg codecs require configuration before they will work and this is the same in OBSs custom ffmpeg output and the ffmpeg command line, as such this seems working as intended?

Since you are recording instead of streaming, I recommend you reencoding your videos for archival.

[kkartaltepe]

As for

punishes users to stick with the ancient H.264 codec, unlike integrated NVENC and AMF encoders which offer also HEVC.

Discounting the misinformation about NVENC in OBS and the historical nature of AMF (and ignoring QSV). You can see the more authoritative answer on the topic of HEVC at https://github.com/obsproject/obs-studio/pull/2246#issuecomment-569532450 where you were already involved in the conversation.

[aufkrawall]

My usecase for example is uploading captured videos to YouTube. For this I would like to have the best compression to reduce upload time and required disk space.

Edit: While it's arguable that it's not a bug, I'd still be happy if it was documented how to configure the custom ffmpeg encoder option. E.g. modern Intel IGPs since Gen 9.5 can encode VP9 in hardware and it works perfectly fine in ffmpeg.

[Fenrirthviti]

ffmpeg documentation can be found here: https://ffmpeg.org/ffmpeg.html

We have no interest in further documenting ffmpeg inside our wiki or elsewhere, and frankly, custom ffmpeg mode was a concession and is more or less unsupported due to extremely limited and niche use cases.

[GloriousEggroll]

I believe the issue here is that the video codec arguements are only applied to the video codec, since the hardware options need to come before that, they aren't added properly.

With the ffmpeg custom option, one major use case for this is for use with DaVinci Resolve. DR on linux does not support AAC audio, so when recording in OBS specifically for DR compatibility, we are forced to use the ffmpeg custom tab with pcm_s24le audio or some other compatible codec, and standard libx264 cpu encoding. The problem here is then we cannot use hevc_vaapi or h264_vaapi for the video encoding since we cannot specify the hardware arguments.

[ryzendew]

We need a way on standard to choose our audio codec please! with resolve and a proper workflow it's needed badly

[kkartaltepe]

I believe the issue here is that the video codec arguements are only applied to the video codec, since the hardware options need to come before that, they aren't added properly.

Im not sure what you mean by this. The original issue was that ffmpeg_custom required additional options to successfully start, which will always be the case unless we implement all of the codec/container parameter handling of the ffmpeg CLI which i doubt anyone is interested in doing and maintaining.

With the ffmpeg custom option, one major use case for this is for use with DaVinci Resolve. DR on linux does not support AAC audio, so when recording in OBS specifically for DR compatibility, we are forced to use the ffmpeg custom tab with pcm_s24le audio or some other compatible codec, and standard libx264 cpu encoding. The problem here is then we cannot use hevc_vaapi or h264_vaapi for the video encoding since we cannot specify the hardware arguments.

I dont follow how the video and audio codec parameters are related (other than potentially by container compatibility issues in ffmpeg/davinci).

[GloriousEggroll]

Im not sure what you mean by this. The original issue was that ffmpeg_custom required additional options to successfully start, which will always be the case unless we implement all of the codec/container parameter handling of the ffmpeg CLI which i doubt anyone is interested in doing and maintaining.

when using ffmpeg custom and selecting a video codec from the drop down, you can usually run them as-is without additional options necessary for the codec (such as selecting standard libx264). However, when you select hevc_vaapi or h264_vaapi, these two require hardware options to be specified before the video codec is specified, such as which hardware vaapi device to use. Since you can only provide video codec options into the argument text box for the video codec, any hardware options placed there are invalid since they are not in the correct order. With no hardware options specified, the vaapi codecs fail.

I dont follow how the video and audio codec parameters are related (other than potentially by container compatibility issues in ffmpeg/davinci).

If you use the FFMPEG VAAPI encoder option, instead of using custom ffmpeg, this allows you to record using vaapi, as the device is then automatically configured. However, this does not allow you to change the audio format, thus forcing the default AAC codec to be used. While this is fine for simple single audio track recordings, AAC does not support encoding into multiple audio tracks, and is not able to be used at all in Davinci Resolve (They outright do not support the codec in any way). This means while we are able to utilize vaapi gpu encoding, we are stuck with AAC audio only. The end result is that the video recorded using FFMPEG VAAPI cannot be used in Resolve without first re-encoding the audio to a usable format, and even then all of the tracks defined in OBS are mashed into one, which completely defeats the purpose of having multiple audio tracks configured. The FFMPEG VAAPI tab is fine for streaming, but not a viable option for recording due to this.

Say for example I have a mixer and a microphone defined on two separate tracks.

If I use FFMPEG VAAPI, great I can encode the video with my gpu, but both my tracks get smashed into one AAC track. So I can't go back and remove my voice track in recordings.

If I use custom ffmpeg, great, I can now change my audio to allow multiple tracks via pcm or some other codec, however now I cannot encode the video portion using vaapi for my gpu, because I cannot properly define the required hardware arguements for ffmpeg.

[kkartaltepe]

So if I understand, you want to be able to configure the equivalent of hwaccel_device or vaapi_device which you cannot do via the video codec parameters currently as its required for ffmpeg's vaapi codecs?

[GloriousEggroll]

So if I understand, you want to be able to configure the equivalent of hwaccel_device or vaapi_device which you cannot do via the video codec parameters currently as its required for ffmpeg's vaapi codecs?

Bingo, precisely.

[Bleuzen]

Any news on this? Would love to record with vp9_vaapi and opus, but this currently doesn't work :/

[ryzendew]

Any news on this? Would love to record with vp9_vaapi and opus, but this currently doesn't work :/

Don't we all but doubt they will do anything

[BurnerWah]

I did plenty of testing and experimentation, and short of making a weird fork of ffmpeg that has a hardcoded hardware acceleration device, or making a fork of obs studio, there isn't a way to get things working.

For me, it doesn't matter that much. I'm already using the patch from #2246 which adds a ui for hevc vaapi. It only covers recording, but it's better than nothing. (everything i needed was just in the obs-hvec-vaapi-git package on aur - https://aur.archlinux.org/packages/obs-hevc-vaapi-git/)

Though it's definitely a little annoying that the solution the obs team is going with here reads off a bit like "there is no issue", as opposed to even a won't fix or something.

Oh and just in case, here's the documentation for vaapi encoders. https://ffmpeg.org/ffmpeg-codecs.html#VAAPI-encoders

[Fenrirthviti]

For this original issue, further context is that we just simple expose everything that ffmpeg reports as supported codecs, and do no validation if those options are correct or will work. Adding a complicated "sanity check" matrix to the ffmpeg output is what we are not interested in doing.

I don't know why you'd need to make a fork when you could just open a PR for the changes required, however. #2246 was rejected because it's HEVC specifically, and we're not really interested in pushing that. If there's a more generic change to custom ffmpeg output for the hwaccel_device or vaapi_device selection, that's much more likely to be accepted.

[ryzendew]

For this original issue, further context is that we just simple expose everything that ffmpeg reports as supported codecs, and do no validation if those options are correct or will work. Adding a complicated "sanity check" matrix to the ffmpeg output is what we are not interested in doing.

I don't know why you'd need to make a fork when you could just open a PR for the changes required, however. #2246 was rejected because it's HEVC specifically, and we're not really interested in pushing that. If there's a more generic change to custom ffmpeg output for the hwaccel_device or vaapi_device selection, that's much more likely to be accepted.

Sometimes it’s best to do what’s good for ease of use and the people rather than personal bias that pr is amazing and should be merged unlike the current obs it’s useful and solves a big issues

[dodgepong]

We're not interested in pushing HEVC because it's fraught with licensing issues that we don't want to be held responsible for, not because we're "biased" against it.

[aufkrawall]

Why can ffmpeg build shipped by regular Arch repository encode HEVC and they are not sued?

[ryzendew]

We're not interested in pushing HEVC because it's fraught with licensing issues that we don't want to be held responsible for, not because we're "biased" against it.

That makes obs on linux useless then since it’s the only worth a dam codec oh well I guess

[dodgepong]

Why can ffmpeg build shipped by regular Arch repository encode HEVC and they are not sued?

I'm not a lawyer, but my reading of HEVC licensing suggests MPEG-LA are within their right to sue if they wanted. Why they haven't enforced their licensing requirements against other open source projects could be a matter of distribution, since apps like FFmpeg and Handbrake come from France rather than the US, where OBS is from. Bottom line is that we judge the notion of distributing an HEVC encoder to be too risky to have to rely on MPEG-LA being nice. It's a subject we've discussed many times, both internally and with lawyers, and it's the decision we've landed on.

You're absolutely free to build things yourself for your own use. Be mad at software patent law, not us. Meanwhile, we'll happily be backing AV1 as a next-gen encoding standard.

That makes obs on linux useless then since it’s the only worth a dam codec oh well I guess

Hard disagree but I suppose that's a matter of opinion.

[aufkrawall]

Thanks for sharing your perspective and motivation again. It would still be nice if more options for the VAAPI encoder could be offered (more quality parameter tuning, VP9, bitdepth...).

[Fenrirthviti]

Again, to be clear, we're certainly open to general changes for custom ffmpeg output to expose additional parameters or options that might be necessary for vaapi output to work. That change was rejected because it only touched HEVC.

Keep in mind the additional parameters for the encoders are not inserted as -option value but rather as option=value in OBS, as we're not using the CLI but a direct API to ffmpeg. Many of the things you're talking about, such as parameter tuning, are already possible in the custom output.