Custom Output (FFMPEG) vaapi encoder crashes on init FFMPEG 4.4

[DanielGaaA]

Operating System Info

Other

Other OS

Fedora 35

OBS Studio Version

27.1.3

OBS Studio Version (Other)

No response

OBS Studio Log URL

https://obsproject.com/logs/cf95rQ7oRBI3w8up

OBS Studio Crash Log URL

No response

Expected Behavior

To be able to record with hardware acceleration while selecting Custom output(FFmpeg).

Current Behavior

I am trying to set up hardware accelerated encoding via vaapi and trying to pass vaapi_device=/dev/dri/renderD128 to ffmpeg but when i start recording OBS it will crash. In logs there will be Failed to set vaapi_device=/dev/dri/renderD128

I tried to write it to muxer setting or video setting but it crash in both cases. As far i understand vaapi_device is not video encoder /muxer settings. And there is no other option to pass variable.

Steps to Reproduce

1. Settings from screenshot
2. Start recording and crash OBS

Anything else we should know?

I found different https://github.com/obsproject/obs-studio/issues/2296#issuecomment-840957925 that this is not possible, but issue was closed anyway. Yes i saw last comment but it's not working as you can see from my settings

[kkartaltepe]

As mentioned in the issue https://github.com/obsproject/obs-studio/issues/2296#issuecomment-840967668

You cannot set this parameter via any of the existing settings parameters. It belongs to a different set of options that are not exposed in the existing plugin. It requires someone to expose this in the plugins. Generally advanced ffmpeg is at your own risk so if some encoders or options dont work its not a bug, but if someone wants to submit code to enable these encoders/options it probably would be accepted.

However obs shouldnt crash when you send bad options. Are you sure obs is crashing (the program itself is closing?)

[DanielGaaA]

Is there any plan to implement this feature as functionality is already in obs for selecting vaapi_device in default preset

Yes obs will crash 2 second after clicking on start recording

[New Thread 0x7f5ecbfff640 (LWP 33773)]

Thread 29 "obs" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f5ecbfff640 (LWP 33773)]
0x00007f5ef48d0abc in ff_vaapi_encode_close (avctx=0x7f5ec4007140) at libavcodec/vaapi_encode.c:2542
2542            vaDestroyContext(ctx->hwctx->display, ctx->va_context);
(gdb) bt
#0  0x00007f5ef48d0abc in ff_vaapi_encode_close (avctx=0x7f5ec4007140) at libavcodec/vaapi_encode.c:2542
#1  0x00007f5ef4978bc5 in avcodec_open2 (avctx=<optimized out>, codec=<optimized out>, options=<optimized out>) at libavcodec/avcodec.c:447
#2  0x00007f5e68ec9d17 in ffmpeg_data_init () at /usr//lib64/obs-plugins/obs-ffmpeg.so
#3  0x00007f5e68eca75e in start_thread () at /usr//lib64/obs-plugins/obs-ffmpeg.so
#4  0x00007f5ef29ccb17 in start_thread () at /lib64/libc.so.6
#5  0x00007f5ef2a516c0 in clone3 () at /lib64/libc.so.6

[kkartaltepe]

Is there any plan to implement this feature as functionality is already in obs for selecting vaapi_device in default preset

These are two separate plugins. No i dont know of anyone planning to implement this functionality.

Thanks for the backtrace it helps a lot.

[kkartaltepe]

This looks like an ffmpeg bug, the crash should occur even if you dont provide a vaapi_device= option. Is that right?

[DanielGaaA]

Yes that correct. It will crash when i don't provide vaapi_device= option.

[kkartaltepe]

Thanks i dont think there is any fix on our end, we could probably filter out the broken vaapi encoders though this might break people using older versions of ffmpeg where it works. But the crash is entirely within ffmpeg with no obvious way for us to prevent it.

[DanielGaaA]

But ffmpeg vaapi is working for me. Only it expect vaapi_device= option which obs doesn't provide . Default OBS preset for x264_vaapi also works because it provide vaapi_device= option

[kkartaltepe]

If you are interested in submitting a PR to support this functionality you can look at the ffmpeg code and our vaapi implementation.

[GloriousEggroll]

I have an Idea on this --

Since OBS already has it's own VAAPI and other internal encoders -- is it possible to remove the the ffmpeg versions from the ffmpeg custom output Video Encoder list, and replace them with obs's internal ones? Then when they select those from the dropdown, the custom options for those specific encoders would come up the same as they do on the standard output page.

This would allow users to use vaapi and still give them the other custom output options. OBS's custom internal encoders already use ffmpeg codecs so trying to redefine stuff that's already set in a different area of OBS is kind of like reinventing the wheel.

I've attached screenshots for hopefully a better reference.

In the first screenshot we see the 'standard' layout with all of obs's built-in Encoder drop down list In the second screenshot we see the 'Custom Output' layout, with ffmpeg's video encoder dropdown list.

Why not take -all- of the Encoder drop down options in the standard layout, and add them to the Video Encoder list in the custom layout. Then just have their options come up when they are selected.

-OR-

Even the opposite might work -- give us audio codec options on the standard layout

[GloriousEggroll]

For those wondering, there's finally a way to do HW encode via custom ffmpeg using amd's AMF encoder instead of vaapi on linux.

Fedora: https://www.youtube.com/watch?v=gQtO3tlMqE4&lc

Arch: https://www.youtube.com/watch?v=SRGAA7PyAEI

Vulkan switcher on arch: https://aur.archlinux.org/packages/amd-vulkan-prefixes

Relevant ubuntu packages:

Vulkan 64 bit: http://repo.radeon.com/amdgpu/21.50.2/ubuntu/pool/proprietary/v/vulkan-amdgpu-pro/vulkan-amdgpu-pro_21.50.2-1384495_amd64.deb

Vulkan 32 bit: http://repo.radeon.com/amdgpu/21.50.2/ubuntu/pool/proprietary/v/vulkan-amdgpu-pro/vulkan-amdgpu-pro_21.50.2-1384495_i386.deb

AMF: http://repo.radeon.com/amdgpu/21.50.2/ubuntu/pool/proprietary/a/amf-amdgpu-pro/amf-amdgpu-pro_1.4.24-1384495_amd64.deb

StreamFX: https://github.com/Xaymar/obs-StreamFX/releases/tag/0.11.1

Notes: ffmpeg on your distro must be compiled with AMF support (just like with nvenc) obs and any game you capture with obs using vulkan must both be run using amd's pro vulkan driver because AMF requires it:

obs: VK_ICD_FILENAMES="/usr/share/vulkan/icd.d/amd_pro_icd32.json:/usr/share/vulkan/icd.d/amd_pro_icd64.json" obs

steam game: VK_ICD_FILENAMES="/usr/share/vulkan/icd.d/amd_pro_icd32.json:/usr/share/vulkan/icd.d/amd_pro_icd64.json" %command%

or if you have vulkan switcher on your distro just use: vk_pro obs vk_pro %command%

[kkartaltepe]

I was looking into this recently, its almost reasonable to add this to the UI and I have written a proof of concept at https://github.com/kkartaltepe/obs-studio/tree/ffmpeg-hwctx which works on my machine with h264_vaapi. If this approach solves the problem for other encoders and some of the awful hacks in it can be ironed out it can probably make it into 28.

There are still some random crashes in this branch, probably due to basically no error handling of ffmpeg and probably leaking or double freeing references. But it works well enough to record for 10 seconds to prove it works. Let me know if you can snag a backtrace from a crash.

--- edit Since the code is annoying to read, this just adds a "VHWDevice" string field when you select an encoder that reports hardware encoding. You can either leave this field blank to let ffmpeg choose the default device, or enter a string equivalent to the vaapi_device option from ffmpeg.

[xmotif]

I'm looking forward to the release of this UI patch.

[ShadwDrgn]

@kkartaltepe any plans to implement something like this or submit a pull request? I've been wanting AV1 VAAPI for some time but unfortunately while ffmpeg supports it OBS does not. This would allow me to use it with custom ffmpeg.

[kkartaltepe]

@kkartaltepe any plans to implement something like this or submit a pull request? I've been wanting AV1 VAAPI for some time but unfortunately while ffmpeg supports it OBS does not. This would allow me to use it with custom ffmpeg.

No, there is already a completed AV1 support PR that has been reviewed you can apply on top of master if you want.

[shmerl]

No, there is already a completed AV1 support PR that has been reviewed you can apply on top of master if you want.

But that AV1 is for a different plugin? I.e. custom ffmpeg with av1_vaapi encoder isn't working same as h264_vaapi encoder due to the missing device possibly.

I just hit this issue here.

It would be good to have a PR that allows setting the device for all VAAPI encoders.