search

CasparCG / server

CasparCG Server is a Windows and Linux software used to play out professional graphics, audio and video to multiple outputs. It has been in 24/7 broadcast production since 2006. Ready-to-use downloads are available under the Releases tab https://casparcg.com.
GNU General Public License v3.0
901 stars 269 forks source link

Option --enable-vaapi Intel Hardware Encoders in FFmpeg Consumers for Linux #1300

Open walterav1984 opened 4 years ago

walterav1984 commented 4 years ago

Description

Building ffmpeg with --enable-vaapi would allow hardware accelerated encoding (decoding not interested yet) of mpeg2/h264/vc1/h265/vp8/vp9/mjpeg/etc on supported Intel hardware (Haswell graphics gen7.5 4200/4400/4600 or higher have OpenGL 4.6 support) and AMDGPU devices (experimental according to FFmpeg). Doing this encoding on the GPU would free CPU for other intensive tasks. Although this works for FFmpeg this doesn't work yet for CasparCG server.

Solution suggestion

Like #1282 open the discussion to make all the pieces fall into place. Building ffmpeg with --enable-vaapi is not the challenge but parsing the right parameters/syntax in CasparCG Server "casparcg.config" or "AMCP" is with specifying:

Scope

This request is for the basic VAAPI support, since VAAPI can be made complex by using proprietary/partly closed intelhybrid qsv / libmfx driver (Broadwell gen 8 or higher) which allows better quality and more options, but that will be part of a other pull request (with a special build flag)!

Additional information

For quick testing both Debian Buster 10 and Ubuntu 20.04 ship a CasparCG server with --enable-vaapi FFmpeg by default.

The FFmpeg wiki shows that the specifying the vaapi_device /dev/dri/renderD128 goes before the -i input file/source which is a pipe in CasparCG server, however the following command works with ffmpeg cli with specifying vaapi_device after the -i which may work with CasparCG but it doesn't:

FFmpeg default apt versions of Ubuntu 18.04/20.04

sudo apt install ffmpeg vainfo

vainfo
libva info: VA-API version 1.7.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_7
libva error: /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so init failed
libva info: va_openDriver() returns 1
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_1_6
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.7 (libva 2.6.0)
vainfo: Driver version: Intel i965 driver for Intel(R) Haswell - 2.4.0
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            :VAEntrypointVLD
      VAProfileMPEG2Simple            :VAEntrypointEncSlice
      VAProfileMPEG2Main              :VAEntrypointVLD
      VAProfileMPEG2Main              :VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline:VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:VAEntrypointEncSlice
      VAProfileH264Main               :VAEntrypointVLD
      VAProfileH264Main               :VAEntrypointEncSlice
      VAProfileH264High               :VAEntrypointVLD
      VAProfileH264High               :VAEntrypointEncSlice
      VAProfileH264MultiviewHigh      :VAEntrypointVLD
      VAProfileH264MultiviewHigh      :VAEntrypointEncSlice
      VAProfileH264StereoHigh         :VAEntrypointVLD
      VAProfileH264StereoHigh         :VAEntrypointEncSlice
      VAProfileVC1Simple              :VAEntrypointVLD
      VAProfileVC1Main                :VAEntrypointVLD
      VAProfileVC1Advanced            :VAEntrypointVLD
      VAProfileNone                   :VAEntrypointVideoProc
      VAProfileJPEGBaseline           :VAEntrypointVLD

ffmpeg -i clean.dv -vaapi_device /dev/dri/renderD128 -vf 'format=nv12,hwupload' -c:v h264_vaapi outputcc.mp4

ffmpeg -i clean.dv -vaapi_device /dev/dri/renderD128 -vf 'format=nv12,hwupload' -c:v h264_vaapi -b:v 15M -maxrate 15M outputcc15m.mp4

CasparCG casparcg.config consumer fails parsing

        <ffmpeg>
            <path>vaapi.mov</path>
            <args>-vaapi_device:v /dev/dri/renderD128 -filter:v format=nv12,hwupload -codec:v h264_vaapi -codec:a aac</args>
        </ffmpeg>

CasparCG AMCP fails parsing

 ADD 1-1 FILE vaapi01.mov -vaapi_device:v /dev/dri/renderD128 -filter:v format=nv12,hwupload -c:v h264_vaapi -codec:a aac

CasparCG server fail parsing log (casparcg.config / AMCP similar)

ADD 1-1 FILE vaapi05.mov -vaapi_device:v /dev/dri/renderD128 -filter:v format=nv12,hwupload -c:v h264_vaapi -codec:a aac
[2020-04-30 15:13:21.034] [info]    Received message from Console: ADD 1-1 FILE vaapi05.mov -vaapi_device:v /dev/dri/renderD128 -filter:v format=nv12,hwupload -c:v h264_vaapi -codec:a aac\r\n
[2020-04-30 15:13:21.034] [info]    ffmpeg[vaapi05.mov] Initialized.
#202 ADD OK
[2020-04-30 15:13:21.035] [error]   [ffmpeg] [hwupload @ 0x7f7e45ee7d40] A hardware device reference is required to upload frames to.
[2020-04-30 15:13:21.035] [error]
[2020-04-30 15:13:21.035] [error]   [ffmpeg] [Parsed_hwupload_1 @ 0x7f7e7a803dc0] Query format failed for 'Parsed_hwupload_1': Invalid argument
[2020-04-30 15:13:21.035] [error]
[2020-04-30 15:13:21.035] [error]   [ffmpeg] [hwupload @ 0x7f7e45ee7d40] A hardware device reference is required to upload frames to.
[2020-04-30 15:13:21.035] [error]
[2020-04-30 15:13:21.035] [error]   [ffmpeg] [Parsed_hwupload_1 @ 0x7f7e7a803dc0] Query format failed for 'Parsed_hwupload_1': Invalid argument
[2020-04-30 15:13:21.035] [error]
[2020-04-30 15:13:21.036] [error]   Exception: /build/casparcg-server-ESSf4r/casparcg-server-2.2.0+dfsg/src/modules/ffmpeg/consumer/ffmpeg_consumer.cpp(260): Throw in function caspar::ffmpeg::Stream::Stream(AVFormatContext*, std::string, AVCodecID, const caspar::core::video_format_desc&, bool, std::map<std::__cxx11::basic_string<char>, std::__cxx11::basic_string<char> >&)
[2020-04-30 15:13:21.036] [error]   Dynamic exception type: boost::wrapexcept<caspar::ffmpeg::ffmpeg_error_t>
[2020-04-30 15:13:21.036] [error]   [caspar::tag_stacktrace_info*] =  0# caspar::stacktrace_info() in casparcg
[2020-04-30 15:13:21.036] [error]    1# caspar::ffmpeg::Stream::Stream(AVFormatContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, AVCodecID, caspar::core::video_format_desc const&, bool, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&) in casparcg
[2020-04-30 15:13:21.036] [error]    2# caspar::ffmpeg::ffmpeg_consumer::initialize(caspar::core::video_format_desc const&, int)::{lambda()#1}::operator()() const in casparcg
[2020-04-30 15:13:21.036] [error]    3# 0x00007F7E8A577CB4 in /lib/x86_64-linux-gnu/libstdc++.so.6
[2020-04-30 15:13:21.036] [error]    4# 0x00007F7E8A82F609 in /lib/x86_64-linux-gnu/libpthread.so.0
[2020-04-30 15:13:21.036] [error]    5# clone in /lib/x86_64-linux-gnu/libc.so.6
[2020-04-30 15:13:21.036] [error]
+
[2020-04-30 15:13:21.036] [error]   [boost::errinfo_errno_*] = 22, "Invalid argument"
[2020-04-30 15:13:21.036] [error]   [boost::errinfo_api_function_*] = avfilter_graph_config(graph.get(), nullptr)
[2020-04-30 15:13:21.036] [error]
[2020-04-30 15:13:21.036] [error]    0# caspar::log::get_stack_trace[abi:cxx11]() in casparcg
[2020-04-30 15:13:21.036] [error]    1# caspar::core::output::impl::operator()(caspar::core::const_frame, caspar::core::video_format_desc const&) in casparcg
[2020-04-30 15:13:21.036] [error]    2# caspar::core::output::operator()(caspar::core::const_frame, caspar::core::video_format_desc const&) in casparcg
[2020-04-30 15:13:21.036] [error]    3# caspar::core::video_channel::impl::impl(int, caspar::core::video_format_desc const&, std::unique_ptr<caspar::core::image_mixer, std::default_delete<caspar::core::image_mixer> >, std::function<void (caspar::core::monitor::state)>)::{lambda()#1}::operator()() const in casparcg
[2020-04-30 15:13:21.036] [error]    4# 0x00007F7E8A577CB4 in /lib/x86_64-linux-gnu/libstdc++.so.6
[2020-04-30 15:13:21.036] [error]    5# 0x00007F7E8A82F609 in /lib/x86_64-linux-gnu/libpthread.so.0
[2020-04-30 15:13:21.036] [error]    6# clone in /lib/x86_64-linux-gnu/libc.so.6
[2020-04-30 15:13:21.036] [error]
[2020-04-30 15:13:21.056] [warning] Assertion Failed: counter < 8 file:/build/casparcg-server-ESSf4r/casparcg-server-2.2.0+dfsg/src/core/consumer/frame_consumer.cpp line:96
[2020-04-30 15:13:21.056] [info]    ffmpeg[vaapi05.mov] Uninitialized.
LRTNZ commented 4 years ago

Out of curiosity, what does the log throw if you create a channel with vaapi in the casparcg.conf file? And could you provide the arguments used in the config file when you do that please?

walterav1984 commented 4 years ago

Both "casparcg.config" and "AMCP" error quiet similar, updated my original topic with "casparcg.config" ffmpeg consumer arguments @LRTNZ .

LRTNZ commented 4 years ago

@walterav1984 reading the FFmpeg wiki it says "hardware generally wants the nv12 layout, but most software functions use the yuv420p layout" - have you tried yuv420p?

walterav1984 commented 4 years ago

Changing the "pre" filter format from "nv12" to "yuv420p" doesn't change the error log @LRTNZ , I guess CasparCG server feeds pure 422 or 444 to its ffmpeg consumer.

If we look closer to the log it already errors first mentioning the hardware device which cannot be found but, strange thing is its called 2 times like its passing the filter options to the vaapi_device argument?

[2020-04-30 15:13:21.035] [error]   [ffmpeg] [hwupload @ 0x7f7e45ee7d40] A hardware device reference is required to upload frames to.
[2020-04-30 15:13:21.035] [error]
[2020-04-30 15:13:21.035] [error]   [ffmpeg] [Parsed_hwupload_1 @ 0x7f7e7a803dc0] Query format failed for 'Parsed_hwupload_1': Invalid argument
[2020-04-30 15:13:21.035] [error]
[2020-04-30 15:13:21.035] [error]   [ffmpeg] [hwupload @ 0x7f7e45ee7d40] A hardware device reference is required to upload frames to.
[2020-04-30 15:13:21.035] [error]
[2020-04-30 15:13:21.035] [error]   [ffmpeg] [Parsed_hwupload_1 @ 0x7f7e7a803dc0] Query format failed for 'Parsed_hwupload_1': Invalid argument
[2020-04-30 15:13:21.035] [error]
LRTNZ commented 4 years ago

@walterav1984 poking around a few examples of using vaapi in C++ online, and taking a quick look in the Caspar code, I am not suprised this hasn't worked out of the box.

Unlike some of the other hardware accelerated solutions, that just 'slot in' with more standard ffmpeg parameters like nvenc encoding does, vaapi requires a bit more configuration with options I do not think Caspar is setup to handle yet.

Also, it looks like according to the ffmpeg hwaccel documentation that vaapi is only is supported on Linux at this time: https://trac.ffmpeg.org/wiki/HWAccelIntro
So for this to be added in, something would need to be added to check that Caspar is not on windows when trying to use vaapi (Which wouldn't be to hard I would think), and to have a sensible fallback if that is the case.

The devs will have a better idea than me on the work required to add this in, but I believe from looking at the code (Albeit with only a little cpp experience), that it would require quite a sizable refactoring to implement this feature.

Potentially in the future, depending on the release of Intel dedicated GPUs and what they offer in the way of hardware encoding, this might become something more feasible to consider then.

walterav1984 commented 4 years ago

@walterav1984 poking around a few examples of using vaapi in C++ online, and taking a quick look in the Caspar code, I am not suprised this hasn't worked out of the box.

Unlike some of the other hardware accelerated solutions, that just 'slot in' with more standard ffmpeg parameters like nvenc encoding does, vaapi requires a bit more configuration with options I do not think Caspar is setup to handle yet.

Was hoping that just some input sanitation would be enough, but the "pre-" "format" option that must be applied to the VAAPI device itself and perhaps "again" to the output itself is indeed confusing if CasparCG expects it a single time.

Also, it looks like according to the ffmpeg hwaccel documentation that vaapi is only is supported on Linux at this time: https://trac.ffmpeg.org/wiki/HWAccelIntro So for this to be added in, something would need to be added to check that Caspar is not on windows when trying to use vaapi (Which wouldn't be to hard I would think), and to have a sensible fallback if that is the case.

Invalid or wrong command handling is currently already erratic with CasparCG Server, but indeed VAAPI is missing for Windows and commands containing it will probably fail/error.

The devs will have a better idea than me on the work required to add this in, but I believe from looking at the code (Albeit with only a little cpp experience), that it would require quite a sizable refactoring to implement this feature.

Potentially in the future, depending on the release of Intel dedicated GPUs and what they offer in the way of hardware encoding, this might become something more feasible to consider then.

Perhaps going the more complex "libmfx" route with Intel (Broadwell Gen8) and future dedicated GPU's ahead would be a better fit for both Linux and Windows support. Installing the libmfx driver part for Ubuntu 20.04 is easy but it still needs ffmpeg to be recompiled with --enable-libmfx which can be done in the FFmpeg docker part.

The Windows zeranoe FFmpeg builds already contain --enable-libmfx but sadly I have no Intel Gen8 to test it, still libmfx uses VAAPI on Linux as a backend and similar syntax which investigating benefit both solutions.