V4L2 Video Decoding support

Hexxeh commented 4 years ago

It appears that V4L2 decoding support was disabled here some versions ago: https://github.com/saiarcot895/chromium-ubuntu-build/blob/86ca7452d0a8984d3c3dca47bb73b519cc1c1a2b/debian/rules#L136

Why was that? Does this no longer work in Chromium? I'm trying to get accelerated video decoding via the Hantro VPU on an RK3399 chipset to take the load off of the CPUs.

saiarcot895 commented 4 years ago

If I recall correctly, support for Linux V4L2 was added in as part of Wayland on Ozone. I had gotten it to compile once, but the UI didn't appear to be as polished, and so I've disabled it.

SteffenDE commented 3 years ago

As the ozone support seems to be getting better, would it be possible to add V4L2 support back?

nicman23 commented 3 years ago

hello as per https://chromium.googlesource.com/chromium/src/+/master/docs/ozone_overview.md

It is not required to build Ozone for Linux anymore for the purpose of testing. It is enough to start Chrome with the following flags - ./chrome --enable-features=UseOzonePlatform --ozone-platform={x11/wayland}.

ozone is compiled in by default in chrome and supports both x11 (and might be replacing the current x11 backend) and wayland.

enabling v4l2 would be great for arm64 (and users have reported that it works for rpis 4 with the above mentioning this very issue)

bjharper commented 3 years ago

Firstly thanks for the work on this ppa package it is really nice to have a non snapd sandboxed installation for Ubuntu so we can use gpu hardware acceleration on devices with lower cpu power.

I'm currently using this on arm64 on various odroid sbc devices with mesa panfrost hardware drivers and the GPU components are working quiet nicely.

With limited cpu resources available for decoding higher quality video the CPU limits are currently reached at around 1080p 30fps video (some videos at this resolutions are fine others some issues with max cpu) so I am looking at video decoding which is supported by the amlogic meson v4l2 drivers and firmware provided by the kernel.

It appears that it might be possible to use hardware video decoding with chromium on these devices using various methods

Enable support in Chromium to use direct v4l2 interface into the kernel and firmware (as to why I post here against this bug as it seems like the ideal solution)
Enable platform ffmpeg and either unblacklist required hardware devices so that hardware decoding can be supported via ffmpeg with either H264 or vp9 (both supported by the platform ffmpeg interfacing with the kernel)
Use vappi with https://github.com/bootlin/libva-v4l2-request to allow interface between vappi and the v4l2 request kernel api (libva-v4l2-request is not overly maintained and due to change in 5.11.x kernel api not currently compatible - I need later kernels for panfrost support for GPU)
Possibly use some kind of pipewire support (although my investigation so far it appears this is only relevant for input devices and not output decoding to drm)
Possibly use gstreamer interface in the pipeline although this appears to be rather old and abandoned

I guess my question is can v4l2 support be enabled in your chromium builds so some further testing can be achieved?

bjharper commented 3 years ago

I have just attempted a build on arm64 with the following options enabled:

use_v4lplugin=true use_v4l2_codec=true use_linux_v4l2_only=true

The following errors encountered:

[16296/40670] CXX obj/media/gpu/v4l2/v4l2/v4l2_h264_accelerator_chromium.o
FAILED: obj/media/gpu/v4l2/v4l2/v4l2_h264_accelerator_chromium.o
/usr/bin/clang++-10 -MMD -MF obj/media/gpu/v4l2/v4l2/v4l2_h264_accelerator_chromium.o.d -DMEDIA_GPU_IMPLEMENTATION -DUSE_UDEV -DUSE_AURA=1 -DUSE_GLIB=1 -DUSE_NSS_CERTS=1 -DUSE_OZONE=1 -DUSE_X11=1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D_FORTIFY_SOURCE=2 -D_LIBCPP_ABI_UNSTABLE -D_LIBCPP_DISABLE_VISIBILITY_ANNOTATIONS -D_LIBCXXABI_DISABLE_VISIBILITY_ANNOTATIONS -D_LIBCPP_ENABLE_NODISCARD -D_LIBCPP_HAS_NO_VENDOR_AVAILABILITY_ANNOTATIONS -DCR_LIBCXX_REVISION=8fa87946779682841e21e2da977eccfb6cb3bded -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DGL_GLEXT_PROTOTYPES -DUSE_GLX -DUSE_EGL -DGLIB_VERSION_MAX_ALLOWED=GLIB_VERSION_2_40 -DGLIB_VERSION_MIN_REQUIRED=GLIB_VERSION_2_40 -DWEBP_EXTERN=extern -DVK_USE_PLATFORM_XCB_KHR -DSK_FAVOR_WUFFS_V_0_3_OVER_V_0_2 -DSK_CODEC_DECODES_PNG -DSK_CODEC_DECODES_WEBP -DSK_ENCODE_PNG -DSK_ENCODE_WEBP -DSK_USER_CONFIG_HEADER=\"../../skia/config/SkUserConfig.h\" -DSK_GL -DSK_CODEC_DECODES_JPEG -DSK_ENCODE_JPEG -DSK_HAS_WUFFS_LIBRARY -DSK_VULKAN_HEADER=\"../../skia/config/SkVulkanConfig.h\" -DSK_VULKAN=1 -DSK_SUPPORT_GPU=1 -DSK_GPU_WORKAROUNDS_HEADER=\"gpu/config/gpu_driver_bug_workaround_autogen.h\" -DVK_USE_PLATFORM_XCB_KHR -DU_USING_ICU_NAMESPACE=0 -DU_ENABLE_DYLOAD=0 -DUSE_CHROMIUM_ICU=1 -DU_ENABLE_TRACING=1 -DU_ENABLE_RESOURCE_TRACING=0 -DU_STATIC_IMPLEMENTATION -DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_FILE -DGOOGLE_PROTOBUF_NO_RTTI -DGOOGLE_PROTOBUF_NO_STATIC_INITIALIZER -DHAVE_PTHREAD -DCRASHPAD_ZLIB_SOURCE_EXTERNAL -DLEVELDB_PLATFORM_CHROMIUM=1 -DLEVELDB_PLATFORM_CHROMIUM=1 -DLIBGAV1_MAX_BITDEPTH=10 -DLIBGAV1_THREADPOOL_USE_STD_MUTEX -DLIBGAV1_ENABLE_LOGGING=0 -DLIBGAV1_PUBLIC= -DVK_NO_PROTOTYPES -DUSE_VULKAN_XCB -I../.. -Igen -I../../third_party/libyuv/include -I../../third_party/perfetto/include -Igen/third_party/perfetto/build_config -Igen/third_party/perfetto -I../../third_party/libwebp/src -I../../third_party/khronos -I../../gpu -I../../third_party/vulkan-deps/vulkan-headers/src/include -Igen/third_party/dawn/src/include -I../../third_party/dawn/src/include -I../../third_party/abseil-cpp -I../../third_party/boringssl/src/include -I../../third_party/protobuf/src -Igen/protoc_out -I../../third_party/mesa_headers -I../../third_party/skia -I../../third_party/wuffs/src/release/c -I../../third_party/vulkan/include -I../../third_party/vulkan-deps/vulkan-headers/src/include -I../../third_party/icu/source/common -I../../third_party/icu/source/i18n -I../../third_party/ced/src -I../../third_party/protobuf/src -I../../third_party/crashpad/crashpad -I../../third_party/crashpad/crashpad/compat/linux -I../../third_party/crashpad/crashpad/compat/non_win -I../../third_party/zlib -I../../third_party/libwebm/source -I../../third_party/leveldatabase -I../../third_party/leveldatabase/src -I../../third_party/leveldatabase/src/include -I../../third_party/libgav1/src -I../../third_party/libgav1/src/src -fno-delete-null-pointer-checks -fno-ident -fno-strict-aliasing --param=ssp-buffer-size=4 -fstack-protector -funwind-tables -fPIC -pthread -fcolor-diagnostics -fmerge-all-constants --target=aarch64-linux-gnu -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -Xclang -fdebug-compilation-dir -Xclang . -no-canonical-prefixes -Wall -Wextra -Wimplicit-fallthrough -Wunreachable-code -Wthread-safety -Wextra-semi -Wno-missing-field-initializers -Wno-unused-parameter -Wno-c++11-narrowing -Wno-unneeded-internal-declaration -Wno-unknown-warning-option -Wno-undefined-var-template -Wno-psabi -Wno-ignored-pragma-optimize -Wno-implicit-int-float-conversion -Wno-final-dtor-non-final-class -Wno-builtin-assume-aligned-alignment -Wno-deprecated-copy -Wno-non-c-typedef-for-linkage -Wno-max-tokens -O2 -fdata-sections -ffunction-sections -fno-omit-frame-pointer -g1 -ftrivial-auto-var-init=pattern -fvisibility=hidden -Wheader-hygiene -Wstring-conversion -Wtautological-overlap-compare -I/usr/include/glib-2.0 -I/usr/lib/aarch64-linux-gnu/glib-2.0/include -DPROTOBUF_ALLOW_DEPRECATED=1 -Wno-shorten-64-to-32 -std=c++14 -fno-trigraphs -Wno-trigraphs -fno-exceptions -fno-rtti -nostdinc++ -isystem../../buildtools/third_party/libc++/trunk/include -isystem../../buildtools/third_party/libc++abi/trunk/include -fvisibility-inlines-hidden -c ../../media/gpu/v4l2/v4l2_h264_accelerator_chromium.cc -o obj/media/gpu/v4l2/v4l2/v4l2_h264_accelerator_chromium.o
In file included from ../../media/gpu/v4l2/v4l2_h264_accelerator_chromium.cc:5:
In file included from ../../media/gpu/v4l2/v4l2_h264_accelerator_chromium.h:13:
In file included from ../../media/gpu/h264_decoder.h:16:
In file included from ../../base/memory/ref_counted.h:19:
In file included from ../../base/sequence_checker.h:9:
../../base/sequence_checker_impl.h:27:48: warning: invalid capability name 'context'; capability name must be 'mutex' or 'role' [-Wthread-safety-attributes]
class THREAD_ANNOTATION_ATTRIBUTE__(capability("context"))
                                               ^
In file included from ../../media/gpu/v4l2/v4l2_h264_accelerator_chromium.cc:5:
In file included from ../../media/gpu/v4l2/v4l2_h264_accelerator_chromium.h:13:
In file included from ../../media/gpu/h264_decoder.h:16:
In file included from ../../base/memory/ref_counted.h:19:
../../base/sequence_checker.h:103:48: warning: invalid capability name 'context'; capability name must be 'mutex' or 'role' [-Wthread-safety-attributes]
class THREAD_ANNOTATION_ATTRIBUTE__(capability("context"))
                                               ^
../../media/gpu/v4l2/v4l2_h264_accelerator_chromium.cc:12:10: fatal error: 'linux/media/h264-ctrls.h' file not found
#include <linux/media/h264-ctrls.h>
         ^~~~~~~~~~~~~~~~~~~~~~~~~~
2 warnings and 1 error generated.
[16299/40670] CXX obj/media/gpu/v4l2/v4l2/v4l2_device.o
FAILED: obj/media/gpu/v4l2/v4l2/v4l2_device.o
/usr/bin/clang++-10 -MMD -MF obj/media/gpu/v4l2/v4l2/v4l2_device.o.d -DMEDIA_GPU_IMPLEMENTATION -DUSE_UDEV -DUSE_AURA=1 -DUSE_GLIB=1 -DUSE_NSS_CERTS=1 -DUSE_OZONE=1 -DUSE_X11=1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D_FORTIFY_SOURCE=2 -D_LIBCPP_ABI_UNSTABLE -D_LIBCPP_DISABLE_VISIBILITY_ANNOTATIONS -D_LIBCXXABI_DISABLE_VISIBILITY_ANNOTATIONS -D_LIBCPP_ENABLE_NODISCARD -D_LIBCPP_HAS_NO_VENDOR_AVAILABILITY_ANNOTATIONS -DCR_LIBCXX_REVISION=8fa87946779682841e21e2da977eccfb6cb3bded -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DGL_GLEXT_PROTOTYPES -DUSE_GLX -DUSE_EGL -DGLIB_VERSION_MAX_ALLOWED=GLIB_VERSION_2_40 -DGLIB_VERSION_MIN_REQUIRED=GLIB_VERSION_2_40 -DWEBP_EXTERN=extern -DVK_USE_PLATFORM_XCB_KHR -DSK_FAVOR_WUFFS_V_0_3_OVER_V_0_2 -DSK_CODEC_DECODES_PNG -DSK_CODEC_DECODES_WEBP -DSK_ENCODE_PNG -DSK_ENCODE_WEBP -DSK_USER_CONFIG_HEADER=\"../../skia/config/SkUserConfig.h\" -DSK_GL -DSK_CODEC_DECODES_JPEG -DSK_ENCODE_JPEG -DSK_HAS_WUFFS_LIBRARY -DSK_VULKAN_HEADER=\"../../skia/config/SkVulkanConfig.h\" -DSK_VULKAN=1 -DSK_SUPPORT_GPU=1 -DSK_GPU_WORKAROUNDS_HEADER=\"gpu/config/gpu_driver_bug_workaround_autogen.h\" -DVK_USE_PLATFORM_XCB_KHR -DU_USING_ICU_NAMESPACE=0 -DU_ENABLE_DYLOAD=0 -DUSE_CHROMIUM_ICU=1 -DU_ENABLE_TRACING=1 -DU_ENABLE_RESOURCE_TRACING=0 -DU_STATIC_IMPLEMENTATION -DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_FILE -DGOOGLE_PROTOBUF_NO_RTTI -DGOOGLE_PROTOBUF_NO_STATIC_INITIALIZER -DHAVE_PTHREAD -DCRASHPAD_ZLIB_SOURCE_EXTERNAL -DLEVELDB_PLATFORM_CHROMIUM=1 -DLEVELDB_PLATFORM_CHROMIUM=1 -DLIBGAV1_MAX_BITDEPTH=10 -DLIBGAV1_THREADPOOL_USE_STD_MUTEX -DLIBGAV1_ENABLE_LOGGING=0 -DLIBGAV1_PUBLIC= -DVK_NO_PROTOTYPES -DUSE_VULKAN_XCB -I../.. -Igen -I../../third_party/libyuv/include -I../../third_party/perfetto/include -Igen/third_party/perfetto/build_config -Igen/third_party/perfetto -I../../third_party/libwebp/src -I../../third_party/khronos -I../../gpu -I../../third_party/vulkan-deps/vulkan-headers/src/include -Igen/third_party/dawn/src/include -I../../third_party/dawn/src/include -I../../third_party/abseil-cpp -I../../third_party/boringssl/src/include -I../../third_party/protobuf/src -Igen/protoc_out -I../../third_party/mesa_headers -I../../third_party/skia -I../../third_party/wuffs/src/release/c -I../../third_party/vulkan/include -I../../third_party/vulkan-deps/vulkan-headers/src/include -I../../third_party/icu/source/common -I../../third_party/icu/source/i18n -I../../third_party/ced/src -I../../third_party/protobuf/src -I../../third_party/crashpad/crashpad -I../../third_party/crashpad/crashpad/compat/linux -I../../third_party/crashpad/crashpad/compat/non_win -I../../third_party/zlib -I../../third_party/libwebm/source -I../../third_party/leveldatabase -I../../third_party/leveldatabase/src -I../../third_party/leveldatabase/src/include -I../../third_party/libgav1/src -I../../third_party/libgav1/src/src -fno-delete-null-pointer-checks -fno-ident -fno-strict-aliasing --param=ssp-buffer-size=4 -fstack-protector -funwind-tables -fPIC -pthread -fcolor-diagnostics -fmerge-all-constants --target=aarch64-linux-gnu -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -Xclang -fdebug-compilation-dir -Xclang . -no-canonical-prefixes -Wall -Wextra -Wimplicit-fallthrough -Wunreachable-code -Wthread-safety -Wextra-semi -Wno-missing-field-initializers -Wno-unused-parameter -Wno-c++11-narrowing -Wno-unneeded-internal-declaration -Wno-unknown-warning-option -Wno-undefined-var-template -Wno-psabi -Wno-ignored-pragma-optimize -Wno-implicit-int-float-conversion -Wno-final-dtor-non-final-class -Wno-builtin-assume-aligned-alignment -Wno-deprecated-copy -Wno-non-c-typedef-for-linkage -Wno-max-tokens -O2 -fdata-sections -ffunction-sections -fno-omit-frame-pointer -g1 -ftrivial-auto-var-init=pattern -fvisibility=hidden -Wheader-hygiene -Wstring-conversion -Wtautological-overlap-compare -I/usr/include/glib-2.0 -I/usr/lib/aarch64-linux-gnu/glib-2.0/include -DPROTOBUF_ALLOW_DEPRECATED=1 -Wno-shorten-64-to-32 -std=c++14 -fno-trigraphs -Wno-trigraphs -fno-exceptions -fno-rtti -nostdinc++ -isystem../../buildtools/third_party/libc++/trunk/include -isystem../../buildtools/third_party/libc++abi/trunk/include -fvisibility-inlines-hidden -c ../../media/gpu/v4l2/v4l2_device.cc -o obj/media/gpu/v4l2/v4l2/v4l2_device.o
In file included from ../../media/gpu/v4l2/v4l2_device.cc:5:
In file included from ../../media/gpu/v4l2/v4l2_device.h:22:
In file included from ../../base/memory/ref_counted.h:19:
In file included from ../../base/sequence_checker.h:9:
../../base/sequence_checker_impl.h:27:48: warning: invalid capability name 'context'; capability name must be 'mutex' or 'role' [-Wthread-safety-attributes]
class THREAD_ANNOTATION_ATTRIBUTE__(capability("context"))
                                               ^
In file included from ../../media/gpu/v4l2/v4l2_device.cc:5:
In file included from ../../media/gpu/v4l2/v4l2_device.h:22:
In file included from ../../base/memory/ref_counted.h:19:
../../base/sequence_checker.h:103:48: warning: invalid capability name 'context'; capability name must be 'mutex' or 'role' [-Wthread-safety-attributes]
class THREAD_ANNOTATION_ATTRIBUTE__(capability("context"))
                                               ^
../../media/gpu/v4l2/v4l2_device.cc:801:30: error: no member named 'config_store' in 'v4l2_buffer'
  buffer_data_->v4l2_buffer_.config_store = config_store;
  ~~~~~~~~~~~~~~~~~~~~~~~~~~ ^
../../media/gpu/v4l2/v4l2_device.cc:1487:14: error: use of undeclared identifier 'V4L2_PIX_FMT_VP8_FRAME'
      return V4L2_PIX_FMT_VP8_FRAME;
             ^
../../media/gpu/v4l2/v4l2_device.cc:1492:14: error: use of undeclared identifier 'V4L2_PIX_FMT_VP9_FRAME'
      return V4L2_PIX_FMT_VP9_FRAME;
             ^
../../media/gpu/v4l2/v4l2_device.cc:1605:10: error: use of undeclared identifier 'V4L2_PIX_FMT_VP8_FRAME'
    case V4L2_PIX_FMT_VP8_FRAME:
         ^
../../media/gpu/v4l2/v4l2_device.cc:1609:10: error: use of undeclared identifier 'V4L2_PIX_FMT_VP9_FRAME'
    case V4L2_PIX_FMT_VP9_FRAME:
         ^
2 warnings and 5 errors generated.

This was built against 5.11.x kernel headers (I do plan to move to 5.12) where the new v4l2 stateless decoder has now been implemented (See https://www.phoronix.com/scan.php?page=news_item&px=Stateless-H264-Out-Of-Staging for some basic details)

Some changes will be required to support current mainline v4l2, I will do some more investigation to see how much work it would be to port chromium over to the new stateless api in current mainline. Im not an expert but will see how far I can get.

jolting commented 2 years ago

that looks like some chromeos specific patch. https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/7bad859205fbd2d7e3df8715f4314e03b1578af5/sys-kernel/linux-headers/files/0001-CHROMIUM-media-headers-Import-V4L2-headers-from-Chro.patch

probably based on this. https://www.spinics.net/lists/linux-media/msg81031.html

Something used to store a v4l2 surface ID. https://source.chromium.org/chromium/chromium/src/+/main:media/gpu/v4l2/v4l2_decode_surface.h;l=120?q=config_store_&ss=chromium

That implementation might take some work to support on vanilla Linux is a standard v4l2 implementation.

jolting commented 2 years ago

Looks like there isn't a way to conditionally compiling out V4L2ConfigStoreDecodeSurface, which isn't needed when using the request api.. It just needs the V4L2RequestDecodeSurface. That should be fairly easy thing to add.

saiarcot895 commented 2 years ago

There was some work done when initially bringing up Wayland on Ozone that excluded the ChromeOS-specific stuff for the regular Linux build. I don't know if that patchset would still work today or not.

jolting commented 2 years ago

Speaking of excluding ChromeOS-specific stuff, I spotted this today. media/gpu/v4l2: Split upstream compatible and ChromeOS specific code https://chromium-review.googlesource.com/c/chromium/src/+/3380426

amazingfate commented 2 years ago

Speaking of excluding ChromeOS-specific stuff, I spotted this today. media/gpu/v4l2: Split upstream compatible and ChromeOS specific code https://chromium-review.googlesource.com/c/chromium/src/+/3380426

Great job! Looking forward to seeing this commit get merged.

amazingfate commented 2 years ago

@bjharper @jolting hello, I've made a build applying @jolting's patch above I have a radxa rock3a board and the vpu is hantro G1. When I start chrome with flag --enable-features=VaapiVideoDecoder, I can see Video Decode: Hardware accelerated in page chrome://gpu. But vpu is not used when I play bbb_sunflower_1080p_30fps_normal.mp4 in chrome. I can see the vpu usage by using cammand watch -n 1 'cat /proc/interrupts |grep codec'. The cpu load is also high when playing the video. Do you guys have any idea?

jolting commented 2 years ago

You built with use_v4l2_codec=true?

amazingfate commented 2 years ago

You built with use_v4l2_codec=true?

Yes, I did. I aslo build with use_vaapi=false since build would fall when vaapi is enabled.

jolting commented 2 years ago

I'm not sure what works yet. That patch was merged yesterday. I'm just following this passively.

There appears to be some allocator work necessary. https://chromium-review.googlesource.com/c/chromium/src/+/3380427/7

Collabora mentioned some Video Codec work in their blog too. https://www.collabora.com/news-and-blog/news-and-events/kernel-5.18-milestones-for-the-road-ahead.html

jolting commented 1 year ago

It's enabled in sid if you can figure out how to get it working. I'm not sure if it's working fully yet. https://salsa.debian.org/chromium-team/chromium/-/commit/26e90f4253b18738fae00e4c3a6f49632cd02919

leezu commented 1 year ago

Per comments from Sep 9 and 12 2022 at https://chromium-review.googlesource.com/c/chromium/src/+/3380426 V4L2 is never used on Linux as GetPreferredLinuxDecoderImplementation never returns kV4L2 unlike the GetPreferredCrosDecoderImplementation.

gizmo98 commented 1 year ago

@leezu It does not seem to be enabled. I started a native build with these changes last wednesday. Chromium build on a quad core Cortex A53 takes approx. 6-7 days. [46900/53137] atm.

v4l2_patch.txt

jolting commented 1 year ago

@gizmo98 are you building it with your patch?

gizmo98 commented 1 year ago

@gizmo98 are you building it with your patch?

Yes. I have an i.MX8M Mini with a Hantro VPU. I use kernel 5.19 with Debian Testing and Sid. V4L2 stateless video decoding is working with gstreamer. Chromium builds now with _use_v4l2codec=true without any error message but cpu utilization is still very high.

I have found a patch for VA-API GL support. VA-API only seems to work at the moment if vulkan is used: https://chromium-review.googlesource.com/c/chromium/src/+/3752147

If you look at the code path kV4L2 will never be selected. Unlike the Chrome OS function GetPreferredCrosDecoderImplementation() the linux version GetPreferredLinuxDecoderImplementation() always returns kVDA or kVAAPI even if chromium was compiled with _use_v4l2codec=true. GetActualPlatformDecoderImplementation() switch _(GetPreferredLinuxDecoderImplementation(gpu_preferences, gpuinfo)) has no case for kV4L2. I added the code path for kV4L2. If build is finished i will report if it is working now.

If everything is correct V4L2 should run with --enable-features=UseChromeOSDirectVideoDecoder,VaapiVideoDecoder.

#if BUILDFLAG(IS_CHROMEOS)
VideoDecoderType GetPreferredCrosDecoderImplementation(
    gpu::GpuPreferences gpu_preferences) {
  // TODO(b/195769334): eventually, we may turn off USE_VAAPI and USE_V4L2_CODEC
  // on LaCrOS if we delegate all video acceleration to ash-chrome. In those
  // cases, GetPreferredCrosDecoderImplementation() won't be able to determine
  // the video API in LaCrOS.
  if (gpu_preferences.disable_accelerated_video_decode)
    return VideoDecoderType::kUnknown;
  if (gpu_preferences.enable_chromeos_direct_video_decoder) {
#if BUILDFLAG(USE_VAAPI)
    return VideoDecoderType::kVaapi;
#elif BUILDFLAG(USE_V4L2_CODEC)
    return VideoDecoderType::kV4L2;
#endif
  }
  return VideoDecoderType::kVda;
}
#else
VideoDecoderType GetPreferredLinuxDecoderImplementation(
    gpu::GpuPreferences gpu_preferences,
    const gpu::GPUInfo& gpu_info) {
  // VaapiVideoDecoder flag is required for both VDA and VaapiVideoDecoder.
  if (!base::FeatureList::IsEnabled(kVaapiVideoDecodeLinux))
    return VideoDecoderType::kUnknown;
  // Regardless of vulkan support, if direct video decoder is disabled, revert
  // to using the VDA implementation.
  if (!base::FeatureList::IsEnabled(kUseChromeOSDirectVideoDecoder))
    return VideoDecoderType::kVda;
  return VideoDecoderType::kVaapi;
}
#endif  // BUILDFLAG(IS_CHROMEOS)
VideoDecoderType GetActualPlatformDecoderImplementation(
    gpu::GpuPreferences gpu_preferences,
    const gpu::GPUInfo& gpu_info) {
#if BUILDFLAG(IS_CHROMEOS)
  return GetPreferredCrosDecoderImplementation(gpu_preferences);
#else
  // On linux, VDA and Vaapi have GL restrictions.
  switch (GetPreferredLinuxDecoderImplementation(gpu_preferences, gpu_info)) {
    case VideoDecoderType::kUnknown:
      return VideoDecoderType::kUnknown;
    case VideoDecoderType::kVda: {
      return gpu_preferences.gr_context_type == gpu::GrContextType::kGL
                 ? VideoDecoderType::kVda
                 : VideoDecoderType::kUnknown;
    }
    case VideoDecoderType::kVaapi: {
      if (gpu_preferences.gr_context_type != gpu::GrContextType::kVulkan)
        return VideoDecoderType::kUnknown;
      if (!gpu_info.vulkan_info.has_value())
        return VideoDecoderType::kUnknown;
      if (gpu_info.vulkan_info->physical_devices.empty())
        return VideoDecoderType::kUnknown;
      constexpr int kIntel = 0x8086;
      const auto& device = gpu_info.vulkan_info->physical_devices[0];
      switch (device.properties.vendorID) {
        case kIntel: {
          if (device.properties.driverVersion < VK_MAKE_VERSION(21, 1, 5))
            return VideoDecoderType::kUnknown;
          return VideoDecoderType::kVaapi;
        }
        default: {
          // NVIDIA drivers have a broken implementation of most va_* methods,
          // ARM & AMD aren't tested yet, and ImgTec/Qualcomm don't have a vaapi
          // driver.
          if (base::FeatureList::IsEnabled(kVaapiIgnoreDriverChecks))
            return VideoDecoderType::kVaapi;
          return VideoDecoderType::kUnknown;
        }
      }
    }
    default:
      return VideoDecoderType::kUnknown;
  }
#endif
}

P.S. according to GetActualPlatformDecoderImplementation() VA-API should only work if --enable-features=UseChromeOSDirectVideoDecoder, GPU context is Vulkan --enable-features=Vulkan, --enable-features=VaapiVideoDecoder is used, a intel GPU is present or --enable-features=VaapiIgnoreDriverChecks is set.

--> --enable-features=UseChromeOSDirectVideoDecoder,Vulkan,VaapiVideoDecoder,VaapiIgnoreDriverChecks

VA-API only seems to work with X11/XWayland at the moment. https://chromium-review.googlesource.com/c/chromium/src/+/3646633

VDA should work with: --disable-features=UseChromeOSDirectVideoDecoder --enable-features=VaapiVideoDecoder --use-gl=desktop

jolting commented 1 year ago

@gizmo98 Have you tried submitting this patch to gerrit? I have no idea if that will get the chrome developer's attention.

gizmo98 commented 1 year ago

@jolting build is still not finished. Lintian is running at the moment. If this patch is working i will submit it.

p.s. has someone a good guide to crosscompile chromium deb package for arm or arm64?

amazingfate commented 1 year ago

@jolting build is still not finished. Lintian is running at the moment. If this patch is working i will submit it.

p.s. has someone a good guide to crosscompile chromium deb package for arm or arm64?

Have you tried qemu+docker to compile arm binaries on x86_64?

jolting commented 1 year ago

I'm not sure if docker is easier. I'd use chroot. https://wiki.debian.org/QemuUserEmulation

You can tar up your existing rootfs, copy it over and run it on amd64.

amazingfate commented 1 year ago

I'm not sure if docker is easier. I'd use chroot. https://wiki.debian.org/QemuUserEmulation

You can tar up your existing rootfs, copy it over and run it on amd64.

Just one command docker run --rm --privileged multiarch/qemu-user-static --reset -p yes then we can run arm docker images for example: docker run --rm trapexit/debian:stretch-armhf uname -a

jolting commented 1 year ago

If you don't have prior docker experience then using chroot directly might be easier.

This is basically docker in docker. You could also do chroot in docker if you don't want to install qemu-user-static on your host system. That will allow you to re-use your existing rootfs. You can just mount your rootfs as a volume into qemu-user-static. Or you can rebuild your dev environment from scratch using @amazingfate's suggestion and save it as a docker image after you're done. That's always convenient.

Sometimes user emulation doesn't work perfectly. I've had to submit bug fixes. Let me know if you need any help.

overwatch666 commented 1 year ago

hello,I want to know if you have successfully enabled v4l2 decoder on arm chromium. I'm trying to build chromium with use_v4l2_codec=true, and I get a lot of compilation errors. Although it finally compiles successfully, it doesn't seem to take effect

gizmo98 commented 1 year ago

@overwatch666 Not yet. Chromium build on a imx8 with 2GB ram and zram takes > 3 weeks. There was also a update on the chromium mailing list. Mainline linux Mesa libgbm misses YUV buffer allocation capabilities. https://chromium-review.googlesource.com/c/chromium/src/+/3380426

There is a MR to overcome this road blocker: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23214

amazingfate commented 1 year ago

Chromium build on a imx8 with 2GB ram and zram takes > 3 weeks

If you have an launchpad account you can create a ppa repo to build chromium for you. It takes about 13 hours to build: https://launchpad.net/~liujianfeng1994/+archive/ubuntu/rockchip-multimedia/+build/25689009

gizmo98 commented 1 year ago

@amazingfate Thanks for the info! I have no account atm but this looks nice.

amazingfate commented 9 months ago

I need this patch to enable VDA on snapdragon 865 platform. I will test on a v4l2 stateless platform later.

obbardc commented 9 months ago

@gizmo98 Did you ever manage to test the changes to GetPreferredLinuxDecoderImplementation ? I ran into exactly the same function today...

amazingfate commented 8 months ago

kVDA is dropped by chromium so now we have to use kV4L2 for both stateless and stateful v4l2. Here it the patch I use for chromium v118 to enable kV4L2 on linux:

diff --git a/media/mojo/services/gpu_mojo_media_client_linux.cc b/media/mojo/services/gpu_mojo_media_client_linux.cc
index 197e369f8..eee4e419e 100644
--- a/media/mojo/services/gpu_mojo_media_client_linux.cc
+++ b/media/mojo/services/gpu_mojo_media_client_linux.cc
@@ -28,6 +28,12 @@ VideoDecoderType GetPreferredLinuxDecoderImplementation() {
     return VideoDecoderType::kOutOfProcess;
   }

+#if BUILDFLAG(USE_VAAPI)
+  return VideoDecoderType::kVaapi;
+#elif BUILDFLAG(USE_V4L2_CODEC)
+  return VideoDecoderType::kV4L2;
+#endif
+
   return VideoDecoderType::kVaapi;
 }

@@ -56,6 +62,8 @@ VideoDecoderType GetActualPlatformDecoderImplementation(
       return VideoDecoderType::kUnknown;
     case VideoDecoderType::kOutOfProcess:
       return VideoDecoderType::kOutOfProcess;
+    case VideoDecoderType::kV4L2:
+      return VideoDecoderType::kV4L2;
     case VideoDecoderType::kVaapi: {
       // Allow VaapiVideoDecoder on GL.
       if (gpu_preferences.gr_context_type == gpu::GrContextType::kGL) {

On snapdragon 865 platform with v4l2 stateful decoder I can see v4l2 decoder is detected by chromium from page chrome://gpu. But when I play a 1080p h264 video it will fall back to ffmpeg decoder. Here is the log I get:

[6207:6315:1027/191749.546680:VERBOSE1:v4l2_video_decoder.cc(454)] SetupInputFormat(): Input (OUTPUT queue) Fourcc: H264
[6207:6315:1027/191749.546803:VERBOSE1:v4l2_video_decoder.cc(431)] InitializeBackend(): Requesting: 8 OUTPUT buffers of type V4L2_MEMORY_MMAP
[6207:6315:1027/191749.556307:VERBOSE1:v4l2_video_decoder.cc(497)] SetupOutputFormat(): Output (CAPTURE queue) candidate: NV12
[6207:6315:1027/191749.556476:VERBOSE1:v4l2_video_decoder.cc(497)] SetupOutputFormat(): Output (CAPTURE queue) candidate: Q08C
[6207:6315:1027/191749.556596:VERBOSE1:video_decoder_pipeline.cc(1133)] PickDecoderOutputFormat(): Initializing ImageProcessor; max buffers: 16
[6207:6315:1027/191749.556834:VERBOSE1:video_decoder_pipeline.cc(1146)] PickDecoderOutputFormat(): Unable to find ImageProcessor to convert format
[6207:6315:1027/191749.556949:VERBOSE1:v4l2_video_decoder.cc(535)] SetupOutputFormat(): Failed to pick an output format.
[6207:6315:1027/191749.557067:ERROR:v4l2_video_decoder.cc(107)] SetupOutputFormat failed at ContinueChangeResolution@media/gpu/v4l2/v4l2_video_decoder.cc:867 : Failed to setup output format, status= 6
[6207:6315:1027/191749.557183:VERBOSE1:v4l2_video_decoder.cc(1063)] SetState(): Error occurred, stopping queues.
[6207:6315:1027/191749.557451:ERROR:v4l2_video_decoder_backend_stateful.cc(700)] Backend failure when changing resolution (9).

@gizmo98 Would YUV buffer allocation you mentioned solve this?

amazingfate commented 8 months ago

After enabing vulkan a new error come from ConvertFrame: https://github.com/chromium/chromium/blob/main/media/gpu/chromeos/mailbox_video_frame_converter.cc#L325 frame->storage_type() is STORAGE_DMABUFS instead of STORAGE_GPU_MEMORY_BUFFER.

obbardc commented 8 months ago

@amazingfate I can reproduce the same issue with a stateless decoder (rkvdec on Rockchip RK3399).

I understand that the v4l2 output buffer is allocated by the kernel in mainline and on chromiumos the buffer is allocated by the GPU using minigbm.

I also saw this implementation of a converter https://gist.github.com/kvasdopil/2d9942bcf168d75c76bfa29ca26c72aa in GL

But I think the conversion is all a bit... pointless ?

amazingfate commented 8 months ago

Decoders are always outputing YUV frames. I remember the legacy VDA is using EGL to render the frames, but VDA is not usable on chromium v118. Here is the patch I tried to enable libyuv convert:

diff --git a/media/gpu/chromeos/image_processor_factory.cc b/media/gpu/chromeos/image_processor_factory.cc
index fe1fb27f8..6f52e4bd6 100644
--- a/media/gpu/chromeos/image_processor_factory.cc
+++ b/media/gpu/chromeos/image_processor_factory.cc
@@ -174,6 +174,7 @@ std::unique_ptr<ImageProcessor> CreateLibYUVImageProcessorWithInputCandidates(
     return nullptr;

   if (input_candidates[0].fourcc != Fourcc(Fourcc::MM21) &&
+      input_candidates[0].fourcc != Fourcc(Fourcc::NV12) &&
       input_candidates[0].fourcc != Fourcc(Fourcc::MT2T)) {
     return nullptr;
   }
diff --git a/media/gpu/chromeos/libyuv_image_processor_backend.cc b/media/gpu/chromeos/libyuv_image_processor_backend.cc
index 8ee9e2117..eb77370e2 100644
--- a/media/gpu/chromeos/libyuv_image_processor_backend.cc
+++ b/media/gpu/chromeos/libyuv_image_processor_backend.cc
@@ -47,6 +47,7 @@ static constexpr struct {
 #define CONV(in, out, trans, result) \
   {Fourcc::in, Fourcc::out, Transform::trans, SupportResult::result}
     // Conversion.
+    CONV(NV12, AR24, kConversion, Supported),
     CONV(NV12, NV12, kConversion, Supported),
     CONV(YM16, NV12, kConversion, Supported),
     CONV(YM16, YU12, kConversion, Supported),
@@ -371,6 +372,9 @@ int LibYUVImageProcessorBackend::DoConversion(const VideoFrame* const input,
           fr->GetWritableVisibleData(VideoFrame::kUVPlane)), \
       fr->stride(VideoFrame::kUVPlane)

+#define ARGB_DATA(fr) \
+  fr->GetWritableVisibleData(VideoFrame::kARGBPlane), fr->stride(VideoFrame::kARGBPlane)
+
 #define LIBYUV_FUNC(func, i, o)                      \
   libyuv::func(i, o, output->visible_rect().width(), \
                output->visible_rect().height())
@@ -508,6 +512,13 @@ int LibYUVImageProcessorBackend::DoConversion(const VideoFrame* const input,
     }
   }

+  if (output->format() == PIXEL_FORMAT_ARGB) {
+    if (input_config_.fourcc == Fourcc(Fourcc::NV12)) {
+      return LIBYUV_FUNC(NV12ToARGB, Y_UV_DATA(input),
+                         ARGB_DATA(output));
+    }
+  }
+
 #undef Y_U_V_DATA
 #undef Y_V_U_DATA
 #undef Y_UV_DATA

But I get MESA: error: bo_map:580: mmap failed: Permission denied when trying to do AllocateGpuMemoryBufferHandle: https://github.com/chromium/chromium/blob/118.0.5993.127/media/gpu/chromeos/platform_video_frame_utils.cc#L170

obbardc commented 8 months ago

libyuv is in software, so this will be slow ? You may need something like https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23214 to be able to allocate YUV surfaces.

So right now it seems like we have the following:

file (somehow) -> v4l2 input buffer -> stateless device -> v4l2 output buffer -> expensive sofware yuv conversion -> yuv buffer (should be allocated by gbm?) -> screen

amazingfate commented 8 months ago

I'm already running mesa with that pr. Libyuv is software, but it should be enough to do color space convertion.

amazingfate commented 8 months ago

Finally I get V4L2VideoDecoder work with qcom venus stateful api. Here are the patches I use: https://github.com/amazingfate/chromium-libv4l2-patches/tree/main/v118.0.5993.70. And a kernel patch is needed to get better performance mentioned by Justin Green from chromium dev. And I guess this should also work with hantro stateless api. I will confirm it later.

obbardc commented 8 months ago

I tried your libyuv patch on rk3399 and kV4L2 but it doesn't seem to register the converter.

amazingfate commented 8 months ago

I tried your libyuv patch on rk3399 and kV4L2 but it doesn't seem to register the converter.

Add some log print to see why.

amazingfate commented 8 months ago

Here is the udev rules I use on rk3566 to create v4l2 devices for chromium:

SUBSYSTEM=="video4linux", ATTR{name}=="rockchip,rk3568-vpu-dec", SYMLINK+="video-dec%n"
SUBSYSTEM=="media", ATTR{model}=="hantro-vpu", SYMLINK+="media-dec%n"

And chromium v118 is packaged for jammy in my ppa: https://launchpad.net/~liujianfeng1994/+archive/ubuntu/chromium

iamdrq commented 8 months ago

Here is the udev rules I use on rk3566 to create v4l2 devices for chromium:
SUBSYSTEM=="video4linux", ATTR{name}=="rockchip,rk3568-vpu-dec", SYMLINK+="video-dec%n"
SUBSYSTEM=="media", ATTR{model}=="hantro-vpu", SYMLINK+="media-dec%n"
And chromium v118 is packaged for jammy in my ppa: https://launchpad.net/~liujianfeng1994/+archive/ubuntu/chromium

My /etc/udev/rules.d/60-chromium.rules

SUBSYSTEM=="video4linux", ATTR{name}=="rockchip,rk3399-vpu-dec", SYMLINK+="video-dec%n"
SUBSYSTEM=="video4linux", ATTR{name}=="rkvdec", SYMLINK+="video-dec%n"
SUBSYSTEM=="media", ATTR{model}=="hantro-vpu", SYMLINK+="media-dec%n"
SUBSYSTEM=="media", ATTR{model}=="rkvdec", SYMLINK+="media-dec%n"

I have a rk3399 device(just use chromium v118 https://launchpad.net/~liujianfeng1994/+archive/ubuntu/chromium), and use that udev, then v4l2 decoder work,but play stuck ,need patched mesa or kernel patch?

amazingfate commented 8 months ago

You need kernel patches, I've added them to armbian: https://github.com/armbian/build/commit/f55a87d469b0634f72bc857f773a52a466f66292

iamdrq commented 8 months ago

You need kernel patches, I've added them to armbian: armbian/build@f55a87d

I rebuild kernel with that patch,play is better, but it is not smooth,fps is low. ok,this is a hard work.

overwatch666 commented 7 months ago

hi,friends, I would like to know whether the current chromium already supports the standard v4l2. If so, or where I can find it, please tell me. Thank you.

amazingfate commented 7 months ago

hi,friends, I would like to know whether the current chromium already supports the standard v4l2. If so, or where I can find it, please tell me. Thank you.

My patches are merged upstream since chromium 121.0.6154.0. You have to run chromium with arg --disable_webgpu_shared_images=1

overwatch666 commented 7 months ago

hi,friends, I would like to know whether the current chromium already supports the standard v4l2. If so, or where I can find it, please tell me. Thank you.

My patches are merged upstream since chromium 121.0.6154.0. You have to run chromium with arg --disable_webgpu_shared_images=1

Thank you for your reply. I want to know what your parameter represents.

amazingfate commented 7 months ago

hi,friends, I would like to know whether the current chromium already supports the standard v4l2. If so, or where I can find it, please tell me. Thank you.

My patches are merged upstream since chromium 121.0.6154.0. You have to run chromium with arg --disable_webgpu_shared_images=1

Thank you for your reply. I want to know what your parameter represents.

WebGPU doesn't support CreateSharedImage, so disable it.

overwatch666 commented 5 months ago

hi,I have been using the source ppa:saiarcot895/chromium-beta before, and can successfully enable webGPU, but recently I updated a newer source ppa:xtradeb/apps, and still use the same operation, but cannot enable the GPU. When using chrome://GPU, everything is disabled, but it successfully recognizes my GPU. I don’t know if there is any special processing for ppa:saiarcot895/chromium-beta. version is Chromium 121.0.6167.160 built on Ubuntu , running on Ubuntu 22.04 The following is the relevant log. [15145:15171:0220/071241.560356:ERROR:object_proxy.cc(576)] Failed to call method: org.freedesktop.DBus.Properties.Get: object_path= /org/freedesktop/portal/desktop: org.freedesktop.DBus.Error.InvalidArgs: No such interface “org.freedesktop.portal.FileChooser” [15145:15171:0220/071241.560497:ERROR:select_file_dialog_linux_portal.cc(285)] Failed to read portal version property [15145:15145:0220/071241.570240:ERROR:policy_logger.cc(156)] :components/enterprise/browser/controller/chrome_browser_cloud_management_controller.cc(161) Cloud management controller initialization aborted as CBCM is not enabled. Please use the--enable-chrome-browser-cloud-managementcommand line flag to enable it if you are not using the official Google Chrome build. [15197:15197:0220/071241.585497:FATAL:gpu_init.cc(544)] Passthrough is not supported, GL is egl, ANGLE is [15145:15145:0220/071241.906650:ERROR:gpu_process_host.cc(992)] GPU process exited unexpectedly: exit_code=5 [15145:15145:0220/071241.944771:ERROR:object_proxy.cc(576)] Failed to call method: org.freedesktop.ScreenSaver.GetActive: object_path= /org/freedesktop/ScreenSaver: org.freedesktop.DBus.Error.NotSupported: This method is not implemented [15257:15257:0220/071241.974344:FATAL:gpu_init.cc(544)] Passthrough is not supported, GL is egl, ANGLE is [15145:15145:0220/071242.025948:ERROR:gpu_process_host.cc(992)] GPU process exited unexpectedly: exit_code=5 [15271:15271:0220/071242.060301:FATAL:gpu_init.cc(544)] Passthrough is not supported, GL is egl, ANGLE is [15145:15145:0220/071242.112546:ERROR:gpu_process_host.cc(992)] GPU process exited unexpectedly: exit_code=5

jolting commented 4 months ago

@overwatch666 I think your issue is unrelated to this one.

saiarcot895 / chromium-ubuntu-build

V4L2 Video Decoding support #65