is it possible to support AV1 decoding?

[JeffyCN]

it's not hard to support more(mpp supported) formats here, but the thing is that i'm not sure if chromium supports that in v4l2vda framework

[amazingfate]

Here is a upstream commit merged in v108: https://chromium-review.googlesource.com/c/chromium/src/+/3857237.

[JeffyCN]

well, i don't have environment to test r108, i can try to make an untested patch based on that commit this week

[amazingfate]

Thank you very much! I can help testing it.

[JeffyCN]

Thank you very much! I can help testing it.

thanks, i have uploaded https://github.com/JeffyCN/libv4l-rkmpp/tree/wip/av1 1/ it seems like only rk3588 supports av1. 2/ that commit is for stateless decoder, we are actually using stateful decoder, so maybe you need to add something like this for av1: https://github.com/JeffyCN/meta-rockchip/blob/master/dynamic-layers/recipes-browser/chromium/chromium_105.0.5195/0015-media-Enable-HEVC-by-default-for-V4L2VDA.patch profiles: https://chromium.googlesource.com/chromium/src/+/master/media/base/video_codecs.h#90

[amazingfate]

I modified patches to make them applied to chromium r108: https://github.com/amazingfate/meta-rockchip/tree/chromium-108/dynamic-layers/recipes-browser/chromium/chromium_108.0.5359. But when I run the built chromium with env LIBV4L_RKMPP_LOG_LEVEL=10, I can't see any rkmpp related output. It seems that the v4l plugin is not loaded.

[JeffyCN]

1/ you can ads log in vda to make sure it's initialize() been called

2/ you can try to add logs in v4l2_device.cc to make sure it called libv4l

3/ check chrome://gpu to make sure hardware video decoding is enabled

[amazingfate]

Because of this upstream commit v4l2 plugin is not loaded. I've fixed that: https://github.com/amazingfate/meta-rockchip/blob/chromium-108/dynamic-layers/recipes-browser/chromium/chromium_108.0.5359/0001-Add-support-for-V4L2-VDA-and-VEA-on-Linux.patch. Now using branch wip/av1 I can see av1 is supported from page chrome://gpu. But when I opened 1080p av1 videos on bilibili, my board crashed. That should be a kernel crash because the heartbeat led of my board stopped.

[amazingfate]

Because of this upstream commit v4l2 plugin is not loaded. I've fixed that: https://github.com/amazingfate/meta-rockchip/blob/chromium-108/dynamic-layers/recipes-browser/chromium/chromium_108.0.5359/0001-Add-support-for-V4L2-VDA-and-VEA-on-Linux.patch. Now using branch wip/av1 I can see av1 is supported from page chrome://gpu. But when I opened 1080p av1 videos on bilibili, my board crashed. That should be a kernel crash because the heartbeat led of my board stopped.

Maybe this is a mpp related issue rockchip-linux/mpp#349

[amazingfate]

@JeffyCN which kernel you're using that supports av1 decoding? I am using the latest gstreamer-rockchip from https://github.com/JeffyCN/rockchip_mirrors/tree/gstreamer-rockchip, and kernel tag is linux-5.10-gen-rkr3.6, mpp is that latest from https://github.com/rockchip-linux/mpp/tree/develop/. When I use gst-play-1.0 to play an av1 video, I get a green screen. Here is the syslog: https://paste.ubuntu.com/p/gNYzHsnhZn/.

[JeffyCN]

i'm using sdk's newest kernel(might not for the public users), maybe you can mail and ask the mpp maintainers(they also maintain the related kernel drivers)

[amazingfate]

I can make av1 decodings work after adding av1d and av1d_mmu to devicetree. Now I can play av1 videos from local file and youtube. But av1 videos on bilibili are strange. I made a screen record for it: https://www.bilibili.com/video/BV13Y4y1f7R7/.

[amazingfate]

@JeffyCN I think we can merge this av1 decode support to master. Issues when playing bilibili videos should be related to the encoding of bilibili or the decoding of mpp.

[JeffyCN]

done