github-actions[bot]
commented
2 months ago Download the artifacts for this pull request:
Closed Kwiboo closed 1 month ago
github-actions[bot]
commented
2 months ago Download the artifacts for this pull request:
philipl
commented
2 months ago Separately, does this approach have agreement from the various SoC communities? The rockchip and rpi forks have been around for ages and there are a bunch of applications and application forks that assume the HWDEVICE_TYPE_DRM approach. Are they going to change to follow this - otherwise it won't solve our fork problem.
Kwiboo
commented
2 months ago Obviously, we need to see how the ffmpeg side reviews go and what the final state is, but actually getting upstream support for v4l2request is a big deal.
Fully agree, last time around (2019 and 2020) there was limited interest and after RPi4 launched there was hope/indication that RPi could assist on getting something upstream, unfortunately that never happened.
RFC: http://ffmpeg.org/pipermail/ffmpeg-devel/2019-April/242316.html v1: https://patchwork.ffmpeg.org/project/ffmpeg/cover/20201209202513.27449-1-jonas@kwiboo.se/
This time around I have tried to fix feedback from the RFC and v1, include improvements along the way made by others in LibreELEC team and add similar performance improvements (using multiple requests) made in the RPi fork (that fork mainly focused on rpivid and HEVC). Hoping to have a v2 series on ffmpeg-devel list later this week/weekend.
I don't know what additional state from AVCodecHWConfig you need to make a decision on initialising the hwdec, but if you do need more, you will need to take the approach of adding more fields to hwdec_info and passing the additional metadata in
add_all_hwdec_methodsinvideo/decode/vd_lavc.c. That's currently the only place whereavcodec_get_hw_configis called.
The main thing is that the AVCodecHWConfig.device_type can be used to indicate what hwdevice should be used/created by the application to use that specific hwaccel to be able to produce output in AVCodecHWConfig.pix_fmt.
Main inconvenience in mpv seem to be that there is no real distinction between the imgfmt and hwcontext, however ffmpeg with the help of AVCodecHWConfig can hint for applications what output (pix_fmt), what method hwdevice/hwframes/internal (methods) and what hwdevice type (device_type) is required to use a hwaccel.
Kodi already used the device_type to limit what AVCodecHWConfig it supported, with xbmc/xbmc#25467 this was relaxed and make Kodi work with both HWDEVICE_TYPE_DRM, HWDEVICE_TYPE_V4L2REQUEST or any possible future hwdevice that can produce AV_PIX_FMT_DRM_PRIME frames.
Adding a new imgfmt to support the v4l2request hwdevice and drm_prime pix fmt combo could possible be done, but that seem more like a quick fix and I am instead hoping that maybe a mpv developer can pick this up to make better use of the AVCodecHWConfig information.
Kwiboo
commented
2 months ago Separately, does this approach have agreement from the various SoC communities? The rockchip and rpi forks have been around for ages and there are a bunch of applications and application forks that assume the
HWDEVICE_TYPE_DRMapproach. Are they going to change to follow this - otherwise it won't solve our fork problem.
The current required use of HWDEVICE_TYPE_DRM has typically also been accompanied with a separate hack in ffmpeg that make it possible to use HWDEVICE_TYPE_DRM without a path to a working drm device, something that was rejected in upstream last time around.
Decoding with V4L2 Request API has really nothing to do with a drm device, and was originally only used because of little to no knowledge into the inner workings of FFmpeg, the only thing that was known was that we wanted to produce frames using the AV_PIX_FMT_DRM_PRIME format and using the HWDEVICE_TYPE_DRM was one way to make that happen.
Two reasons why we should not continue to use HWDEVICE_TYPE_DRM:
V4L2 decoding has really nothing to to with a drm device, e.g. on rk3588 this became apparent when there was no drm/gpu driver support in mainline, we can still use a V4L2 media/video device to decode video, just not present them on a display.
With the addition of a separate hwdevice type this can be used to help assist what v4l2 media/video device should be used to decode. The RPi fork uses env vars, but we should really be able to specify it using the FFmpeg command line, e.g.:
ffmpeg -init_hw_device v4l2request:/dev/media1 -hwaccel v4l2request -hwaccel_output_format drm_prime ...OK. I can try and take a look in the next few weeks. There's no kernel patches required right? It's enough to have the ffmpeg patches on top of upstream?
Kwiboo
commented
2 months ago OK. I can try and take a look in the next few weeks.
Great, thanks!
There's no kernel patches required right? It's enough to have the ffmpeg patches on top of upstream?
Correct, there should not be any need for kernel patches, uapi headers should all be upstream and last change was included in v6.5-rc1 (addition of AV1 and a new parameter in a HEVC control). The ffmpeg patches have ifdefs to adopt depending on the installed version of kernel uapi headers.
The ffmpeg patches should work with current mainline kernel v4l2 stateless drivers, I am testing with cedrus, hantro and rkvdec. For RPi and the rpivid driver I test with the downstream rpi-6.6.y kernel.
I do not have any hw to test the mtk-vcodec or tegra-vde drivers, they are using drm formats that is missing in the pix format mapping in ffmpeg patches so will not fully work, should hopefully only be a matter of mapping their v4l2 pixel format to a drm pixel format to also support these drivers.
philipl
commented
2 months ago @Kwiboo Ok. I got your patches applied and working on my Chromebook Plus (RK3399) running Arch (BTW). What video decoding formats should work? I can only get h.264 and mpeg2 to work.
I also haven't seen any patches sent to ffmpeg-devel.
Kwiboo
commented
2 months ago I also haven't seen any patches sent to ffmpeg-devel.
Sorry for the delay, I was not able to finish getting the patches out the door before some vacation time. I am now back working on the patches and will run a final round of tests with latest ffmpeg master before I send them out, hoping later today.
I got your patches applied and working on my Chromebook Plus (RK3399) running Arch (BTW). What video decoding formats should work? I can only get h.264 and mpeg2 to work.
FFmpeg patches is currently limited to mpeg2, h264 and hevc for now, I did not manage to finish looking over the vp8/vp9/av1 support before vacation time, they will be added in a follow up ffmpeg series (or a later patch series revision).
Current kernel drivers for rk3399 should support mpeg2, h264, vp8 and vp9.
To support h264 hi10 and hevc on rk3399 two kernel series for rkvdec is needed, new revision with minimal changes should hit linux-media list soon, hoping tomorrow, and hopefully they are merged in time for linux v6.12.
Also note that the latest set of ffmpeg patches now enforce a limitation that is reported by the kernel driver related to frame sizes. Kernel may need a fix to correctly report supported frame sizes, for now 16 pixel aligned width/height videos should work with existing kernel drivers.
philipl
commented
2 months ago Great. I've got a PoC here of an approach that I'm happy with; I should have some patches up later this week.
philipl
commented
2 months ago Alright. I've put up #14639 which adds the necessary logic to handle the different device types. And then I have an initial version of the v4l2request hwdecs based on your code here but now no longer breaking the regular drmprime support.
https://github.com/philipl/mpv/tree/v4l2request
It still needs more work to refactor properly, but please try it out.
Kwiboo
commented
2 months ago Perfect, will test it out and get back with results.
FFmpeg v4l2-request patches is now on ffmpeg-devel list at https://lists.ffmpeg.org/pipermail/ffmpeg-devel/2024-August/332034.html
Kwiboo
commented
2 months ago https://github.com/philipl/mpv/tree/v4l2request
It still needs more work to refactor properly, but please try it out.
@philipl I gave it a go and seem to work for the most part, had to add ra_hwdec_v4l2request_overlay to ra_hwdec_drivers to make v4l2request-overlay work, see fixup: vo: hwdec: drmprime: [WIP] add separate hwdec for v4l2request.
See https://github.com/Kwiboo/mpv/commits/v4l2request-test-20240808/ for the code that I tested.
I have tested using vanilla aarch64 ubuntu 24.04 on a RK3399 device (ROCK Pi 4B+ v1.73) together with ppa:kisak/kisak-mesa to get newer mesa, self-compiled ffmpeg using ./configure --enable-v4l2-request and mpv using v4l2request=enabled, egl-drm=enabled, egl-wayland=enabled and wayland=enabled.
However there are a few minor and/or strange issues:
Use of --vo=gpu-next ends with a Segmentation fault (core dumped), have not tried to investigate this.
I had to allow use of cursor planes to be used as video/draw planes on RK3399, the RK3399 will default to use its limited display controller (vop-lit) for console (crtc-0).
Fixed by commit drm_atomic: allow use of cursor planes to ignore the legacy plane type indicator and allow render to all planes. Or can be worked around with the kernel series https://lore.kernel.org/linux-rockchip/20240615170417.3134517-1-jonas@kwiboo.se/ that will reorder vop-big as crtc-0 and vop-lit as crtc-1.
Playback of one of my sample videos H264_10_1080i_50_AC3-Astra19.2_ProSieben_HD.ts using wayland and egl rendering cause frame to "jump" back and forth, not sure if this is a rendering issue, ffmpeg issue or mpv issue.
Playback using v4l2request-overlay or v4l2request on console using egl-drm does not seem to have this issue.
Remember seeing similar playback issue in early kodi-gbm and hantro/rkvdec development days when implementing support for field encoded H264. Or possible when a frame reference was not held until after the frame is no longer on screen.
On console using v4l2request-overlay:
./mpv --hwdec=v4l2request --hwdec-codecs=all --gpu-hwdec-interop=v4l2request-overlay --drm-drmprime-video-plane=0 --drm-draw-plane=1 --no-audio <video-path>In wayland:
./mpv --hwdec=v4l2request --hwdec-codecs=all <video-path>3. Playback of one of my sample videos
H264_10_1080i_50_AC3-Astra19.2_ProSieben_HD.tsusingwaylandandeglrendering cause frame to "jump" back and forth, not sure if this is a rendering issue, ffmpeg issue or mpv issue.
Playback of jellyfish-3-mbps-hd-h264.mkv under wayland is also affected by same rendering issue. Kodi under wayland does not suffer from this issue.
Could be possible that mpv is releasing the AVFrame before it is done being presented on screen, meaning FFmpeg is free to continue decoding and overwriting the framebuffer tied to that frame.
philipl
commented
1 month ago Did you try with --vo=dambuf-wayland? Any difference?
Kwiboo
commented
1 month ago Did you try with
--vo=dambuf-wayland? Any difference?
Thanks, that worked much better!
We have noticed that the new ffmpeg v4l2-request code help trigger a decoding issue only seen using hantro on armv7 (rk3288 / rk322x). Everything points to it being specific to just the hantro driver, as rkvdec work on same devices and cedurs work on e.g. Allwinner armv7 (H5) devices. This may delay sending an updated ffmpeg v4l2-request v3 series.
philipl
commented
1 month ago Did you try with
--vo=dambuf-wayland? Any difference?Thanks, that worked much better!
Great. We strongly recommend that people use dmabuf-wayland with wayland on these kinds of devices as it gives the most direct, lowest overhead path to the display as the wayland compositor should ideally be doing direct scanout.
llyyr
commented
1 month ago I don't have hardware to test, but I don't think gpu{-next} should be presenting frames jumping back and forth even if that is the case. I wonder if it's similar to #13811
kasper93
commented
1 month ago rendering cause frame to "jump" back and forth
I cannot reproduce with your samples, but I've seen that with interlaced dvb-t samples on amd. Sometimes frames start jumping back and forth, other times it just stutter a lot, and video decode engine is at 100%.
It happens only with limited number of samples. This is good example, https://github.com/GPUOpen-LibrariesAndSDKs/AMF/files/11082371/Sample2_.2750.zip if you start playback from beginning. Sometimes seeking recovers it.
I'm not sure if this is comparable to your case. I'm sure at one point (years ago) those samples were working fine, so it got regressed in AMD driver, but question is if ffmpeg is not doing something unexpected.
It happens also with LAV Filters on Windows, so it is not mpv related. But since we are discussing completely unrelated platforms, it's probably different issues...
See: https://github.com/Nevcairiel/LAVFilters/issues/496 https://github.com/GPUOpen-LibrariesAndSDKs/AMF/issues/380 https://github.com/mpv-player/mpv/issues/13811
Kwiboo
commented
1 month ago @llyyr @kasper93 the issue I noticed without use of dmabuf-wayland look similar to #13811. The best I can describe it would be that the decoder is writing and changing to the framebuffer currently being presented on screen, shows on screen similar to tearing without a double buffer and/or vsync.
Also I do not think that this wayland rendering issue is really an issue worth pursing, with use of dmabuf-wayland playback is working as intended.
Kwiboo
commented
1 month ago This RFC is now obsolete, see #14690 for a proper implementation that can work with both AV_HWDEVICE_TYPE_DRM and AV_HWDEVICE_TYPE_V4L2REQUEST.
Current out-of-tree FFmpeg V4L2 Request API hwaccel patches abuse the drm hwdevice type.
Patches being prepared for upstream FFmpeg changes this to use a new dedicated v4l2request hwdevice type.
This PR add basic support for using the new v4l2request hwdevice type, however, it cripples old use of
AV_HWDEVICE_TYPE_DRM, and possible break use with e.g.rkmppdecoders.Please assist on how the the hwdevice created based on the picked
AVCodecHWConfigcan be re-used instead of recreating a new hwdevice.Ideally
mpvshould behave similar to e.g.ffmpeg -init_hw_device v4l2request, or implicitly create the hwdevice based on theAVCodecHWConfig.device_type.With the updated patches one would instead of
drmusev4l2requestas the-hwaccel, e.g.:Created as a draft PR because work-in-progress FFmpeg patches are still being worked on.