search

Elyotna / linux

Hacking in a V4L2 M2M decoder for AMLogic SoCs
Other
18 stars 4 forks source link

Unexpected poll() behavior on /dev/video0 during h264 decoding #3

Open tmm1 opened 4 years ago

tmm1 commented 4 years ago

I'm working on some patches for the v4l2 decoder in ffmpeg, and in testing on rpi3/rpi4/lepotato I have noticed some unexpected behavior in how the meson-vdec driver behaves during decoding.

In short, ffmpeg will often hang indefinitely inside poll(POLLIN | POLLRDNORM | POLLPRI | POLLOUT | POLLWRNORM, -1). When the decoder is not able to produce frames, I would expect it to return with POLLOUT set to indicate that it needs more data. This happens more readily when the driver is configured with fewer capture buffers (i.e. 8 output buffers and 2 capture buffers).

tmm1 commented 4 years ago

Here is my patchset on top of ffmpeg master which surfaces this issue readily: https://github.com/FFmpeg/FFmpeg/compare/master...tmm1:v4l2-dec-flow

It works as expected with the bcm2835-codec v4l2 driver on the rpi.

I'm using a 5.3.0-rc7 kernel with the commits from your mjourdan/v5.3/aml/vdec branch including https://github.com/Elyotna/linux/commit/eed65f15fa53a27b62856d6dc37e3f6c84df320f

warpme commented 4 years ago

Maxime, It looks like also I have probably similar issue on s905, v4l2 m2m decode under mythtv. mythtv playback process enters endless log output like this:

2019-09-18 10:39:05.524353 I  TV::HandleStateChange(): Changing from None to WatchingPreRecorded
2019-09-18 10:39:05.530615 D  [NULL @ 0000ffff946f3d88] nal_unit_type: 7(SPS), nal_ref_idc: 3
2019-09-18 10:39:05.530669 D  [NULL @ 0000ffff946f3d88] nal_unit_type: 8(PPS), nal_ref_idc: 3
2019-09-18 10:39:05.530699 D  [NULL @ 0000ffff946f3d88] nal_unit_type: 8(PPS), nal_ref_idc: 3
qt.qpa.events: Event | XCB_PROPERTY_NOTIFY(28) | sequence: 613
2019-09-18 10:39:05.538155 D  [NULL @ 0000ffff946f3d88] ct_type:0 pic_struct:1
2019-09-18 10:39:05.540715 I  TV::HandleStateChange(): Main UI disabled.
2019-09-18 10:39:05.540768 D  TV::HandleStateChange(): (0) -- end
2019-09-18 10:39:05.541257 D  TV::StartTV(): tv->Playback() -- end
2019-09-18 10:39:05.541385 I  TV::StartTV(): Entering main playback loop.
2019-09-18 10:39:05.544043 D  [h264_v4l2m2m @ 0000ffff946f3d88] VIDIOC_STREAMON on output context
2019-09-18 10:39:05.544304 D  [NULL @ 0000ffff946f3d88] ct_type:0 pic_struct:2
2019-09-18 10:39:05.545865 D  ScreenSaverX11Private: ResetTimer -- begin
2019-09-18 10:39:05.545991 D  ScreenSaverX11Private: StopTimer
2019-09-18 10:39:05.546386 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.546664 D  [h264_v4l2m2m @ 0000ffff946f3d88] VIDIOC_STREAMON on output context
2019-09-18 10:39:05.546789 D  [NULL @ 0000ffff946f3d88] ct_type:0 pic_struct:1
2019-09-18 10:39:05.546979 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.547004 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.547165 D  [h264_v4l2m2m @ 0000ffff946f3d88] VIDIOC_STREAMON on output context
2019-09-18 10:39:05.547278 D  [NULL @ 0000ffff946f3d88] ct_type:0 pic_struct:2
2019-09-18 10:39:05.547444 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.547465 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.547481 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.547577 D  [h264_v4l2m2m @ 0000ffff946f3d88] VIDIOC_STREAMON on output context
2019-09-18 10:39:05.547652 D  [NULL @ 0000ffff946f3d88] ct_type:0 pic_struct:1
2019-09-18 10:39:05.547796 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.547819 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.547837 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.547854 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.547939 D  [h264_v4l2m2m @ 0000ffff946f3d88] VIDIOC_STREAMON on output context
2019-09-18 10:39:05.548020 D  [NULL @ 0000ffff946f3d88] ct_type:0 pic_struct:2
2019-09-18 10:39:05.548151 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.548168 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.548185 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.548201 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.548216 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.548306 D  [h264_v4l2m2m @ 0000ffff946f3d88] VIDIOC_STREAMON on output context
2019-09-18 10:39:05.548382 D  [NULL @ 0000ffff946f3d88] ct_type:0 pic_struct:1
2019-09-18 10:39:05.548509 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.548530 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.548546 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.548563 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.548579 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.548597 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.548683 D  [h264_v4l2m2m @ 0000ffff946f3d88] VIDIOC_STREAMON on output context
2019-09-18 10:39:05.548894 D  [NULL @ 0000ffff946f3d88] ct_type:0 pic_struct:2
2019-09-18 10:39:05.549090 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.549111 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.549127 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.549142 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.549158 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.549175 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.549190 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.549296 D  [h264_v4l2m2m @ 0000ffff946f3d88] VIDIOC_STREAMON on output context
2019-09-18 10:39:05.549956 D  [NULL @ 0000ffff946f3d88] ct_type:0 pic_struct:1
2019-09-18 10:39:05.550595 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.550623 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.550641 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.550657 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.550673 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.550690 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.550707 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.550722 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.550880 D  [h264_v4l2m2m @ 0000ffff946f3d88] VIDIOC_STREAMON on output context
2019-09-18 10:39:05.551030 D  [NULL @ 0000ffff946f3d88] ct_type:0 pic_struct:2
2019-09-18 10:39:05.552036 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.552064 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.552080 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.552096 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.552111 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.552127 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.552143 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.552159 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.552175 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.552357 D  [h264_v4l2m2m @ 0000ffff946f3d88] VIDIOC_STREAMON on output context
2019-09-18 10:39:05.552472 D  [NULL @ 0000ffff946f3d88] ct_type:0 pic_struct:1
2019-09-18 10:39:05.552638 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.552659 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.552675 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.552690 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.552705 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.552720 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.552735 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.552751 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.552767 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.552783 W  [h264_v4l2m2m @ 0000ffff946f3d88] output  POLLERR
2019-09-18 10:39:05.552891 D  [h264_v4l2m2m @ 0000ffff946f3d88] VIDIOC_STREAMON on output context

Interesting that testing amlogic v4l2 m2m video by: mythffmpeg -c:v h264_v4l2m2m -i /usr/local/share/h264.FVDO_Freeway_720p.264 -f null -

works ok:

ffmpeg version e867514 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 7.4.0 (GCC)
  configuration: --arch=aarch64 --cross_prefix=aarch64-minimyth-linux-gnu- --sysroot=/home/piotro/minimyth-dev/images/main --sysinclude=/home/piotro/minimyth-dev/images/main/usr/include --cc=aarch64-minimyth-linux-gnu-gcc --cxx=aarch64-minimyth-linux-gnu-g++ --ld=aarch64-minimyth-linux-gnu-gcc --cpu=generic --target_os=linux --pkg_config=//home/piotro/minimyth-dev/images/build/usr/bin/pkg-config --prefix=/home/piotro/minimyth-dev/images/main/usr --libdir=/home/piotro/minimyth-dev/images/main/usr/lib --as=aarch64-minimyth-linux-gnu-gcc --objcc=aarch64-minimyth-linux-gnu-gcc --dep_cc=aarch64-minimyth-linux-gnu-gcc --host_cc=gcc --host_ld=gcc --enable-cross-compile --disable-libmp3lame --disable-libx264 --disable-libx265 --disable-libvpx --disable-libxvid --disable-vdpau --enable-libxml2 --enable-libass --disable-dxva2 --enable-libbluray --disable-libfontconfig --disable-libfreetype --disable-libiec61883 --disable-crystalhd --disable-sdl2 --disable-ffplay --extra-cflags=' -pipe -pipe -march=armv8-a -O2 -I/home/piotro/minimyth-dev/images/main/usr/include/avahi-compat-libdns_sd -fPIC -w' --extra-ldflags='-Wl,--as-needed -pipe -pipe -march=armv8-a -O2' --enable-stripping --strip='echo skipping strip' --disable-manpages --disable-podpages --disable-doc --disable-nvenc --enable-shared --disable-static --enable-gpl --enable-pic --disable-demuxer=mpegtsraw --disable-indev=dshow
  libavutil      56. 22.100 / 56. 22.100
  libavcodec     58. 35.100 / 58. 35.100
  libavformat    58. 20.100 / 58. 20.100
  libavdevice    58.  5.100 / 58.  5.100
  libavfilter     7. 40.101 /  7. 40.101
  libswscale      5.  3.100 /  5.  3.100
  libswresample   3.  3.100 /  3.  3.100
  libpostproc    55.  3.100 / 55.  3.100
Input #0, h264, from '/usr/local/share/h264.FVDO_Freeway_720p.264':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: h264 (High), yuv420p(progressive), 1280x720, 25 fps, 25 tbr, 1200k tbn, 50 tbc
[h264_v4l2m2m @ 0x3f78c910] driver 'meson-vdec' on card 'Amlogic Video Decoder'
[h264_v4l2m2m @ 0x3f78c910] Using device /dev/video0
[h264_v4l2m2m @ 0x3f78c910] driver 'meson-vdec' on card 'Amlogic Video Decoder'
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (h264_v4l2m2m) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf58.20.100
    Stream #0:0: Video: wrapped_avframe, nv12, 1280x720, q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc
    Metadata:
      encoder         : Lavc58.35.100 wrapped_avframe
frame=  228 fps=211 q=-0.0 Lsize=N/A time=00:00:10.00 bitrate=N/A speed=9.27x

also: testing drm with v4l2 m2m decode by: ffmpeg-drm --video /usr/local/share/h264.FVDO_Freeway_720p.264 --codec h264_v4l2m2m --width 1280 --height 720

works ok

For me it looks like real life cooperation between amlogic v4l2 video and ffmpeg 4.0.2 has some issues....

warpme commented 4 years ago

Oh - btw: I forgot to mention: when I try playback on mythtv (and mythtv-vdec enters this endless POOLERR state) - hw decoder enters errant state where mythffmpeg and ffmpeg-drm stops to work correctly. mythffmpeg reports in such case:

Input #0, h264, from '/usr/local/share/h264.FVDO_Freeway_720p.264':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: h264 (High), yuv420p(progressive), 1280x720, 25 fps, 25 tbr, 1200k tbn, 50 tbc
[h264_v4l2m2m @ 0x33b610c0] driver 'meson-vdec' on card 'Amlogic Video Decoder'
[h264_v4l2m2m @ 0x33b610c0] Using device /dev/video0
[h264_v4l2m2m @ 0x33b610c0] driver 'meson-vdec' on card 'Amlogic Video Decoder'
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (h264_v4l2m2m) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
[h264_v4l2m2m @ 0x33b610c0] output  POLLERR
    Last message repeated 3575 times

only solution to recover is hw reset by reboot...

tmm1 commented 4 years ago

@warpme Can you try with ffmpeg from master? I have made many commits to the v4l2 codec implementation including importing several amlogic related fixes from the libreelec ffmpeg fork.

warpme commented 4 years ago

Hi Aman, It will be great to test on current master of ffmpeg but in my usecase (mythtv) it is not so easy: myth is using integrated ffmpeg (currently 4.0.2 level) as there is many hooks & changes related to some specific myth functionality (closed captions for tv, subtitles, etc). This make replacing build-in ffmpeg with new version quite difficult. I can however try back port Your work to myth's 4.0.2 by applying patches to myth's ffmpeg (assuming it will apply cleanly). Is it to possible to provide me ordered list of patches? br

Elyotna commented 4 years ago

@tmm1 Hi, thanks for working on the ffmpeg v4l implementation!

The issues you're having with the meson/vdec is most likely tied to the amount of capture buffers. There is a minimum amount you need to comply with depending on the bitstream parameters, which is indicated by the driver's control V4L2_CID_MIN_BUFFERS_FOR_CAPTURE after it parsed some bitstream.

The next issue you'll hit however is that the driver is currently not spec compliant, and if your rework of the ffmpeg v4l2 implementation is to make it spec compliant, then you're likely to run into other issues.

warpme commented 4 years ago

Maxime,

The next issue you'll hit however is that the driver is currently not spec compliant, and if your rework of the ffmpeg v4l2 implementation is to make it spec compliant, then you're likely to run into other issues.

I'm a bit struggling to get conclusion from above statement. What is Your advice regarding future development of app. exploiting amlogic v4l2 m2m vdec (like i.e. mythtv appliance)?

tmm1 commented 4 years ago

The issues you're having with the meson/vdec is most likely tied to the amount of capture buffers. There is a minimum amount you need to comply with depending on the bitstream parameters, which is indicated by the driver's control V4L2_CID_MIN_BUFFERS_FOR_CAPTURE after it parsed some bitstream.

That matches my experience where the driver worked more reliably when larger numbers of capture buffers were provided. I had tried querying V4L2_CID_MIN_BUFFERS_FOR_CAPTURE before, but it always returned 1. I guess I need to try it again after enqueuing some of the bitstream- can you advise as to how much of the bitstream must be provided to determine the capture buffers required? In the case of ffmpeg, all the buffers are pre-allocated so modifying the number of capture buffers after decoding begins will require some refactoring.

Elyotna commented 4 years ago

I'm a bit struggling to get conclusion from above statement. What is Your advice regarding future development of app. exploiting amlogic v4l2 m2m vdec (like i.e. mythtv appliance)?

That matches my experience where the driver worked more reliably when larger numbers of capture buffers were provided. I had tried querying V4L2_CID_MIN_BUFFERS_FOR_CAPTURE before, but it always returned 1. I guess I need to try it again after enqueuing some of the bitstream- can you advise as to how much of the bitstream must be provided to determine the capture buffers required? In the case of ffmpeg, all the buffers are pre-allocated so modifying the number of capture buffers after decoding begins will require some refactoring.

In the past few months the stateful v4l2 m2m decoding spec was written by v4l developers, and subsequently most drivers are currently being patched to comply with it. This will also force some of the userspace to change, since most v4l decoding stacks are not compliant. I think the implementations that come closest to compliance are the ones from chromium/android. While breaking current userspace is never nice for anyone, there was no official spec and every driver needed quirks to work properly (up to some drivers functioning vastly differently from others), which is not desirable.

For instance, in the new spec, userspace is supposed to STREAMON the OUTPUT queue, fill in some bitstream buffers, and wait for an event from the driver that it parsed enough bitstream to proceed. Once the event is received, things like V4L2_CID_MIN_BUFFERS_FOR_CAPTURE and coded resolution are populated, and userspace can properly configure the CAPTURE queue.

I am currently in the process of bringing the meson vdec up to spec but it's a lengthy process, hence its staging status in the kernel tree.

You can find the latest spec here: https://www.mail-archive.com/linux-media@vger.kernel.org/msg149210.html

warpme commented 4 years ago

Maxime, Thx. this clears things considerably.

I'm curious: what is You hunch about how things will go in future?

I understand first milestone is making v4l2 api spec GA. Do You foreseen this in weeks or months or years or?

Next I think will be 2 activities: 1.kernel driver guys (like You) will need to update code to spec. 2.userspace guys (like ffmpeg) will also need to update to spec. is this correct thinking?

also i'm wonder how Kodi switch to unified v4l2 m2m will help here? will it be "wait till spec is GA" or rather "we are waiting too long for GA - so go with workarounds"?

xlazom00 commented 4 years ago

@Elyotna just off topic Do you have patch for gstreamer so driver will work? :)