Gstreamer errors on specific h264 bytestream on Bullseye and Buster, works if Buster has firmware downgraded.

[icecube45]

Hi there, I work with a project that utilizes a C++ embedded gstreamer pipeline to process and display a h264 stream of video. This pipeline, on a raspberry pi 4, resolves to:

appsrc name=mysrc is-live=true block=false max-latency=100 do-timestamp=true stream-type=stream ! queue ! h264parse ! v4l2h264dec ! videocrop top=0 bottom=0 name=videocropper ! capsfilter caps=video/x-raw name=mycapsfilter ! qtquick2videosink

(a dotfile was captured for this pipeline, here)

We're experiencing an issue where some Pi 4s updated to, or past, commit b4e395b3e87dba4964f314e12871630cabb35f70 error out on this pipeline, throwing

0:00:29.404484942 23266 0xa330e000 WARN            videodecoder gstvideodecoder.c:1161:gst_video_decoder_sink_event_default:<v4l2h264dec0> error: No valid frames decoded before end of stream
0:00:29.404541645 23266 0xa330e000 WARN            videodecoder gstvideodecoder.c:1161:gst_video_decoder_sink_event_default:<v4l2h264dec0> error: no valid frames found

when these affected pis are downgraded below the stateful decode update commit, the embedded gstreamer pipeline works as expected.

Furthermore, these afflicted Pis can properly execute gst-launch-1.0 filesrc location=testfile ! h264parse ! v4l2h264dec ! video/x-raw,format=I420 ! autovideosink where testfile was recorded with appsrc name=mysrc is-live=true block=false max-latency=100 do-timestamp=true stream-type=stream ! filesink location=testfile

These afflicted Pis can also properly execute an embedded pipeline of appsrc name=mysrc is-live=true block=false max-latency=100 do-timestamp=true stream-type=stream ! queue ! h264parse ! avdec_h264 ! videocrop top=0 bottom=0 name=videocropper ! capsfilter caps=video/x-raw name=mycapsfilter ! qtquick2videosink

Out of four Pi 4s this has been tested one, two were afflicted, and two worked even post the stateful decode update commit. All tested pis were on bullseye.

Afflicted revision numbers: c03112 b03114

Unafflicted revision number: b03111 a03111

I'd appreciate any help in resolving this manner, as I'm fully out of my depth debugging this issue further. Please let me know if you need any additional information.

[JamesH65]

@6by9 Not sure if you will have seen this so flagging up just in case.

[icecube45]

Found another pi 4 that is unafflicted, revision code a03111

If I had to wager a guess here - if I'm reading how to decode revision codes correctly, the only link I see is that this isn't affecting revision 1 pi 4s

[JamesH65]

It seems extremely unlikely (not impossible) this is revision based - the differences are on the whole unrelated to the issue being described. I'd suspect a timing problem, but @6by9 is more likely to figure it out than I am.

[popcornmix]

I agree the board revision is unlikely to be significant. Can you confirm that you are testing the same sdcard on each of the four Pi4's (just in case there is something on some of the sdcards that has an effect)? And can you confirm this is repeatable? (i.e. ten passes on good revision and ten fails on bad revision)?

[icecube45]

This has been a different sdcard per Pi (one of them was USB boot, as well), but all set up using the same installation script on top of vanilla raspbian bullseye. I can confirm that it's repeatable - the non working pis consistently don't work past the commit given, consistently do prior - the working Pis continue to work throughout. I don't have an exact number, but they've definitely been tested at least 7-10 times through the course of me trying to figure out what's going on.

[popcornmix]

Are you able to swap sdcard between a working and non-working board and confirm the problem remains with the board (and not the sdcard)?

[popcornmix]

A long shot - add over_voltage=2 to config on a bad pi. Any improvement?

[icecube45]

Swapping the physical sdcard won't be possible here due to the distances between pis, but what I can do is take an image of the working pi's sdcard, and flash it into the sdcard of a non working pi.

Will report back with results (and over_voltage test as well) when I have them.

[icecube45]

Tested b03111, a03111, c03112 on the same image (https://github.com/icecube45/G37_Builder/suites/4934864438/artifacts/142769129) without any other modifications. No change observed (b03111 and a03111 continued to work, whereas c03112 did not). Also tried over_voltage on c03112 and b03114 with no change (did not work)

[icecube45]

Managed to source another pi (b03112) locally and tested an sd card swap from a pi that was working (with the same, unmodified image as linked above), and the issue did not appear.

[icecube45]

Continuing to gather whatever information I can that may be relevant. While I can't swap an sd card from a working Pi to one that isn't, here are three sdcard speed tests from the agnostics package, one from a working pi, two from non working pis. rpdiags_gstreamer_bad_sdcard_pass_b03114.txt rpdiags_gstreamer_bad_sdcard_pass_c03112.txt rpdiags_gstreamer_ok_sdcard_fail_b03111.txt

[icecube45]

Ok! I did a lot more digging into this and think I found not only the actual core of this issue, but also a proper means of replicating it. I will edit the issue title to properly reflect this.

It's not a matter of revision (as @JamesH65 properly guessed), nor is it an SD card issue (as @popcornmix suggested testing - I was able to swap cards on "bad" and "good" pis finally). Instead, it's a h264 bytestream issue.. kinda?

The attached .h264 file replicates this issue on all pi revisions I've tested on so far, by running gst-launch-1.0 filesrc location=testfile.h264 ! h264parse ! v4l2h264dec ! video/x-raw,format=I420 ! autovideosink

On bullseye: This command and testfile always fails with "Internal data stream error", regardless of firmware version

On buster: This command fails with "No valid frames decoded before end of stream" when on latest firmware, after a sudo rpi-update 15471b6e99b6db2f3206f0aed67d0aebbfc9f9cf (the commit before the v4l2 stateful update), this command works as expected and the video is shown.

testfile.zip

[icecube45]

I've done a bit of further debugging here: It looks like on buster the error is thrown if the color_description matrix_coefficients is set (non zero), and works when set to zero (confirmed by hex editing the above testfile). And on bullseye the error is thrown if the color_description data is in the header at all (present_flag + the color primaries, transfer characteristics, matrix_coefficients) - confirmed again by hex editing.

At this point I am fully outside of my knowledge, but hoping my novice debugging of h264 can help out in some way.

[6by9]

The colour description fields translates to the V4L2 colourspaces - https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/colorspaces-defs.html The codec will decode these and tell the client if they change, in the same way that it will tell the client if the resolution changes.

Trying on Bullseye, (GStreamer 1.18.4), the kernel logging from V4L2 shows a load of attempts at TRY_FMT on the encoded stream which doesn't inherently have a colourspace), but no format being set or buffers passed in.

[ 1078.558072] videodev: v4l2_open: video10: open (0)
[ 1078.558090] video10: VIDIOC_QUERYCAP: driver=bcm2835-codec, card=bcm2835-codec-decode, bus=platform:bcm2835-codec, version=0x00050f12, capabilities=0x84204000, device_caps=0x04204000
[ 1078.558207] video10: VIDIOC_ENUM_FMT: error -22: index=0, type=vid-out, flags=0x0, pixelformat=.... little-endian (0x00000000), mbus_code=0x0000, description=''
[ 1078.558269] video10: VIDIOC_ENUM_FMT: index=0, type=vid-out-mplane, flags=0x1, pixelformat=MPG4 little-endian (0x3447504d), mbus_code=0x0000, description='MPEG-4 Part 2 ES'
[ 1078.558431] video10: VIDIOC_ENUM_FMT: index=1, type=vid-out-mplane, flags=0x1, pixelformat=H264 little-endian (0x34363248), mbus_code=0x0000, description='H.264'
[ 1078.558579] video10: VIDIOC_ENUM_FMT: index=2, type=vid-out-mplane, flags=0x1, pixelformat=MJPG little-endian (0x47504a4d), mbus_code=0x0000, description='Motion-JPEG'
[ 1078.558720] video10: VIDIOC_ENUM_FMT: index=3, type=vid-out-mplane, flags=0x1, pixelformat=VC1G little-endian (0x47314356), mbus_code=0x0000, description='VC-1 (SMPTE 412M Annex G)'
[ 1078.558863] video10: VIDIOC_ENUM_FMT: index=4, type=vid-out-mplane, flags=0x1, pixelformat=H263 little-endian (0x33363248), mbus_code=0x0000, description='H.263'
[ 1078.559001] video10: VIDIOC_ENUM_FMT: index=5, type=vid-out-mplane, flags=0x1, pixelformat=MPG2 little-endian (0x3247504d), mbus_code=0x0000, description='MPEG-2 ES'
[ 1078.559137] video10: VIDIOC_ENUM_FMT: error -22: index=6, type=vid-out-mplane, flags=0x0, pixelformat=.... little-endian (0x00000000), mbus_code=0x0000, description=''
[ 1078.559228] video10: VIDIOC_ENUM_FMT: error -22: index=0, type=vid-cap, flags=0x0, pixelformat=.... little-endian (0x00000000), mbus_code=0x0000, description=''
[ 1078.559286] video10: VIDIOC_ENUM_FMT: index=0, type=vid-cap-mplane, flags=0x0, pixelformat=YU12 little-endian (0x32315559), mbus_code=0x0000, description='Planar YUV 4:2:0'
[ 1078.559474] video10: VIDIOC_ENUM_FMT: index=1, type=vid-cap-mplane, flags=0x0, pixelformat=YV12 little-endian (0x32315659), mbus_code=0x0000, description='Planar YVU 4:2:0'
[ 1078.559620] video10: VIDIOC_ENUM_FMT: index=2, type=vid-cap-mplane, flags=0x0, pixelformat=NV12 little-endian (0x3231564e), mbus_code=0x0000, description='Y/CbCr 4:2:0'
[ 1078.559762] video10: VIDIOC_ENUM_FMT: index=3, type=vid-cap-mplane, flags=0x0, pixelformat=NV21 little-endian (0x3132564e), mbus_code=0x0000, description='Y/CrCb 4:2:0'
[ 1078.559903] video10: VIDIOC_ENUM_FMT: index=4, type=vid-cap-mplane, flags=0x0, pixelformat=RGBP little-endian (0x50424752), mbus_code=0x0000, description='16-bit RGB 5-6-5'
[ 1078.560046] video10: VIDIOC_ENUM_FMT: error -22: index=5, type=vid-cap-mplane, flags=0x0, pixelformat=.... little-endian (0x00000000), mbus_code=0x0000, description=''
[ 1078.563134] video10: VIDIOC_QUERYCTRL: error -22: id=0x990a95, type=0, name=, min/max=0/0, step=0, default=0, flags=0x00000000
[ 1078.563170] video10: VIDIOC_QUERYCTRL: error -22: id=0x990a96, type=0, name=, min/max=0/0, step=0, default=0, flags=0x00000000
[ 1078.563216] video10: VIDIOC_QUERYCTRL: error -22: id=0x990a0e, type=0, name=, min/max=0/0, step=0, default=0, flags=0x00000000
[ 1078.563246] video10: VIDIOC_QUERYCTRL: error -22: id=0x990a0f, type=0, name=, min/max=0/0, step=0, default=0, flags=0x00000000
[ 1078.563301] video10: VIDIOC_QUERYCTRL: error -22: id=0x990a67, type=0, name=, min/max=0/0, step=0, default=0, flags=0x00000000
[ 1078.563331] video10: VIDIOC_QUERYCTRL: error -22: id=0x990a6b, type=0, name=, min/max=0/0, step=0, default=0, flags=0x00000000
[ 1078.563733] videodev: v4l2_release: video10: release
[ 1079.649495] videodev: v4l2_open: video10: open (0)
[ 1079.649567] video10: VIDIOC_QUERYCAP: driver=bcm2835-codec, card=bcm2835-codec-decode, bus=platform:bcm2835-codec, version=0x00050f12, capabilities=0x84204000, device_caps=0x04204000
[ 1079.649891] video10: VIDIOC_ENUMINPUT: error -25: index=0, name=, type=0, audioset=0x0, tuner=0, std=0x00000000, status=0x0, capabilities=0x0
[ 1079.649975] video10: VIDIOC_ENUMSTD: error -25: index=0, id=0x0, name=, fps=0/0, framelines=0
[ 1079.650090] video10: VIDIOC_QUERYCTRL: id=0x980001, type=6, name=User Controls, min/max=0/0, step=0, default=0, flags=0x00000044
[ 1079.650202] video10: VIDIOC_QUERYCTRL: id=0x980927, type=1, name=Min Number of Capture Buffers, min/max=1/1, step=1, default=1, flags=0x00000004
[ 1079.650289] video10: VIDIOC_QUERYCTRL: error -22: id=0x80980927, type=0, name=, min/max=0/0, step=0, default=0, flags=0x00000000
[ 1079.650720] video10: VIDIOC_ENUM_FMT: index=0, type=vid-out-mplane, flags=0x1, pixelformat=MPG4 little-endian (0x3447504d), mbus_code=0x0000, description='MPEG-4 Part 2 ES'
[ 1079.651005] video10: VIDIOC_ENUM_FMT: index=1, type=vid-out-mplane, flags=0x1, pixelformat=H264 little-endian (0x34363248), mbus_code=0x0000, description='H.264'
[ 1079.651287] video10: VIDIOC_ENUM_FMT: index=2, type=vid-out-mplane, flags=0x1, pixelformat=MJPG little-endian (0x47504a4d), mbus_code=0x0000, description='Motion-JPEG'
[ 1079.651508] video10: VIDIOC_ENUM_FMT: index=3, type=vid-out-mplane, flags=0x1, pixelformat=VC1G little-endian (0x47314356), mbus_code=0x0000, description='VC-1 (SMPTE 412M Annex G)'
[ 1079.651656] video10: VIDIOC_ENUM_FMT: index=4, type=vid-out-mplane, flags=0x1, pixelformat=H263 little-endian (0x33363248), mbus_code=0x0000, description='H.263'
[ 1079.651800] video10: VIDIOC_ENUM_FMT: index=5, type=vid-out-mplane, flags=0x1, pixelformat=MPG2 little-endian (0x3247504d), mbus_code=0x0000, description='MPEG-2 ES'
[ 1079.651945] video10: VIDIOC_ENUM_FMT: error -22: index=6, type=vid-out-mplane, flags=0x0, pixelformat=.... little-endian (0x00000000), mbus_code=0x0000, description=''
[ 1079.652221] video10: VIDIOC_ENUM_FRAMESIZES: index=0, pixelformat=H264 little-endian (0x34363248), type=3, min=32x32, max=1920x1920, step=2x2
[ 1079.652437] video10: VIDIOC_ENUM_FRAMEINTERVALS: error -25: index=0, pixelformat=H264 little-endian (0x34363248), wxh=1920x1920, type=0
[ 1079.652513] video10: VIDIOC_TRY_FMT: type=vid-out-mplane, width=1920, height=1920, format=H264 little-endian (0x34363248), field=none, colorspace=3, num_planes=1, flags=0x0, ycbcr_enc=0, quantization=0, xfer_func=0
[ 1079.652545] plane 0: bytesperline=0 sizeimage=786432
[ 1079.652561] video10: VIDIOC_TRY_FMT: type=vid-out-mplane, width=1920, height=1920, format=H264 little-endian (0x34363248), field=none, colorspace=1, num_planes=1, flags=0x0, ycbcr_enc=0, quantization=0, xfer_func=0
[ 1079.652590] plane 0: bytesperline=0 sizeimage=786432
[ 1079.652603] video10: VIDIOC_TRY_FMT: type=vid-out-mplane, width=1920, height=1920, format=H264 little-endian (0x34363248), field=none, colorspace=2, num_planes=1, flags=0x0, ycbcr_enc=0, quantization=0, xfer_func=0
[ 1079.652632] plane 0: bytesperline=0 sizeimage=786432
[ 1079.652643] video10: VIDIOC_TRY_FMT: type=vid-out-mplane, width=1920, height=1920, format=H264 little-endian (0x34363248), field=none, colorspace=3, num_planes=1, flags=0x0, ycbcr_enc=0, quantization=0, xfer_func=0
[ 1079.652672] plane 0: bytesperline=0 sizeimage=786432
[ 1079.652684] video10: VIDIOC_TRY_FMT: type=vid-out-mplane, width=1920, height=1920, format=H264 little-endian (0x34363248), field=none, colorspace=5, num_planes=1, flags=0x0, ycbcr_enc=0, quantization=0, xfer_func=0
[ 1079.652712] plane 0: bytesperline=0 sizeimage=786432
[ 1079.652727] video10: VIDIOC_TRY_FMT: type=vid-out-mplane, width=1920, height=1920, format=H264 little-endian (0x34363248), field=none, colorspace=6, num_planes=1, flags=0x0, ycbcr_enc=0, quantization=0, xfer_func=0
[ 1079.652755] plane 0: bytesperline=0 sizeimage=786432
[ 1079.652769] video10: VIDIOC_TRY_FMT: type=vid-out-mplane, width=1920, height=1920, format=H264 little-endian (0x34363248), field=none, colorspace=7, num_planes=1, flags=0x0, ycbcr_enc=0, quantization=0, xfer_func=0
[ 1079.652797] plane 0: bytesperline=0 sizeimage=786432
[ 1079.652810] video10: VIDIOC_TRY_FMT: type=vid-out-mplane, width=1920, height=1920, format=H264 little-endian (0x34363248), field=none, colorspace=8, num_planes=1, flags=0x0, ycbcr_enc=0, quantization=0, xfer_func=0
[ 1079.652839] plane 0: bytesperline=0 sizeimage=786432
[ 1079.652852] video10: VIDIOC_TRY_FMT: type=vid-out-mplane, width=1920, height=1920, format=H264 little-endian (0x34363248), field=none, colorspace=9, num_planes=1, flags=0x0, ycbcr_enc=0, quantization=0, xfer_func=0
[ 1079.652880] plane 0: bytesperline=0 sizeimage=786432
[ 1079.652893] video10: VIDIOC_TRY_FMT: type=vid-out-mplane, width=1920, height=1920, format=H264 little-endian (0x34363248), field=none, colorspace=10, num_planes=1, flags=0x0, ycbcr_enc=0, quantization=0, xfer_func=0
[ 1079.652922] plane 0: bytesperline=0 sizeimage=786432
[ 1079.652936] video10: VIDIOC_TRY_FMT: type=vid-out-mplane, width=1920, height=1920, format=H264 little-endian (0x34363248), field=none, colorspace=11, num_planes=1, flags=0x0, ycbcr_enc=0, quantization=0, xfer_func=0
[ 1079.652964] plane 0: bytesperline=0 sizeimage=786432
[ 1079.673188] videodev: v4l2_release: video10: release

The GStreamer logging shows:

0:00:01.185869542   845   0x5bc450 DEBUG           v4l2videodec gstv4l2videodec.c:916:gst_v4l2_video_dec_sink_getcaps:<v4l2h264dec0> Returning sink caps video/x-h264, width=(int)[ 32, 1920, 2 ], height=(int)[ 32, 1920, 2 ], framerate=(fraction)[ 0/1, 2147483647/1 ], stream-format=(string)byte-stream, alignment=(string)au, colorimetry=(string){ bt709, bt601, smpte240m, 2:4:5:2, 2:4:5:3, 1:4:7:1, 2:4:7:1, 2:4:12:8, bt2020, 2:0:0:0 }, parsed=(boolean)true
0:00:01.186374624   845   0x5bc450 DEBUG           v4l2videodec gstv4l2videodec.c:916:gst_v4l2_video_dec_sink_getcaps:<v4l2h264dec0> Returning sink caps video/x-h264, width=(int)[ 32, 1920, 2 ], height=(int)[ 32, 1920, 2 ], framerate=(fraction)[ 0/1, 2147483647/1 ], stream-format=(string)byte-stream, alignment=(string)au, colorimetry=(string){ bt709, bt601, smpte240m, 2:4:5:2, 2:4:5:3, 1:4:7:1, 2:4:7:1, 2:4:12:8, bt2020, 2:0:0:0 }, parsed=(boolean)true
0:00:01.186637710   845   0x5bc450 DEBUG           v4l2videodec gstv4l2videodec.c:916:gst_v4l2_video_dec_sink_getcaps:<v4l2h264dec0> Returning sink caps EMPTY
0:00:01.186955957   845   0x5bc450 DEBUG           v4l2videodec gstv4l2videodec.c:916:gst_v4l2_video_dec_sink_getcaps:<v4l2h264dec0> Returning sink caps EMPTY
0:00:01.187293851   845   0x5bc450 DEBUG           v4l2videodec gstv4l2videodec.c:916:gst_v4l2_video_dec_sink_getcaps:<v4l2h264dec0> Returning sink caps EMPTY
0:00:01.187496980   845   0x5bc450 DEBUG           v4l2videodec gstv4l2videodec.c:916:gst_v4l2_video_dec_sink_getcaps:<v4l2h264dec0> Returning sink caps EMPTY
ERROR: from element /GstPipeline:pipeline0/GstH264Parse:h264parse0: Internal data stream error.
Additional debug info:

so I suspect it to be a colourspace mapping thing, and h264parse0 ends up saying that it can't produce something that meets the caps advertised by h264v4l2dec

Enabling more logging with GST_DEBUG=caps*:7 before gst-launch-1.0, I get

0:00:00.161445531   949   0x540450 TRACE          caps-features gstcapsfeatures.c:191:gst_caps_features_new_empty: created caps features 0xb4916f60
0:00:00.161521955   949   0x540450 DEBUG             capsfilter gstcapsfilter.c:298:gst_capsfilter_transform_caps:<capsfilter1> input:     video/x-raw, width=(int)800, height=(int)480, framerate=(fraction)0/1, colorimetry=(string)2:4:16:3
0:00:00.161573102   949   0x540450 DEBUG             capsfilter gstcapsfilter.c:299:gst_capsfilter_transform_caps:<capsfilter1> filter:    (NULL)
0:00:00.161625545   949   0x540450 DEBUG             capsfilter gstcapsfilter.c:300:gst_capsfilter_transform_caps:<capsfilter1> caps filter:    video/x-raw, format=(string)I420
0:00:00.161685284   949   0x540450 DEBUG             capsfilter gstcapsfilter.c:302:gst_capsfilter_transform_caps:<capsfilter1> intersect: video/x-raw, format=(string)I420, width=(int)800, height=(int)480, framerate=(fraction)0/1, colorimetry=(string)2:4:16:3
0:00:00.161767819   949   0x540450 DEBUG             capsfilter gstcapsfilter.c:298:gst_capsfilter_transform_caps:<capsfilter1> input:     video/x-raw, format=(string)I420, width=(int)800, height=(int)480, framerate=(fraction)0/1, colorimetry=(string)2:4:16:3
0:00:00.161827744   949   0x540450 DEBUG             capsfilter gstcapsfilter.c:299:gst_capsfilter_transform_caps:<capsfilter1> filter:    video/x-raw, width=(int)800, height=(int)480, framerate=(fraction)0/1, colorimetry=(string)2:4:16:3
0:00:00.161886650   949   0x540450 DEBUG             capsfilter gstcapsfilter.c:300:gst_capsfilter_transform_caps:<capsfilter1> caps filter:    video/x-raw, width=(int)800, height=(int)480, framerate=(fraction)0/1, colorimetry=(string)2:4:16:3, format=(string)I420
0:00:00.161945907   949   0x540450 DEBUG             capsfilter gstcapsfilter.c:302:gst_capsfilter_transform_caps:<capsfilter1> intersect: video/x-raw, width=(int)800, height=(int)480, framerate=(fraction)0/1, colorimetry=(string)2:4:16:3, format=(string)I420

so I read that as h264parse viewing the stream as colorimetry=(string)2:4:16:3 (need to find references to decode that).

ffprobe decodes your clip as " bt470bg/bt470bg/smpte170m".

[icecube45]

@6by9, thank you for looking into this. From everything you've detailed, I agree that it looks like a colorspace issue. Similarly, I think there might be two different things going on: 1) Gstreamer not linking on buster due to the caps advertised by h264v4l2dec 2) Potentially something more firmware based (the color space not being supported?)

Bear with me here as I try and explain why I think this is the case - I might be misinterpreting things.

I've repeated your logging on Buster, with fully updated firmware, and downgraded firmware below the commit that this issue first started appearing. I've replaced the gstreamer logging for a dot file export. The dmesg and resultant pipeline can both be found in the files at the end of this issue.

In both cases, it looks as if v4l2 is setting the same format (I see two VIDIOC_S_FMT statements in each log, one for H264, one for YU12). Similarly, the gstreamer pipelines look identical, with no colorspace cap on h264parse, and v4l2h264dec spitting out bt601.

These tests are what made me feel like something more underlying firmware based is occurring.

For the case of bullseye, it does look as if gstreamer is failing to link, for the reasons you mentioned (colorspace caps). That is to say, I can replicate your results. When the video_signal info is removed from the VUI in the testfile, and the same pipeline tested, again we see the same results as buster: no colorimetry from h264parse, bt601 from v4l2h264dec, S_FMT for h264 & YU12.

Similarly, I can get the gstreamer pipeline to link if I lie to it using capssetter: gst-launch-1.0 filesrc location=testfile.h264 ! h264parse ! capssetter replace=true caps="video/x-h264, width=(int)800, height=(int)480, framerate=(fraction)0/1, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)baseline, level=(string)3.1" ! v4l2h264dec ! video/x-raw,format=I420 ! autovideosink Similarly, I see the two S_FMT lines, though the pipeline still errors with ERROR: from element /GstPipeline:pipeline0/v4l2h264dec:v4l2h264dec0: No valid frames decoded before end of stream (the error case from an updated buster install)

And because I was curious, I downgraded the firmware and combined it with the capssetter linking - no difference. bullseye_updated_firmware_capssetter_dmesg.txt bullseye_updated_firmware_video_signal_removed_dmesg.txt buster_firmware_up_to_date_dmesg.txt buster_old_firmware_dmsg.txt

Buster, firmware up to date

Buster, old firmware

Bullseye, updated firmware, video signal removed from testfile.

Bullseye, updated firmware, capssetter

[6by9]

So decoding that colorimetry=(string)2:4:16:3 https://github.com/GStreamer/gst-plugins-base/blob/master/gst-libs/gst/video/video-color.c#L149

    return g_strdup_printf ("%d:%d:%d:%d", cinfo->range, cinfo->matrix,
        cinfo->transfer, cinfo->primaries);

range = 1 -> GST_VIDEO_COLOR_RANGE_0_255 matrix = 4 -> GST_VIDEO_COLOR_MATRIX_BT601 transfer = 16 -> GST_VIDEO_TRANSFER_BT601 primaries = 3 -> GST_VIDEO_COLOR_PRIMARIES_BT470BG

So it is correct that that your stream doesn't match any of the "standard" colorimetries .

However there is the question as to whether h264parse should really be caring about the colorimetry of the encoded stream, or v4l2h264dec should care on the sink pad for the encoded data. Unfortunately I don't think v4l2h264dec gets a choice about it, as it just advertises the list of things it'll support in the caps, and the src pad then matches it.

Otherwise the comment at https://github.com/GStreamer/gst-plugins-good/blob/master/sys/v4l2/gstv4l2object.c#L2394

   /* step 2: probe all colorspace other than default
   * We don't probe all colorspace, range, matrix and transfer combination to
   * avoid ioctl flooding which could greatly increase initialization time
   * with low-speed devices (UVC...) */

is an invalid choice in v4l2h264dec.

[6by9]

Having enabled the V4L2 and videobuf2 logging, it looks like GStreamer doesn't implement the dynamic resolution change handling required by the V4L2 Stateful Decoder Interface.

After parsing the H264 headers the decoder has noted a change in the colourspace, and so triggered a resolution/format change event.

1. After encountering a resolution change in the stream, the decoder sends a V4L2_EVENT_SOURCE_CHANGE event with changes set to V4L2_EVENT_SRC_CH_RESOLUTION.

GStreamer hasn't subscribed to these events (VIDIOC_SUBSCRIBE_EVENT(V4L2_EVENT_SOURCE_CHANGE)), so it doesn't see it.

The decoder will then process and decode all remaining buffers from before the resolution change point.

The last buffer from before the change must be marked with the V4L2_BUF_FLAG_LAST flag, similarly to the Drain sequence above.

Warning: The last buffer may be empty (with v4l2_buffer bytesused = 0) and in that case it must be ignored by the client, as it does not contain a decoded frame.

Note: Any attempt to dequeue more CAPTURE buffers beyond the buffer marked with V4L2_BUF_FLAG_LAST will result in a -EPIPE error from VIDIOC_DQBUF().

On the next DQBUF the decoder does generate a -EPIPE

[  635.646035] videodev: v4l2_poll: video10: poll: 00000041
[  635.646429] videobuf2_common: [cap-71bd83c8] __vb2_wait_for_done_vb: last buffer dequeued already, will not wait for buffers
[  635.646449] video10: VIDIOC_DQBUF: error -32: 00:00:00.000000000 index=0, type=vid-cap-mplane, request_fd=0, flags=0x00000000, field=any, sequence=0, memory=mmap

The client must continue the sequence as described below to continue the decoding process.

The client doesn't (it misinterprets the -EPIPE as only applying to EOS documented under DRAIN), and therefore decode stops.

gstv4l2src calls gst_v4l2_buffer_pool_enable_resolution_change to subscribe to the event, and gst_v4l2_buffer_pool_poll will then respond with GST_V4L2_FLOW_RESOLUTION_CHANGE, but gstvideodec doesn't.

Ah https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/-/merge_requests/240 Except I'm confused as https://bugzilla.gnome.org/show_bug.cgi?id=752962 is resolved, and I could have sworn that resolution change had worked. Perhaps it was just detecting the formats outside of V4L2, and setting the port formats correctly.

[icecube45]

range = 1 -> GST_VIDEO_COLOR_RANGE_0_255 matrix = 4 -> GST_VIDEO_COLOR_MATRIX_BT601 transfer = 16 -> GST_VIDEO_TRANSFER_BT601 primaries = 3 -> GST_VIDEO_COLOR_PRIMARIES_BT470BG

Double checking this (though I don't think it matters in the wider picture of what's going on) isn't the colorspace being reported range =2 -> GST_VIDEO_COLOR_RANGE_16_235 ?

Sounds like an update to gstreamer (or well, the plugins) is needed to correct this, then?

[6by9]

Double checking this (though I don't think it matters in the wider picture of what's going on) isn't the colorspace being reported range =2 -> GST_VIDEO_COLOR_RANGE_16_235 ?

Yes, typo by me. It's 16_235.

Sounds like an update to gstreamer (or well, the plugins) is needed to correct this, then?

AFAICT this is GStreamer not following the spec, so it would be an update required to the plugin.

[icecube45]

I went ahead and recompiled gst-plugins-good with the PR you linked above - the capssetter pipeline works in that case, so seems as though you're correct in the issue at play here. Hopefully that gets merged soon 🤞. Any chance of a kernel flag to work around this until it does? 😅

[6by9]

Seeing as that MR was started 2 years ago, the v4l2src rework was 11 months ago, I suspect it's stalled at present. I'm amazed it'll still apply and compile after all that time.

You may find that changing https://github.com/raspberrypi/linux/blob/rpi-5.15.y/drivers/staging/vc04_services/bcm2835-codec/bcm2835-v4l2-codec.c#L2533 so MMAL_PARAMETER_VIDEO_STOP_ON_PAR_COLOUR_CHANGE is set to 0 instead of 1 will workaround the issue - that could be done as a module parameter. We can't just ignore the parameter at the kernel level as the firmware will already have stopped when it saw the change.

[icecube45]

😅 For what it's worth I actually tried that exact thing a few days back. Tested it again just now - sad to say it doesn't workaround the issue.

[6by9]

You also need to stop it setting the last_buffer_dequeued flag when it receives a MMAL_EVENT_FORMAT_CHANGED event.

index 472d97d1d228..cc6a096d8647 100644
--- a/drivers/staging/vc04_services/bcm2835-codec/bcm2835-v4l2-codec.c
+++ b/drivers/staging/vc04_services/bcm2835-codec/bcm2835-v4l2-codec.c
@@ -1077,9 +1077,9 @@ static void handle_fmt_changed(struct bcm2835_codec_ctx *ctx,
                q_data->field = V4L2_FIELD_NONE;
        }

-       vq = v4l2_m2m_get_vq(ctx->fh.m2m_ctx, V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE);
-       if (vq->streaming)
-               vq->last_buffer_dequeued = true;
+       //vq = v4l2_m2m_get_vq(ctx->fh.m2m_ctx, V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE);
+       //if (vq->streaming)
+       //      vq->last_buffer_dequeued = true;

        queue_res_chg_event(ctx);
 }
@@ -2530,7 +2530,7 @@ static int bcm2835_codec_create_component(struct bcm2835_codec_ctx *ctx)
                 * Enable firmware option to stop on colourspace and pixel
                 * aspect ratio changed
                 */
-               enable = 1;
+               enable = 0;
                vchiq_mmal_port_parameter_set(dev->instance,
                                              &ctx->component->control,
                                              MMAL_PARAMETER_VIDEO_STOP_ON_PAR_COLOUR_CHANGE,

All of these command lines have then worked

pi@raspberrypi:~/tmp $ GST_DEBUG=caps*:7,video*:7,v4l2*:7 gst-launch-1.0 -vvv filesrc location=testfile.h264 ! h264parse ! capssetter caps="video/x-h264, colorimetry=(string)bt601" ! v4l2h264dec ! "video/x-raw,format=I420,colorimetry=(string)bt601" ! filesink location=testfile.i420
pi@raspberrypi:~/tmp $ GST_DEBUG=caps*:7,video*:7,v4l2*:7 gst-launch-1.0 -vvv filesrc location=testfile.h264 ! h264parse ! capssetter caps="video/x-h264, colorimetry=(string)bt601" ! v4l2h264dec ! "video/x-raw,format=NV12,colorimetry=(string)bt601" ! filesink location=testfile.nv12
pi@raspberrypi:~/tmp $ GST_DEBUG=caps*:7,video*:7,v4l2*:7 gst-launch-1.0 -vvv filesrc location=testfile.h264 ! h264parse ! capssetter caps="video/x-h264, colorimetry=(string)bt601" ! v4l2h264dec ! "video/x-raw,format=NV12,colorimetry=(string)bt601" ! kmssink

[icecube45]

Thank you - confirming I can replicate your results by applying the same patch

[ndufresne]

Does NXP MR for resolution changes works with your driver? I've been waiting all this time for folks to actually test this, but I see that workaround in drivers / firmware is more tempting. Note that the submitter has be very slow fixing review comments, but I can take over if some tested-by comment showed up.

[ndufresne]

For an update, I realized you are looking at an old MR, the current implementation is here: https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1381

[icecube45]

@ndufresne once I get some free time (rare for these next 3 weeks), I'll give a go at compiling the MR linked, and testing the above pipeline with it

[6by9]

I'm building GStreamer master now to have a quick look at how https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1381 interacts.

[6by9]

@ndufresne https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1381 looks to solve the issue of v4l2h264dec aborting due to a source changed event.

For @icecube45 's testfile.zip in https://github.com/raspberrypi/firmware/issues/1673#issuecomment-1023866682 I do still need to use capssetter to override the caps from h264parse. h264parse has determined the colorimetry as colorimetry=(string)2:4:16:3, and v4l2h264dec doesn't include that in the caps it advertises, therefore negotiation fails. Seeing as I don't believe V4L2 can express all the colorimetry values that GStreamer lists, and it doesn't matter what the colorimetry value is within the compressed bitstream, is there a way to avoid it validating that?

Playing back Big Buck Bunny 1080p I do get a couple of v4l2h264dec0: Too old frames, bug in decoder -- please file a bug errors logged. Any guidance on the conditions that this gets reported? Is it just that the decoder took too long on a frame, or is it timestamp based?

[fduncanh]

I have been trying to get UxPlay to work on Raspberry Pi 4B 8GB (Raspberry Pi OS 64bit, Bullseye) with v4l2h264dec for GPU h264 decoding. Yesterday It finally worked great, but later was no longer working again (on a different source). Gets a problem with caps ... not accepted, then "Internal data stream error". I think it's the same problem being discussed here, because the testfile.h264 given here (earlier) by @icecube45 produced the similar failure he described.

gst-launch-1.0 filesrc location=testfile.h264 ! h264parse ! v4l2h264dec ! video/x-raw,format=I420 ! autovideosink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
ERROR: from element /GstPipeline:pipeline0/GstH264Parse:h264parse0: Internal data stream error.
Additional debug info:
../libs/gst/base/gstbaseparse.c(3676): gst_base_parse_loop (): /GstPipeline:pipeline0/GstH264Parse:h264parse0:
streaming stopped, reason not-negotiated (-4)
ERROR: pipeline doesn't want to preroll.
Setting pipeline to NULL ...
Freeing pipeline ...

My video pipeline is a h264 stream (Apple iOS Airplay)

appsrc name=video_source stream-type=0 format=GST_FORMAT_TIME is-live=true ! queue ! h264parse ! v4l2h264dec ! v4l2convert ! glimagesink name=video_sink sync=false

works (with poor latency) if v4l2h264dec ! v4l2convert is replaced by avdec_h264 ! videoconvert.

Since I have seen this v4l2 pipeline working beautifully on the RPi4 with a source I cannot find again, I am hoping for some workaround in the pipeline, I am not a gstreamer expert (far from it).

My own error, with GST_DEBUG=2 is

(Begin streaming to GStreamer video pipeline)

0:01:23.953059015  3340   0x559ae8acc0 WARN                GST_CAPS gstpad.c:5701:pre_eventfunc_check:<v4l2h264dec0:sink> caps video/x-h264, width=(int)1440, height=(int)1080, framerate=(fraction)0/1, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, colorimetry=(string)1:3:5:1, parsed=(boolean)true, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)high-10, level=(string)4.2 not accepted
0:01:23.953450030  3340   0x559ae8acc0 WARN                GST_CAPS gstpad.c:5701:pre_eventfunc_check:<v4l2h264dec0:sink> caps video/x-h264, width=(int)1440, height=(int)1080, framerate=(fraction)0/1, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, colorimetry=(string)1:3:5:1, parsed=(boolean)true, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)high-10, level=(string)4.2 not accepted
0:01:23.953715312  3340   0x559ae8acc0 WARN                GST_CAPS gstpad.c:5701:pre_eventfunc_check:<v4l2h264dec0:sink> caps video/x-h264, width=(int)1440, height=(int)1080, framerate=(fraction)0/1, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, colorimetry=(string)1:3:5:1, parsed=(boolean)true, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)high-10, level=(string)4.2 not accepted
0:01:23.953854563  3340   0x559ae8acc0 WARN                GST_CAPS gstpad.c:5701:pre_eventfunc_check:<v4l2h264dec0:sink> caps video/x-h264, width=(int)1440, height=(int)1080, framerate=(fraction)0/1, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, colorimetry=(string)1:3:5:1, parsed=(boolean)true, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)high-10, level=(string)4.2 not accepted
0:01:23.973245016  3340   0x559ae87360 WARN                 basesrc gstbasesrc.c:3127:gst_base_src_loop:<video_source> error: Internal data stream error.
0:01:23.973305161  3340   0x559ae87360 WARN                 basesrc gstbasesrc.c:3127:gst_base_src_loop:<video_source> error: streaming stopped, reason not-negotiated (-4)
0:01:23.973485688  3340   0x559ae87360 WARN                   queue gstqueue.c:990:gst_queue_handle_sink_event:<queue4> error: Internal data stream error.
0:01:23.973512130  3340   0x559ae87360 WARN                   queue gstqueue.c:990:gst_queue_handle_sink_event:<queue4> error: streaming stopped, reason not-negotiated (-4)

(GStreamer error: Internal data stream error.)

[ndufresne]

@ndufresne https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1381 looks to solve the issue of v4l2h264dec aborting due to a source changed event.

That is great, thanks for testing, as I now have two positive reports I'm confident this can land now.

For @icecube45 's testfile.zip in #1673 (comment) I do still need to use capssetter to override the caps from h264parse. h264parse has determined the colorimetry as colorimetry=(string)2:4:16:3, and v4l2h264dec doesn't include that in the caps it advertises, therefore negotiation fails. Seeing as I don't believe V4L2 can express all the colorimetry values that GStreamer lists, and it doesn't matter what the colorimetry value is within the compressed bitstream, is there a way to avoid it validating that?

We've been having issue since the day we introduce colorimetry support (its a nightmare). Code is shared for all type of devices unfortunately, we currently are running off a workaround to avoid 30+ second to load a UVC camera.. I think we need to wave that restriction for decoders, so that anything is allowed. The decoders rarely do CSC, so in practice, we can simply allow everything and copy the information on the output pad. I'll try and see what I can do in this cycle, meanwhile, there might be some quick patch we could do to simply remove colorimetry from caps, and just apply them to the driver as a best effort.

Playing back Big Buck Bunny 1080p I do get a couple of v4l2h264dec0: Too old frames, bug in decoder -- please file a bug errors logged. Any guidance on the conditions that this gets reported? Is it just that the decoder took too long on a frame, or is it timestamp based?

Just read the code to refresh my memory. The driver copy the bitstream timestamp into the picture buffer timestamp (capture buffer). GStreamer keeps track of all the frames, and use that timestamp to fetch the associated frame. This is needed so that we can transfer the metadata (like the presentation timestamp). So if that driver didn't produce a frame, we'd leak that information.

https://gitlab.freedesktop.org/gstreamer/gstreamer/-/blob/main/subprojects/gst-plugins-good/sys/v4l2/gstv4l2videodec.c#L554

That code, that emit a warning once, will garbage collect the frames on a pretty relax condition, basically if the frame counter of the frame being produced is 100 frames more recent then the older frame stored, the older frame metadata is dropped. This is quite a lot of frames before it triggers.

Now, about the driver, I would suspect a problem when copying over the timestamp, or that the driver drops silently frames that would otherwise be corrupted. Though, just reading the code here, I see that if the driver returns corrupted frames with a payload size less then sizeimage, we will drop that buffer without cleaning up the associated frame. For that context, there is a bit of an GStreamer abstraction bug, the decoder is never made aware of the buffer (its discarded early), so it does not have the timestamp to cleanup. Would be nice to check that up, a workaround would be to set the payload size even if the buffer is corrupted.

[ndufresne]

I've just moved over the associated related bug, it is pretty old, the author is CODA driver author, and was at that time usually fixing GStreamer in tandem, but for this one it seems like it never happened. I've attached your test file.

https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/1056

[6by9]

Indeed the decoder doesn't do CSC, so it is effectively passing on metadata. I'll let you worry about it within the GStreamer framework :-)

I can have a guess over the Too old frames, bug in decoder - the old "what's the maximum size of an encoded frame" question.

My BBB 1080p clip (I'd need to try and dig out the original source) has a video frame that is 574139 bytes in size at about the point the first error is logged, and another that is 564779 bytes around the time of the second error. (It's an ES, so no timestamps) The default buffer size we allocate on the decoder is 512kB, and V4L2 mandates that a single encoded frame goes in a single V4L2 buffer. What happens to the remainder of the buffer? AIUI Userspace could request larger buffers if it expected a higher bitrate and therefore bigger buffers, but the driver can't really make that call without just wasting memory. It doesn't know the level until it has the header bytes, by which time the OUTPUT queue is active with all the buffers allocated.

@jc-kynesim was looking to solve this by always using dmabufs allocated via a dma heap. Should he get an incoming buffer that exceeded the current buffer size then he could allocate a new larger buffer from the heap and put that into circulation to the V4L2 decoder. I don't recall if that actually worked for him or not.

[jc-kynesim]

That (dmabuf + realloc) works fine for me in the ffmpeg H265 stateless decode, I haven't rewritten the stateful decode to work that way (yet).

[ndufresne]

The default buffer size we allocate on the decoder is 512kB, and V4L2 mandates that a single encoded frame goes in a single V4L2 buffer. What happens to the remainder of the buffer?

Hans have suggested that we should pass the continuity into another buffer, as the best effort, but this was never implemented in GStreamer. Reading the code, gst just truncate it, and does not even warn about it:

https://gitlab.freedesktop.org/gstreamer/gstreamer/-/blob/main/subprojects/gst-plugins-good/sys/v4l2/gstv4l2bufferpool.c#L160

The could be interesting to rework for sure. I don't have time to add all this, remains a best effort, some HW will fail. I'm thinking we need some sort of re-alloc method on the V4L2 side of thing to really handle this correctly in the long run.

[ndufresne]

Most kernel dev so far have just made their ES max size guess produce a bigger number ... On Stateless size, they just return the picture size.

[ndufresne]

That (dmabuf + realloc) works fine for me in the ffmpeg H265 stateless decode, I haven't rewritten the stateful decode to work that way (yet).

Is that something you did downstream ? Or anything useful for upstream ? A link would be appreciated.

[6by9]

Hans have suggested that we should pass the continuity into another buffer, as the best effort, but this was never implemented in GStreamer. Reading the code, gst just truncate it, and does not even warn about it:

It's one of the few places OpenMAX was nice in supporting splitting a frame between multiple buffers. Our decoder is happy with that. However because the V4L2 spec says that each buffer will be a complete frame, we tell the decoder that, and it will spit out the decoded frame using the information it had.

However I may be wrong in this case. I've just artificially bumped the buffer size up to 1MB, and I still get the errors. I'll throw some extra logging in.

[jc-kynesim]

dmabuf + realloc - its downstream - not sure its appropriate for upstream - I got so bored of trying to "fix" the existing ffmpeg stateless code to do what I wanted/needed for our driver I replaced it wholesale with code I'd already mostly written as an attempt at a VAAPI shim (which worked but absent display allocation it didn't just drop into existing code the way I'd hoped so I went back to ffmpeg + drm_prime as actually easier). If you want to look then github https://github.com/jc-kynesim/rpi-ffmpeg.git branch test/4.3.2/rpi_1 main or dev/4.4/rpi_import_1 files libavcodec/v4l2_req*.

[ndufresne]

Ok, was asking as it came to our mind the "re-alloc" is a better approach then CREATE/DEL_BUFS to solve SVC spacial decoding issues (our main focus being VP9, but its broken for H264/HEVC, stateful or stateless). Getting a bit off topic though, but a mechanism that would serve here too for driver allocated bitstream buffers. The only alternative I could think of would be the use a DMABuf heap, but there is some gap filling to make this approach really generic.

[jc-kynesim]

Its not a particularly difficult bit of code - you just need a dmabuf pool and by and large everything works as long as you use the multiplane interface, the single plane one has broken/legacy length checking that I tried to get fixed but it turns out that some user stuff relies on the old behaviour. There is meant to be a new grand unified interface that fixes everything but I haven't looked at it yet to comment on that.

[ndufresne]

We now have a extrapolate_stride() helper in gst to compensate these legacy API, but I'm realist is just a matter of time before we need per HW quirks. I see indeed that you use DMAbuf HEAP, so it is indeed what I had in mind. the missing bit to make this generic, is an upstream API to let drivers inform userland which heap driver to use, most of the time it will be the generic cma one (not sure it behaves like the v4l2 one when we have an iommu ?), but if not, user-land should not hard code that per driver. And on DRM side, there is no equivalent to the per plane sizeimage there, so also not ready yet.

[bellum07]

I think I’m affected by the same issue.

I’m currently trying to run the following pipeline:

  gst-launch-1.0 --gst-debug-level=1 filesrc location=~/ARD-HD_H264_720p_MP2_2ch.ts ! tsdemux name=dmx \
      dmx. ! h264parse ! queue ! v4l2h264dec capture-io-mode=4 ! queue ! kmssink \
      dmx. ! mpegaudioparse ! queue ! mpg123audiodec ! queue ! alsasink

on a Raspberry Pi 2 with Raspi OS ‘2022-01-28-raspios-bullseye-armhf-lite.img’ and the latest packet updates.

When I’m only playing the video part using:

  gst-launch-1.0 --gst-debug-level=1 filesrc location=~/ARD-HD_H264_720p_MP2_2ch.ts ! tsdemux name=dmx \
      dmx. ! h264parse ! queue ! v4l2h264dec capture-io-mode=4 ! queue ! kmssink

I get the error: gst_video_decoder_sink_event_default:<v4l2h264dec0> error: No valid frames decoded before end of stream

Playing audio only works fine with:

  gst-launch-1.0 --gst-debug-level=1 filesrc location=~/ARD-HD_H264_720p_MP2_2ch.ts ! tsdemux name=dmx \
      dmx. ! mpegaudioparse ! queue ! mpg123audiodec ! queue ! alsasink

The complete pipeline plays without issues using Raspi OS ‘2021-05-07-raspios-buster-armhf-lite.img’ with a selfcompiled GStreamer 1.18.4 on the same Raspberry Pi 2.

I’ve read in this thread some posts above that this has been fixed but I’m not really sure if I’ve understood this correctly and how to get/apply the fix…

Any guidance would be greatly appreciated.

If you need more information about my environment or want me something to test, I’d be glad to do so.

[fduncanh]

With R Pi OS dropping omx, v4l2h264dec becomes the only gstreamer GPU hardware h264 decoder available for PI 4, so hopefully "somebody" will be motivated to give it any necessary fixes.

Is it naive to expect that one "ought" to be able to make a working pipeline "...! h264parse ! avdec_h264 ! videoconvert ! xvimagesink ..." (which has lousy latency on Pi) become hardware decoded with the simple change "... ! h264parse ! v4l2h264dec ! v4l2convert ! xvimagesink ..." (perhaps with some capture-io-mode=4 stuff thrown in).?

One issue I have is that the caps arrive with the SPS, PPS of the h264 stream, I dont set them. I see lots of colorimetry stuff in GST_DEBUG=5 output when using v4l2 on 64bit Pi, but haven't worked out what the caps mismatch I am getting is. Can one selectively alter single bad elements in the caps with capssetter?. The Pi is a marginal case for my app, vaapi and nvdec on X86_64 platforms work fine (but are not essential), but now refugees from former-omx land on the Pi are showing up looking for alternatives. The app (a GPLv3 AirPlay server) is useless on the Pi without GPU h264 decoding. Somehow, v4l2 did work a few times, and when it did, performance was great, no noticeable latency

Any advice or hints on how to proceed? (capssetter perhaps) (Or just give up and wait for fixes in v4l2h264dec)

[bellum07]

Today, I did a rpi-update hoping that updating to the latest kernel/firmware will help.

But unfortunately, with

root@raspberrypi:~# uname -a
Linux raspberrypi 5.15.25-v7+ #1527 SMP Mon Feb 28 16:37:22 GMT 2022 armv7l GNU/Linux

GStreamer still shows ERROR: from element /GstPipeline:pipeline0/v4l2h264dec:v4l2h264dec0: No valid frames decoded before end of stream

How to get this working? Can anybody help? Thanks in advance.

[fduncanh]

@icecube45 @6by9 your comments here are very useful.

I have also confirmed there are two distinct errors, and the CAPS negotiation failure can be fixed with a very minimal capssetter element:

This is now my pipeline (R Pi 4 , gst 1.20.0 manjaro linux (arch))

appsrc name=video_source stream-type=0 format=GST_FORMAT_TIME is-live=true ! queue ! h264parse !  capssetter replace=true caps="video/x-h264,  stream-format=(string)byte-stream, alignment=(string)au " ! v4l2h264dec  ! v4l2convert ! autovideosink name=video_sink sync=false

The correct caps supplied by the h264 PPS and SPS (width, height, chroma, colorimetry, etc) get through without needing to be specified by capssetter, which is very encouraging for the future. Maybe @icecube45 could check that he can also cut down the capssetter entry to its bare essentials.

I now get to the same failure as @icecube45

0:00:11.589447901 26565 0xaaaac5d61400 DEBUG               GST_CAPS gstpad.c:2709:gst_pad_has_current_caps:<h264parse0:src> check current pad caps video/x-h264, width=(int)1440, height=(int)1080, framerate=(fraction)0/1, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, colorimetry=(string)1:3:5:1, parsed=(boolean)true, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)high, level=(string)4.2
GStreamer error: No valid frames decoded before end of stream

Is this still understood to to be a firmware issue, or is it supposed to be fixed by the patch https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1381 ?

I have rebuilt gst 1.20.0 libgstvideo4linux2.so with that 1381 patch, and with it I get a different failure with ** (uxplay:26707): CRITICAL **: 19:01:09.336: gst_v4l2_transform_set_caps: assertion '!GST_V4L2_IS_ACTIVE (self->v4l2output)' failed

Should @6by9 's patch to bcm2835-v4l2-codec.c be applied next?

[6by9]

This is two issues, both within GStreamer. However I'll leave this issue open until we have a workable solution in Raspberry Pi OS.

The firmware/kernel drivers are following the V4L2 stateless decoder API spec. My patch was a hack to not follow the spec on some format changes. https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1381 fixes GStreamer to also follow the spec. I see that it is now merged to GStreamer's main branch, so it may be possible for a backport to Raspberry Pi OS's 1.18 repo.

The caps issue is separate and being tracked in https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/1056 Using capssetter works around it for now, and the GStreamer devs can come up with a plan for how to solve it properly.

[bellum07]

I see that it is now merged to GStreamer's main branch, so it may be possible for a backport to Raspberry Pi OS's 1.18 repo.

Is there already a timeframe available for this?

[6by9]

I see that it is now merged to GStreamer's main branch, so it may be possible for a backport to Raspberry Pi OS's 1.18 repo.

Is there already a timeframe available for this?

I've not even looked at how the trees have changed. There is a significant change in that all the repos have been merged into one repo (monorepo), but as it is only affecting gst-plugins-good I would expect that it can be separated out relatively easily. The question is whether there are any additional dependencies that it relies on.

[hailfinger]

It might be easier to backport that change to GStreamer 1.20 and ship a patched GStreamer 1.20 in Raspberry Pi OS as backport. During GStreamer 1.19 development, quite a few bug fixes in GStreamer affecting streaming on a Raspberry Pi were merged. I went forward from 1.18 and it has been a tremendous improvement.