Hardware h264 encoder fails to mux Matroska and separately fails Tee outputs in GStreamer and FFmpeg

dustinkerstein commented 2 years ago

Describe the bug

These may be two unrelated issues, but I have a hunch they may also be related to #1836 or #4734.

On my Compute Module 4 running 64bit Bullseye, I am unable to mux Matroska files when using the hardware h264 encoder. Further, I am also unable to use the tee functionality of both GStreamer and FFmpeg when using the hardware h264 encoder. Below are results from various tests:

Working - Hardware Encode

ffmpeg -y -f lavfi -i smptebars=duration=1:size=1920x1080:rate=30 -c:v h264_v4l2m2m out.h264

Possibly Benign Errors:
    Failed to set gop size: Invalid argument
    Encoder did not produce proper pts, making some up.
    Application provided invalid, non monotonically increasing dts to muxer in stream 0: 1 >= 1

ffmpeg -y -f lavfi -i smptebars=duration=1:size=1920x1080:rate=30 -c:v h264_v4l2m2m out.mp4
ffmpeg -y -f lavfi -i smptebars=duration=1:size=1920x1080:rate=30 -c:v h264_v4l2m2m out.flv
ffmpeg -y -f lavfi -i smptebars=duration=1:size=1920x1080:rate=30 -c:v h264_v4l2m2m out.ts

Possibly Benign Errors:
    Failed to set gop size: Invalid argument
    Timestamps are unset in a packet for stream 0. 
    This is deprecated and will stop working in the future. 
    Fix your code to set the timestamps properly
    Encoder did not produce proper pts, making some up.

Broken - Hardware Encode

ffmpeg -y -f lavfi -i smptebars=duration=3:size=1920x1080:rate=30 -c:v h264_v4l2m2m out.mkv

Possibly Benign Errors:
    Failed to set gop size: Invalid argument

Breaking Errors:
    Could not write header for output file #0 (incorrect codec parameters ?):
    Invalid data found when processing input
    Error initializing output stream 0:0 --

gst-launch-1.0 -v videotestsrc pattern=snow num-buffers=100 ! v4l2h264enc \
! h264parse ! matroskamux ! filesink location=out.mkv

Breaking Errors::
    ../sys/v4l2/gstv4l2videoenc.c(828): gst_v4l2_video_enc_handle_frame (): 
    /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0:
    Maybe be due to not enough memory or failing driver
    ERROR: pipeline doesn't want to preroll.
    Setting pipeline to NULL ...
    ERROR: from element /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0: 
    Internal data stream error.

Working - Software Encode

ffmpeg -y -f lavfi -i smptebars=duration=3:size=1920x1080:rate=30 -c:v libx264 out.mkv

gst-launch-1.0 -v videotestsrc pattern=snow num-buffers=100 ! x264enc \
! h264parse ! matroskamux ! filesink location=out.mkv

Working Tee - Software Encode

ffmpeg -y -f lavfi -i smptebars=duration=1:size=1920x1080:rate=30 \
-c:v libx264 -f tee -map 0:v "out1.mp4|out2.mp4"

gst-launch-1.0 -v videotestsrc pattern=snow num-buffers=100 \
! video/x-raw,width=640,height=480 ! videoconvert \
! x264enc ! h264parse ! mp4mux ! tee name=t \
t. ! filesink async=0 location=out1.mp4 \
t. ! filesink async=0 location=out2.mp4

Broken Tee - Hardware Encode

ffmpeg -y -f lavfi -i smptebars=duration=1:size=1920x1080:rate=30 \
-c:v h264_v4l2m2m -f tee -map 0:v "out1.mp4|out2.mp4" 

Possibly Benign Errors:
    Failed to set gop size: Invalid argument
    Timestamps are unset in a packet for stream 0. 
    This is deprecated and will stop working in the future. 
    Fix your code to set the timestamps properly
    Encoder did not produce proper pts, making some up.

Breaking Errors:
    Application provided invalid, non monotonically increasing dts to muxer in stream 0: 1 >= 1
    av_interleaved_write_frame(): Invalid argument

gst-launch-1.0 -v videotestsrc pattern=snow num-buffers=100 \
! video/x-raw,width=640,height=480 ! videoconvert \
! v4l2h264enc ! h264parse ! mp4mux ! tee name=t \
t. ! filesink async=0 location=out1.mp4 \
t. ! filesink async=0 location=out2.mp4

Breaking Errors:
    ERROR: from element /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0: 
    Failed to process frame.
    Additional debug info:
    ../sys/v4l2/gstv4l2videoenc.c(828): gst_v4l2_video_enc_handle_frame (): 
    /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0:
    Maybe be due to not enough memory or failing driver
    Execution ended after 0:00:00.021926718
    Setting pipeline to NULL ...
    ERROR: from element /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0: 
    Internal data stream error.
    Additional debug info:
    ../libs/gst/base/gstbasesrc.c(3127): gst_base_src_loop (): 
    /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0:
    streaming stopped, reason error (-5)

Steps to reproduce the behaviour

See the example commands above.

Device (s)

Raspberry Pi CM4

System

Raspberry Pi reference 2021-10-30
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, c12b1df4ed6416fb0df33ba1731c5b13c1bdbdf8, stage2

Jan 20 2022 13:56:48
Copyright (c) 2012 Broadcom
version bd88f66f8952d34e4e0613a85c7a6d3da49e13e2 (clean) (release) (start)

Linux PanoCamX 5.10.92-v8+ #1514 SMP PREEMPT Mon Jan 17 17:39:38 GMT 2022 aarch64 GNU/Linux

Logs

No response

Additional context

Please let me know if there's any particular debug / tests that would be helpful.

dustinkerstein commented 2 years ago

Quick update, same results after running rpi-update.

$ cat /etc/rpi-issue
Raspberry Pi reference 2021-10-30
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, c12b1df4ed6416fb0df33ba1731c5b13c1bdbdf8, stage2
$ vcgencmd version
Feb  4 2022 12:05:40
Copyright (c) 2012 Broadcom
version a26faf97e3bf76bcc23949d7cdab2f96f399a0c3 (clean) (release) (start)
$ uname -a
Linux PanoCamX 5.15.24-v8+ #1526 SMP PREEMPT Fri Feb 18 14:59:51 GMT 2022 aarch64 GNU/Linux

dustinkerstein commented 2 years ago

@6by9 Thanks for locking the dupe forum thread. I just wanted to make sure the issue was searchable over there as I figure not as many users know to search through these GitHub issues.

And apologies if this issue really belongs over at https://github.com/raspberrypi/firmware

raspberrypi / linux