search

raspberrypi / linux

Kernel source tree for Raspberry Pi-provided kernel builds. Issues unrelated to the linux kernel should be posted on the community forum at https://forums.raspberrypi.com/
Other
11.15k stars 5k forks source link

Video streaming can not be started again after streaming at 720p and frame rates higher than 60 #5348

Open Vavooon opened 1 year ago

Vavooon commented 1 year ago

Describe the bug

I use GStreamer with v4l2src and legacy camera stack for H.264/RTP streaming from Raspberry Pi camera v2.1. While 720p60 and below modes work fine, ones with higher frame rates (720p75, 720p90) can only be executed once.

While I'm completely happy with 720p60, the problem I see here is that it won't work with default params, since GStreamer automatically selects highest available frame rate for given resolution which is 90 in my case.

Working pipelines: gst-launch-1.0 v4l2src ! video/x-h264,width=1280,height=720,framerate=30/1 ! h264parse ! fakesink gst-launch-1.0 v4l2src ! video/x-h264,width=1280,height=720,framerate=60/1 ! h264parse ! fakesink Problematic pipelines: gst-launch-1.0 v4l2src ! video/x-h264,width=1280,height=720,framerate=75/1 ! h264parse ! fakesink gst-launch-1.0 v4l2src ! video/x-h264,width=1280,height=720,framerate=90/1 ! h264parse ! fakesink

When I try the same command for second time, GStreamer fails with error ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Failed to allocate required memory.

I have to reload the module in order to make it working again: sudo rmmod bcm2835_v4l2; sudo modprobe bcm2835_v4l2

Steps to reproduce the behaviour

gst-launch-1.0 v4l2src ! video/x-h264,width=1280,height=720,framerate=90/1 ! h264parse ! fakesink Ctrl+C gst-launch-1.0 v4l2src ! video/x-h264,width=1280,height=720,framerate=90/1 ! h264parse ! fakesink

Device (s)

Raspberry Pi Zero, Raspberry Pi 1 Mod. B

System

raspinfo.txt

Logs

720p30 start/stop log

bcm2835-v4l2-0: Clipping/aligning 1280x720 format 34363248
bcm2835-v4l2-0: Not removing padding, so bytes/line = 0
bcm2835-v4l2-0: Now 1280x720 format 34363248
bcm2835-v4l2-0: vidioc_try_fmt_vid_cap: w 1280 h 720 field 1 pfmt 0x34363248 bpl 0 sz_img 921600 colorspace 0x1 priv 0
bcm2835-v4l2-0: Clipping/aligning 1280x720 format 34363248
bcm2835-v4l2-0: Not removing padding, so bytes/line = 0
bcm2835-v4l2-0: Now 1280x720 format 34363248
bcm2835-v4l2-0: vidioc_try_fmt_vid_cap: w 1280 h 720 field 1 pfmt 0x34363248 bpl 0 sz_img 921600 colorspace 0x1 priv 0
bcm2835-v4l2-0: vid_cap - disconnect previous tunnel
bcm2835-v4l2-0: vid_cap - set up encode comp
bcm2835-v4l2-0: vid_cap - cur_buf.size set to 921600
bcm2835-v4l2-0: Set dev->capture.fmt 34363248, 1280x720, stride 0, size 921600
bcm2835-v4l2-0: Set fps range to 30/1 to 30/1
bcm2835-v4l2-0: queue_setup: dev:328ec5e3
bcm2835-v4l2-0: buffer_init: dev:328ec5e3, vb d45f036c
bcm2835-v4l2-0: buffer_init: dev:328ec5e3, vb 08f3d0a4
bcm2835-v4l2-0: buffer_init: dev:328ec5e3, vb b1329c78
bcm2835-v4l2-0: buffer_init: dev:328ec5e3, vb 43dc1005
bcm2835-v4l2-0: buffer_prepare: dev:328ec5e3, vb d45f036c
bcm2835-v4l2-0: buffer_prepare: dev:328ec5e3, vb 08f3d0a4
bcm2835-v4l2-0: buffer_prepare: dev:328ec5e3, vb b1329c78
bcm2835-v4l2-0: buffer_prepare: dev:328ec5e3, vb 43dc1005
bcm2835-v4l2-0: buffer_queue: dev:328ec5e3 buf:d45f036c, idx 0
bcm2835-v4l2-0: buffer_queue: dev:328ec5e3 buf:08f3d0a4, idx 1
bcm2835-v4l2-0: buffer_queue: dev:328ec5e3 buf:b1329c78, idx 2
bcm2835-v4l2-0: buffer_queue: dev:328ec5e3 buf:43dc1005, idx 3
bcm2835-v4l2-0: start_streaming: dev:328ec5e3
bcm2835-v4l2-0: enabled camera (refcount 1)
bcm2835-v4l2-0: Start time 3670053770 size 8
...
bcm2835-v4l2-0: stop_streaming: dev:328ec5e3
bcm2835-v4l2-0: stopping capturing
bcm2835-v4l2-0: disabling connection
bcm2835-v4l2-0: buffer_cb: status:0, buf:08f3d0a4, length:13159, flags 4100, pts 3672105278
bcm2835-v4l2-0: disabling port
bcm2835-v4l2-0: stop_streaming: Waiting for buffers to be returned - 3 outstanding
bcm2835-v4l2-0: stop_streaming: Waiting for buffers to be returned - 3 outstanding
bcm2835-v4l2-0: buffer_cb: status:0, buf:b1329c78, length:0, flags 0, pts 3672005297
bcm2835-v4l2-0: stop_streaming: Waiting for buffers to be returned - 2 outstanding
bcm2835-v4l2-0: buffer_cb: status:0, buf:43dc1005, length:0, flags 0, pts 3672038623
bcm2835-v4l2-0: stop_streaming: Waiting for buffers to be returned - 1 outstanding
bcm2835-v4l2-0: buffer_cb: status:0, buf:d45f036c, length:0, flags 0, pts 3672071951
bcm2835-v4l2-0: Disabling camera
bcm2835-v4l2-0: Camera refcount now 0
bcm2835-v4l2-0: buffer_cleanup: dev:328ec5e3, vb d45f036c
bcm2835-v4l2-0: buffer_cleanup: dev:328ec5e3, vb 08f3d0a4
bcm2835-v4l2-0: buffer_cleanup: dev:328ec5e3, vb b1329c78
bcm2835-v4l2-0: buffer_cleanup: dev:328ec5e3, vb 43dc1005

720p90 first start/stop log

bcm2835-v4l2-0: Clipping/aligning 1280x720 format 34363248
bcm2835-v4l2-0: Not removing padding, so bytes/line = 0
bcm2835-v4l2-0: Now 1280x720 format 34363248
bcm2835-v4l2-0: vidioc_try_fmt_vid_cap: w 1280 h 720 field 1 pfmt 0x34363248 bpl 0 sz_img 921600 colorspace 0x1 priv 0
bcm2835-v4l2-0: Clipping/aligning 1280x720 format 34363248
bcm2835-v4l2-0: Not removing padding, so bytes/line = 0
bcm2835-v4l2-0: Now 1280x720 format 34363248
bcm2835-v4l2-0: vidioc_try_fmt_vid_cap: w 1280 h 720 field 1 pfmt 0x34363248 bpl 0 sz_img 921600 colorspace 0x1 priv 0
bcm2835-v4l2-0: vid_cap - disconnect previous tunnel
bcm2835-v4l2-0: vid_cap - set up encode comp
bcm2835-v4l2-0: vid_cap - cur_buf.size set to 921600
bcm2835-v4l2-0: Set dev->capture.fmt 34363248, 1280x720, stride 0, size 921600
bcm2835-v4l2-0: Set fps range to 90/1 to 90/1
bcm2835-v4l2-0: queue_setup: dev:88c2913a
bcm2835-v4l2-0: buffer_init: dev:88c2913a, vb 05da452e
bcm2835-v4l2-0: buffer_init: dev:88c2913a, vb 328ec5e3
bcm2835-v4l2-0: buffer_init: dev:88c2913a, vb a1539edf
bcm2835-v4l2-0: buffer_init: dev:88c2913a, vb 199c6b0e
bcm2835-v4l2-0: buffer_prepare: dev:88c2913a, vb 05da452e
bcm2835-v4l2-0: buffer_prepare: dev:88c2913a, vb 328ec5e3
bcm2835-v4l2-0: buffer_prepare: dev:88c2913a, vb a1539edf
bcm2835-v4l2-0: buffer_prepare: dev:88c2913a, vb 199c6b0e
bcm2835-v4l2-0: buffer_queue: dev:88c2913a buf:05da452e, idx 0
bcm2835-v4l2-0: buffer_queue: dev:88c2913a buf:328ec5e3, idx 1
bcm2835-v4l2-0: buffer_queue: dev:88c2913a buf:a1539edf, idx 2
bcm2835-v4l2-0: buffer_queue: dev:88c2913a buf:199c6b0e, idx 3
bcm2835-v4l2-0: start_streaming: dev:88c2913a
bcm2835-v4l2-0: enabled camera (refcount 1)
bcm2835-v4l2-0: Start time 3000770028 size 8
...
bcm2835-v4l2-0: stop_streaming: dev:88c2913a
bcm2835-v4l2-0: stopping capturing
bcm2835-v4l2-0: buffer_cb: status:0, buf:199c6b0e, length:13551, flags 4100, pts 3002510251
bcm2835-v4l2-0: disabling connection
bcm2835-v4l2-0: buffer_cb: status:0, buf:05da452e, length:12326, flags 4100, pts 3002521331
bcm2835-v4l2-0: buffer_cb: status:0, buf:328ec5e3, length:4522, flags 4100, pts 3002543494
bcm2835-v4l2-0: disabling port
bcm2835-v4l2-0: buffer_cb: status:0, buf:a1539edf, length:2680, flags 4100, pts 3002554575
bcm2835-v4l2-0: Disabling camera
bcm2835-v4l2-0: Camera refcount now 0
bcm2835-v4l2-0: buffer_cleanup: dev:88c2913a, vb 05da452e
bcm2835-v4l2-0: buffer_cleanup: dev:88c2913a, vb 328ec5e3
bcm2835-v4l2-0: buffer_cleanup: dev:88c2913a, vb a1539edf
bcm2835-v4l2-0: buffer_cleanup: dev:88c2913a, vb 199c6b0e

720p90 second start/stop log (with issue)

bcm2835-v4l2-0: Clipping/aligning 1280x720 format 34363248
bcm2835-v4l2-0: Not removing padding, so bytes/line = 0
bcm2835-v4l2-0: Now 1280x720 format 34363248
bcm2835-v4l2-0: vidioc_try_fmt_vid_cap: w 1280 h 720 field 1 pfmt 0x34363248 bpl 0 sz_img 921600 colorspace 0x1 priv 0
bcm2835-v4l2-0: Clipping/aligning 1280x720 format 34363248
bcm2835-v4l2-0: Not removing padding, so bytes/line = 0
bcm2835-v4l2-0: Now 1280x720 format 34363248
bcm2835-v4l2-0: vidioc_try_fmt_vid_cap: w 1280 h 720 field 1 pfmt 0x34363248 bpl 0 sz_img 921600 colorspace 0x1 priv 0
bcm2835-v4l2-0: vid_cap - disconnect previous tunnel
bcm2835-v4l2-0: vid_cap - set up encode comp
bcm2835-v4l2-0: vid_cap - cur_buf.size set to 921600
bcm2835-v4l2-0: Set dev->capture.fmt 34363248, 1280x720, stride 0, size 921600
bcm2835-v4l2-0: Set fps range to 90/1 to 90/1
bcm2835-v4l2-0: queue_setup: dev:50c03c33
bcm2835-v4l2-0: buffer_init: dev:50c03c33, vb a3e81aa1
bcm2835-v4l2-0: buffer_init: dev:50c03c33, vb 881ba95e
bcm2835-v4l2-0: buffer_init: dev:50c03c33, vb 4df1a523
bcm2835-v4l2-0: buffer_init: dev:50c03c33, vb 3219a953
bcm2835-v4l2-0: buffer_prepare: dev:50c03c33, vb a3e81aa1
bcm2835-v4l2-0: buffer_prepare: dev:50c03c33, vb 881ba95e
bcm2835-v4l2-0: buffer_prepare: dev:50c03c33, vb 4df1a523
bcm2835-v4l2-0: buffer_prepare: dev:50c03c33, vb 3219a953
bcm2835-v4l2-0: buffer_queue: dev:50c03c33 buf:a3e81aa1, idx 0
bcm2835-v4l2-0: buffer_queue: dev:50c03c33 buf:881ba95e, idx 1
bcm2835-v4l2-0: buffer_queue: dev:50c03c33 buf:4df1a523, idx 2
bcm2835-v4l2-0: buffer_queue: dev:50c03c33 buf:3219a953, idx 3
bcm2835-v4l2-0: start_streaming: dev:50c03c33
bcm2835-v4l2-0: enabled camera (refcount 1)
bcm2835-v4l2-0: Failed to enable encode tunnel - error -3
------------[ cut here ]------------
WARNING: CPU: 0 PID: 936 at drivers/media/common/videobuf2/videobuf2-core.c:1552 vb2_start_streaming+0xf8/0x178 [videobuf2_common]
Modules linked in: bcm2835_v4l2(C) rfkill bcm2835_codec(C) v4l2_mem2mem bcm2835_isp(C) bcm2835_mmal_vchiq(C) videobuf2_dma_contig videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common snd_bcm2835(C) snd_pcm videodev raspberrypi_hwmon snd_timer snd vc_sm_cma(C) mc uio_pdrv_genirq fixed uio drm fuse drm_panel_orientation_quirks backlight ip_tables x_tables ipv6 [last unloaded: bcm2835_v4l2]
CPU: 0 PID: 936 Comm: v4l2src0:src Tainted: G        WC        5.15.61+ #1579
Hardware name: BCM2835
Backtrace: 
(show_stack+0x20/0x24)
 r7:00000610 r6:00000009 r5:bf359124 r4:c0ad9ce4
(dump_stack_lvl+0x24/0x28)
(dump_stack+0x14/0x1c)
 r5:bf359124 r4:bf35e06c
(__warn+0xe8/0x110)
(warn_slowpath_fmt+0x68/0xd8)
 r7:00000009 r6:bf359124 r5:00000610 r4:bf35e06c
(vb2_start_streaming+0xf8/0x178 [videobuf2_common])
 r9:bf157230 r8:c2bb6370 r7:40045612 r6:c2bb6510 r5:c2bb6608 r4:ffffffff
(vb2_core_streamon+0x8c/0x170 [videobuf2_common])
 r7:40045612 r6:00000001 r5:c982b280 r4:c2bb6510
(vb2_streamon+0x24/0x64 [videobuf2_v4l2])
 r5:c982b280 r4:00000001
(vb2_ioctl_streamon+0x4c/0x50 [videobuf2_v4l2])
(v4l_streamon+0x2c/0x30 [videodev])
 r5:bf237d64 r4:bf39e104
(__video_do_ioctl+0x230/0x460 [videodev])
 r5:bf237d64 r4:c2bb6058
(video_usercopy+0x120/0x5a8 [videodev])
 r10:c0c2b028 r9:00000000 r8:00000000 r7:c82d1e3c r6:00000000 r5:40045612
 r4:40045612
(video_ioctl2+0x1c/0x24 [videodev])
 r10:0000000b r9:0077a808 r8:c0c2b028 r7:c982b280 r6:00000000 r5:c982b281
 r4:bf23dd50
(v4l2_ioctl+0x4c/0x64 [videodev])
(sys_ioctl+0x114/0x9b0)
 r5:c982b281 r4:40045612
(ret_fast_syscall+0x0/0x1c)
Exception stack(0xc82d1fa8 to 0xc82d1ff0)
1fa0:                   007a89d8 00000000 0000000b 40045612 0077a808 b6bfed40
1fc0: 007a89d8 00000000 b5d80000 00000036 b4209b30 00000004 00000004 07f00010
1fe0: b6d05c58 b5342b94 b5d43924 b6bfed4c
 r10:00000000 r9:c82d0000 r8:c00083e4 r7:00000036 r6:b5d80000 r5:00000000
 r4:007a89d8
---[ end trace a46d7e08762e79e5 ]---
bcm2835-v4l2-0: buffer_cleanup: dev:50c03c33, vb a3e81aa1
bcm2835-v4l2-0: buffer_cleanup: dev:50c03c33, vb 881ba95e
bcm2835-v4l2-0: buffer_cleanup: dev:50c03c33, vb 4df1a523
bcm2835-v4l2-0: buffer_cleanup: dev:50c03c33, vb 3219a953

Additional context

No response

6by9 commented 1 year ago

H264 levels - https://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Levels

The default is level 4.0 (the spec for the encoder), which can do 1080p30 or 720p68.

Exceed the rates defined for the configured level and the encoder will reject it when you attempt to start it.

Use v4l2src extra-controls extra-controls="foo,h264_level=13. More recent versions of GStreamer it will take the H264 level from the caps, so making it video/x-h264,width=1280,height=720,framerate=30/1,level=(string)4.2 should do the same thing (it certainly applies to v4l2h264enc, but don't know if it applies to v4l2src).

Vavooon commented 1 year ago

Thanks for the explanation. It does help to use v4l2src extra-controls="foo,h264_level=13" in 720p mode.

Although, I rather wanted to report a problem with false capabilities advertisement provided by camera V4L2 interface. Again, the issue I see here is inability to pick highest available frame rate for the specified resolution when running a GStreamer pipeline without explicitly specified frame rate. E.g. when I run:

v4l2-ctl --device=/dev/video0 --list-frameintervals width=1920,height=1080,pixelformat=H264

It says:

ioctl: VIDIOC_ENUM_FRAMEINTERVALS
        Interval: Continuous 0.011s - 1.000s (1.000-90.000 fps)

Which is high above supported capabilities, so running something like gst-launch-1.0 v4l2src device=/dev/video0 extra-controls="foo,h264_level=13" ! video/x-h264, width=1920, height=1080 ! rtph264pay makes the same issue.

So the expected behavior to me is to advertise correct frame rates over V4L2 interface, e.g. 1080p30/720p60 for V2 camera or whatever is possible/officially advertised.

6by9 commented 1 year ago

It's a really a mismatch between the V4L2 API and how to expose the capabilities of the SoC.

For VIDIOC_ENUM_FRAMESIZES you either list discrete resolutions, or you can return CONTINUOUS or STEPWISE as you have cropping or scaling that can produce any output resolution. For each resolution and pixel format you can then list the supported frame rates via VIDIOC_ENUM_FRAMEINTERVALS. However if you're dealing with H264 just having the resolution and pixel format is insufficient as you need the codec level as well. There's no nice way to integrate that use of a control in with the VIDIOC_ENUM_FRAMEINTERVALS, so depending on how the last user left the control you could get any number of answers from the call.

Seeing as the legacy camera stack is deprecated in favour of libcamera, it's not a priority for us to fix it.

The lockup when you do exceed the H264 limit is a bug. I thought I'd sorted that a while ago (I remember it just being some state not cleaned up in the error path), but I guess either my memory is going, or the fix has been dropped due to a merge conflict.

Vavooon commented 1 year ago

Ok, thanks for the explanation. Unfortunately the legacy camera stack is the only way for me to work with H.264 video streaming on old Raspberry Pi models since it gives acceptable performance.

I hope this issue will help other people who could be affected by the problem.