rpiCamera source can generate B-frames, leading to WebRTC drops

wryun commented 5 months ago

Which version are you using?

v1.4.0

Which operating system are you using?

[ ] Linux amd64 standard
[ ] Linux amd64 Docker
[x] Linux arm64 standard
[ ] Linux arm64 Docker
[ ] Linux arm7 standard
[ ] Linux arm7 Docker
[ ] Linux arm6 standard
[ ] Linux arm6 Docker
[ ] Windows amd64 standard
[ ] Windows amd64 Docker (WSL backend)
[ ] macOS amd64 standard
[ ] macOS amd64 Docker
[ ] Other (please describe)

Describe the issue

When using a rpiCamera source at 640x480@10fps and flipping the image repeatedly, it's apparently possible to generate b-frames (which cause the WebRTC stream to break).

This was sort of raised as #1491, but in this case mediamtx itself is generating the stream, so there's the hope it would generate a stream without b-frames (some v4l2 option?). Interestingly, supposedly the rpi encoder block should not generate b-frames, which makes me wonder if something else is going on:

https://forums.raspberrypi.com/viewtopic.php?t=231368 https://github.com/raspberrypi/picamera2/issues/871

Describe how to replicate the issue

Start the server on a rpi4 with:

logLevel: debug
webrtc: true
paths:
rpicamera:
source: rpiCamera
rpiCameraWidth: 640
rpiCameraHeight: 480
rpiCameraFPS: 10

View localhost:8889/camera in browser
Repeatedly switch between a pixelated image and a black blank image (every second or so):
WebRTC stream will die within a minute with 'WebRTC doesn't support H264 stream with B-frames'

Did you attach the server logs?

yes

# mediamtx
2024/01/25 04:45:54 INF MediaMTX 1.4.0
2024/01/25 04:45:54 INF configuration loaded from /root/mediamtx.yml
2024/01/25 04:45:54 DEB [path rpicamera] created
2024/01/25 04:45:54 DEB path manager created
2024/01/25 04:45:54 INF [path rpicamera] [RPI Camera source] started
2024/01/25 04:45:54 INF [RTSP] listener opened on :8554 (TCP), :8000 (UDP/RTP), :8001 (UDP/RTCP)
2024/01/25 04:45:54 INF [RTMP] listener opened on :1935
2024/01/25 04:45:54 INF [HLS] listener opened on :8888
2024/01/25 04:45:54 INF [WebRTC] listener opened on :8889 (HTTP), :8189 (ICE/UDP)
2024/01/25 04:45:54 INF [SRT] listener opened on :8890 (UDP)
[1:28:36.391310905] [9850]  INFO Camera camera_manager.cpp:284 libcamera v22.03.5+178-e2cf9511
[1:28:36.457563516] [9853]  INFO RPI vc4.cpp:390 Registered camera /base/soc/i2c0mux/i2c@1/ov5647@36 to Unicam device /dev/media1 and ISP device /dev/media0
[1:28:36.459122979] [9850]  INFO Camera camera.cpp:1033 configuring streams: (0) 640x480-YUV420
[1:28:36.464065423] [9853]  INFO RPI vc4.cpp:512 Sensor: /base/soc/i2c0mux/i2c@1/ov5647@36 - Selected sensor format: 640x480-SGBRG10_1X10 - Selected unicam format: 640x480-pGAA
2024/01/25 04:45:55 INF [path rpicamera] [RPI Camera source] ready: 1 track (H264)
2024/01/25 04:45:56 DEB [WebRTC] [conn 10.42.0.179:35094] OPTIONS /rpicamera/whep
2024/01/25 04:45:56 DEB [WebRTC] [conn 10.42.0.179:35094] [c->s] OPTIONS /rpicamera/whep HTTP/1.1
Host: ekh-newflash:8889
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.5
Connection: keep-alive
Dnt: 1
Origin: http://ekh-newflash:8889
Referer: http://ekh-newflash:8889/rpicamera/
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:122.0) Gecko/20100101 Firefox/122.0

2024/01/25 04:45:56 DEB [WebRTC] [conn 10.42.0.179:35094] [s->c] HTTP/1.1 204 No Content
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Authorization, Content-Type, If-Match
Access-Control-Allow-Methods: OPTIONS, GET, POST, PATCH, DELETE
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Link
Server: mediamtx

2024/01/25 04:45:56 DEB [WebRTC] [conn 10.42.0.179:35094] POST /rpicamera/whep
2024/01/25 04:45:56 DEB [WebRTC] [conn 10.42.0.179:35094] [c->s] POST /rpicamera/whep HTTP/1.1
Host: ekh-newflash:8889
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.5
Connection: keep-alive
Content-Length: 3012
Content-Type: application/sdp
Dnt: 1
Origin: http://ekh-newflash:8889
Referer: http://ekh-newflash:8889/rpicamera/
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:122.0) Gecko/20100101 Firefox/122.0

v=0
o=mozilla...THIS_IS_SDPARTA-99.0 7729654223318688409 0 IN IP4 0.0.0.0
s=-
t=0 0
a=fingerprint:sha-256 51:EB:19:E0:07:59:C4:CA:88:FC:11:16:A2:05:23:22:0A:AF:DB:81:4E:61:DB:E5:D9:12:51:BE:B4:85:AC:91
a=group:BUNDLE 0 1
a=ice-options:trickle
a=msid-semantic:WMS *
m=video 9 UDP/TLS/RTP/SAVPF 120 124 121 125 126 127 97 98 123 122 119
c=IN IP4 0.0.0.0
a=sendrecv
a=extmap:3 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:4 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:5 urn:ietf:params:rtp-hdrext:toffset
a=extmap:6/recvonly http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:7 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=fmtp:126 profile-level-id=42e01f;level-asymmetry-allowed=1;packetization-mode=1
a=fmtp:97 profile-level-id=42e01f;level-asymmetry-allowed=1
a=fmtp:120 max-fs=12288;max-fr=60
a=fmtp:124 apt=120
a=fmtp:121 max-fs=12288;max-fr=60
a=fmtp:125 apt=121
a=fmtp:127 apt=126
a=fmtp:98 apt=97
a=fmtp:119 apt=122
a=ice-pwd:28877f50adfef64bc74e53615b1fcc1b
a=ice-ufrag:18a5c47c
a=mid:0
a=msid:- {b6ee0e93-9870-4161-a5f2-0768bdf0cd3f}
a=rtcp-fb:120 nack
a=rtcp-fb:120 nack pli
a=rtcp-fb:120 ccm fir
a=rtcp-fb:120 goog-remb
a=rtcp-fb:120 transport-cc
a=rtcp-fb:121 nack
a=rtcp-fb:121 nack pli
a=rtcp-fb:121 ccm fir
a=rtcp-fb:121 goog-remb
a=rtcp-fb:121 transport-cc
a=rtcp-fb:126 nack
a=rtcp-fb:126 nack pli
a=rtcp-fb:126 ccm fir
a=rtcp-fb:126 goog-remb
a=rtcp-fb:126 transport-cc
a=rtcp-fb:97 nack
a=rtcp-fb:97 nack pli
a=rtcp-fb:97 ccm fir
a=rtcp-fb:97 goog-remb
a=rtcp-fb:97 transport-cc
a=rtcp-fb:123 nack
a=rtcp-fb:123 nack pli
a=rtcp-fb:123 ccm fir
a=rtcp-fb:123 goog-remb
a=rtcp-fb:123 transport-cc
a=rtcp-fb:122 nack
a=rtcp-fb:122 nack pli
a=rtcp-fb:122 ccm fir
a=rtcp-fb:122 goog-remb
a=rtcp-fb:122 transport-cc
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:120 VP8/90000
a=rtpmap:124 rtx/90000
a=rtpmap:121 VP9/90000
a=rtpmap:125 rtx/90000
a=rtpmap:126 H264/90000
a=rtpmap:127 rtx/90000
a=rtpmap:97 H264/90000
a=rtpmap:98 rtx/90000
a=rtpmap:123 ulpfec/90000
a=rtpmap:122 red/90000
a=rtpmap:119 rtx/90000
a=setup:actpass
a=ssrc:292022122 cname:{cc95b05e-bf98-4b93-9171-8c9fcfd83131}
a=ssrc:448477053 cname:{cc95b05e-bf98-4b93-9171-8c9fcfd83131}
a=ssrc-group:FID 292022122 448477053
m=audio 9 UDP/TLS/RTP/SAVPF 109 9 0 8 101
c=IN IP4 0.0.0.0
a=sendrecv
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:2/recvonly urn:ietf:params:rtp-hdrext:csrc-audio-level
a=extmap:3 urn:ietf:params:rtp-hdrext:sdes:mid
a=fmtp:109 maxplaybackrate=48000;stereo=1;useinbandfec=1;sprop-stereo=1
a=fmtp:101 0-15
a=ice-pwd:28877f50adfef64bc74e53615b1fcc1b
a=ice-ufrag:18a5c47c
a=mid:1
a=msid:- {bab9e33e-fea4-4b90-b408-e43856e61959}
a=rtcp-mux
a=rtpmap:109 opus/48000/2
a=rtpmap:9 G722/8000/1
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:101 telephone-event/8000/1
a=setup:actpass
a=ssrc:3780180618 cname:{cc95b05e-bf98-4b93-9171-8c9fcfd83131}

2024/01/25 04:45:56 INF [WebRTC] [session ff0f312e] created by 10.42.0.179:35094
2024/01/25 04:45:56 DEB [WebRTC] [session ff0f312e] peer connection state: connecting
2024/01/25 04:45:56 DEB [WebRTC] [conn 10.42.0.179:35094] [s->c] HTTP/1.1 201 Created
Accept-Patch: application/trickle-ice-sdpfrag
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: ETag, ID, Accept-Patch, Link, Location
Content-Type: application/sdp
Etag: *
Id: ff0f312e-8627-482b-824a-5a6d008fd841
Location: whep/e9773c90-5c8c-442f-8741-6f33d1705f75
Server: mediamtx

(body of 1757 bytes)
2024/01/25 04:45:56 DEB [WebRTC] [conn 10.42.0.179:35094] PATCH /rpicamera/whep/e9773c90-5c8c-442f-8741-6f33d1705f75
2024/01/25 04:45:56 DEB [WebRTC] [conn 10.42.0.179:35094] [c->s] PATCH /rpicamera/whep/e9773c90-5c8c-442f-8741-6f33d1705f75 HTTP/1.1
Host: ekh-newflash:8889
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.5
Cache-Control: no-cache
Connection: keep-alive
Content-Length: 970
Content-Type: application/trickle-ice-sdpfrag
Dnt: 1
If-Match: *
Origin: http://ekh-newflash:8889
Pragma: no-cache
Referer: http://ekh-newflash:8889/rpicamera/
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:122.0) Gecko/20100101 Firefox/122.0

a=ice-ufrag:18a5c47c
a=ice-pwd:28877f50adfef64bc74e53615b1fcc1b
m=video 9 UDP/TLS/RTP/SAVPF 120 124 121 125 126 127 97 98 123 122 119
a=mid:0
a=candidate:0 1 UDP 2122252543 1a9430cb-f754-42de-a421-a06b0389c231.local 47868 typ host
a=candidate:1 1 TCP 2105524479 1a9430cb-f754-42de-a421-a06b0389c231.local 9 typ host tcptype active
a=candidate:0 2 UDP 2122252542 1a9430cb-f754-42de-a421-a06b0389c231.local 36954 typ host
a=candidate:1 2 TCP 2105524478 1a9430cb-f754-42de-a421-a06b0389c231.local 9 typ host tcptype active
a=
m=audio 9 UDP/TLS/RTP/SAVPF 109 9 0 8 101
a=mid:1
a=candidate:0 1 UDP 2122252543 1a9430cb-f754-42de-a421-a06b0389c231.local 43223 typ host
a=candidate:1 1 TCP 2105524479 1a9430cb-f754-42de-a421-a06b0389c231.local 9 typ host tcptype active
a=candidate:0 2 UDP 2122252542 1a9430cb-f754-42de-a421-a06b0389c231.local 49787 typ host
a=candidate:1 2 TCP 2105524478 1a9430cb-f754-42de-a421-a06b0389c231.local 9 typ host tcptype active
a=

2024/01/25 04:45:56 DEB [WebRTC] [conn 10.42.0.179:35094] [s->c] HTTP/1.1 204 No Content
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Server: mediamtx

2024/01/25 04:45:56 DEB [WebRTC] [session ff0f312e] peer connection state: connected
2024/01/25 04:45:56 INF [WebRTC] [session ff0f312e] peer connection established, local candidate: host/udp/10.42.0.1/8189, remote candidate: prflx/udp/10.42.0.179/47868
2024/01/25 04:45:56 INF [WebRTC] [session ff0f312e] is reading from path 'rpicamera', 1 track (H264)
2024/01/25 04:47:09 DEB [WebRTC] [session ff0f312e] peer connection state: closed
2024/01/25 04:47:09 INF [WebRTC] [session ff0f312e] closed: WebRTC doesn't support H264 streams with B-frames

Did you attach a network dump?

no

aler9 commented 2 months ago

Hello,

Repeatedly switch between a pixelated image and a black blank image (every second or so)

How do you perform this action? do you physically move the camera? do you physically move something in front of the camera? or is it hardware-based?

in this case mediamtx itself is generating the stream

MediaMTX is still using the Raspberry Pi hardware encoder to perform that action. As written in the Raspberry Pi forum, the hardware encoder doesn't support generating B-frames, therefore setting an additional V4L parameter would be useless.

wryun commented 1 month ago

Sorry I missed this earlier. I just put that image on my phone and used a sheet of paper to hide it repeatedly.

I haven't looked into exactly what is causing it, but I'm considering just patching the MediaMTX to ignore the supposed 'b-frames' (since the main issue is that this causes it to completely drop the stream).

bluenviron / mediamtx