WebRTC: Convert RTSP to WebRTC, very weird webrtc playback issue

[mopooo]

Describe the bug Pulling rtsp stream from camera(in h264 codec) and pushing rtmp to srs without any transcoding, webrtc can not play, but exectuting a snapshot on camera will make webrtc playback work, if clicking on play button on webrtc page, the playback will stop again

Version srs:v5.0-b6 in docker.

To Reproduce

1. launch srs docker
2. use ffmpeg to pull rtsp stream from camera and push to a rtmp url which srs is serving
3. open webrtc webpage provided by srs, playback is not working(for rtc player and WHEP tab)
4. use ffmpeg to get a snapshot of camera with same rtsp url
5. the webrtc player suddenly becomes alive
6. click play button on web page again, the playback stops and keeps loading forever
7. the data stream in chrome://webrtc-internals/ is always on for both good and bad cases, the difference is codec info is missing for bad cases, besides, the corresponding rtmp url is always playable in vlc player

Expected behavior webrtc should be working.

Screenshots

webrtc player stuck.

no video recognized in chrome

after a snapshot on camera, video is recognized without restarting anything

plicount keeps increasing until snapshot kicks in, meaning frame data were lost

Additional context snapshot command reference: ffmpeg -re -rtsp_transport tcp -i rtsp://xxx -y -f mjpeg -t 0.001 r13-snap.jpg

[mopooo]

attach client info and stream info here, no much difference between good case and bad case

{
    "code": 0,
    "server": "vid-397379i",
    "service": "69433079",
    "pid": "1",
    "streams": [
        {
            "id": "vid-a29251x",
            "name": "R13_M_CamW",
            "vhost": "vid-b434177",
            "app": "live",
            "tcUrl": "",
            "url": "/live/R13_M_CamW",
            "live_ms": 1699626174794,
            "clients": 2,
            "frames": 35950,
            "send_bytes": 308609887,
            "recv_bytes": 303813236,
            "kbps": {
                "recv_30s": 2035,
                "send_30s": 1937
            },
            "publish": {
                "active": true,
                "cid": "c60b7q46"
            },
            "video": {
                "codec": "H264",
                "profile": "Main",
                "level": "3.1",
                "width": 1280,
                "height": 720
            },
            "audio": null
        }
    ]
}

{
    "code": 0,
    "server": "vid-397379i",
    "service": "69433079",
    "pid": "1",
    "clients": [
        {
            "id": "9997v5i7",
            "vhost": "vid-b434177",
            "stream": "vid-a29251x",
            "ip": "127.0.0.1",
            "pageUrl": "",
            "swfUrl": "",
            "tcUrl": "",
            "url": "/live/R13_M_CamW",
            "name": "R13_M_CamW",
            "type": "rtc-play",
            "publish": false,
            "alive": 262.46,
            "send_bytes": 68099516,
            "recv_bytes": 142311,
            "kbps": {
                "recv_30s": 4,
                "send_30s": 1936
            }
        },
        {
            "id": "c60b7q46",
            "vhost": "vid-b434177",
            "stream": "vid-a29251x",
            "ip": "127.0.0.1",
            "pageUrl": "",
            "swfUrl": "",
            "tcUrl": "rtmp://127.0.0.1:1935/live",
            "url": "/live/R13_M_CamW",
            "name": "R13_M_CamW",
            "type": "fmle-publish",
            "publish": true,
            "alive": 1373.21,
            "send_bytes": 4291,
            "recv_bytes": 348432993,
            "kbps": {
                "recv_30s": 2014,
                "send_30s": 0
            }
        }
    ]
}

[mopooo]

looks like key frame is not recognized properly in srs for some rtsp source

ffprobe、ffmpeg pull rtsp again also can trigger the webrtc play again, but not for long, in webrtc debug console, it only triggers 2 keyframe decoded each time, but still, the rtmp output always works, so which one could be the problem, the camera or the webrtc implementation in srs?

[mopooo]

recorded 2 mp4 files by ffmpeg pulling rtsp stream from 2 cameras, bad.mp4 means webrtc cannot render properly(ffmpeg push mp4 file to srs directly), but both can be played with vlc or normal mp4 players

good.mp4

https://github.com/ossrs/srs/assets/6480394/e7a4abbd-bcdb-4489-a40b-8def8e1e8b1a

bad.mp4

https://github.com/ossrs/srs/assets/6480394/617914dd-6730-4290-b112-5a51afb5d7a6

[winlinvip]

I believe this issue is caused by the video content from your RTSP camera. Therefore, the best solution is to use ffmpeg to transcode your video stream. For example, you can pull the RTSP stream with ffmpeg and then transcode it to RTMP to publish to SRS. It is a straightforward conversion, but it should work very well.

Please note that when you transcode a stream with ffmpeg, you should disable the B-frames. You can simply use the baseline profile because WebRTC does not support B-frames.