WebRTC: Stuttering when coverting RTC to RTMP, especially for HD 720p.

[superyhee]

Note: Issues that do not provide the following information will be directly deleted. (Please follow the issue template, or we will delete it.)

Note: For inquiries and discussions, please submit them to the SRS forum at http://bbs.ossrs.net.

Description

When streaming through the RTC SDK using a browser, the video resolution is set to 1280720. However, there is severe buffering and playback issues when using RTMP or FLV. It is almost impossible to play the video smoothly. Interestingly, playback through RTC is smooth. If the video resolution is lowered to 640480, it can be played normally, but occasional buffering still occurs. Is there any issue with the configuration? Are there any limitations on the video resolution for RTC streaming currently?

Please describe your issue

I am unable to play RTMP and FLV files properly.

1. SRS Version: 4.0177 1. SRS Log:

xxxxxxxxxxxx

1. SRS Configuration:

# main config for srs.
# @see full.conf for detail config.

listen              1935;
max_connections     1000;
srs_log_tank        console;
daemon              off;

http_api {
    enabled         on;
    listen          1985;
}
http_server {
    enabled         on;
    listen          8080;
    dir             ./objs/nginx/html;
}
# stats {
#     network         0;
#     disk            sda sdb xvda xvdb;
# }
rtc_server {
    enabled         on;
    # Listen at udp://8000
    listen          8000;
    #
    # The $CANDIDATE means fetch from env, if not configed, use * as default.
    #
    # The * means retrieving server IP automatically, from all network interfaces,
    # @see https://github.com/ossrs/srs/issues/307#issuecomment-599028124
    candidate       $CANDIDATE;
}
vhost __defaultVhost__ {
**# Minimum Delay Opening (Default is open), when this option is open, MR is closed by default.**
    # tcp_nodelay     on;
    # min_latency     on;
**# Merged-Read, for RTMP protocol, in order to improve performance, SRS uses merged-read for reading from the upstream, meaning SRS reads data for N milliseconds at a time during read and write operations.**
    # mr {
    #     enabled     on;
**# Default is 350ms, range [300-2000].**
    #     latency     800;
    # }
**# Merged-Write: SRS always uses Merged-Write, which means sending packets to the client for N milliseconds at a time. This algorithm can improve the efficiency of RTMP downstream by about 5 times, range [350-1800].**
    # mw_latency      700;
    # enabled         on;
    #https://github.com/simple-rtmp-server/srs/wiki/v2_CN_LowLatency#gop-cache
    # gop_cache       on;
**# Configure the length of the live queue. The server will place data in the live queue, and if it exceeds this length, it will clear until the last I-frame.**
    #https://github.com/simple-rtmp-server/srs/wiki/v2_CN_LowLatency#%E7%B4%AF%E7%A7%AF%E5%BB%B6%E8%BF%9F
    # queue_length    10;
    # http_remux {
    #     enabled     on;
    #     mount       [vhost]/[app]/[stream].flv;
    # }
    rtc {
        enabled     on;
        # @see https://github.com/ossrs/srs/wiki/v4_CN_WebRTC#rtmp-to-rtc
        rtmp_to_rtc on;
        # @see https://github.com/ossrs/srs/wiki/v4_CN_WebRTC#rtc-to-rtmp
        rtc_to_rtmp on;
    }

    http_remux {
        enabled     on;
        mount       [vhost]/[app]/[stream].flv;
    }
    # play {
    #     gop_cache       off;
    #     queue_length    10;
    #     mw_latency      100;
    # }

    http_hooks {
        enabled         on;
        on_connect      http://localhost:3000/api/;
        on_close        http://localhost:3000/api/;
        on_publish      http://localhost:3000/api/;
        on_unpublish    http://localhost:3000/api/;
        on_play         http://localhost:3000/api/;
        on_stop         http://localhost:3000/api/;
    }

    # publish {
    #     mr off;
    # }
}

Replay

Steps to reproduce the bug (How to replay bug?)

1. Browser RTC streaming
2. xxxxxx
3. xxxxxx

Expected behavior The RTMP stream should play normally.

Please describe your expectation.

TRANS_BY_GPT3

[winlinvip]

This is an adaptation problem during the conversion of RTMP and RTC. High bitrate may cause some unknown issues, and it takes time to coordinate and debug together.

I have already notified @xiaozhihong and need to wait for his availability. You can also analyze it more, as this problem is quite difficult.

TRANS_BY_GPT3

[superyhee]

Thank you, the CPU resource consumption is not significant when converting RTC to RTMP with a 720p high bitrate, but there is basically no output on RTMP.

TRANS_BY_GPT3

[xiaozhihong]

Thank you, the CPU resource consumption is not high when RTC is converted to RTMP at a high bitrate of 720p, but the RTMP output is basically not available.

Can you provide me with a video file or let me know the streaming method and the corresponding configuration? For example, the bitrate, fps, and the duration between keyframes.

TRANS_BY_GPT3

[superyhee]

Without using a video file, in Chrome use navigator.mediaDevices.getUserMedia(constraints) to obtain the camera stream, and then pass the stream to rts_sdk for streaming. The information for the constraint is as follows: var constraints = { video: { deviceId: this.props.stateContext.config.cameraId, width: 1280, height: 720, frameRate: 20 }, audio: { deviceId: this.props.stateContext.config.microphoneId, } };

TRANS_BY_GPT3

[xiaozhihong]

Without using a video file, use navigator.mediaDevices.getUserMedia(constraints) in Chrome to obtain camera stream, and then pass the stream to rts_sdk for streaming. The constraint information is as follows: var constraints = { video: { deviceId: this.props.stateContext.config.cameraId, width: 1280, height: 720, frameRate: 20 }, audio: { deviceId: this.props.stateContext.config.microphoneId, } };

When buffering occurs, is it played locally on localhost, or in the local network, or in the public network?

TRANS_BY_GPT3

[superyhee]

Local playback, unrelated to the internet, RTC playback is very smooth, but RTMP streaming is basically unusable.

TRANS_BY_GPT3

[bywwcnll]

Thumbs up, thumbs up, thumbs up

TRANS_BY_GPT3

[superyhee]

I tested it with the latest XCORE-SRS/4.0.195 (Leo) version using the previous method, but it still freezes a lot. Sometimes I can play rtmp, but there is a delay of more than 10 seconds, and the audio and video are not synchronized.

TRANS_BY_GPT3

[xiaozhihong]

I tested it using the latest XCORE-SRS/4.0.195(Leo) version in the same way as before, but it still lags. Sometimes it can play RTMP, but with a delay of over 10 seconds and the audio and video are not synchronized.

Are you in the SRS WeChat group? If you are, please add me so that I can take a look at your setup.

TRANS_BY_GPT3

[superyhee]

Not here, how can I add you on WeChat?

John @.*** wrote on Thursday, November 11, 2021, at 9:55 PM:

I tested it using the latest XCORE-SRS/4.0.195 (Leo) version with the previous method, but it is still lagging. Sometimes it can play RTMP, but there is a delay of more than 10 seconds, and the audio and video are not synchronized.

Are you in the SRS WeChat group? If you are, please add me so that I can see your scenario.

- You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ossrs/srs/issues/2677#issuecomment-966321180, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCT5ZCCJ2DMRWOL3DUAWYDULPDNPANCNFSM5F6QHLEQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

TRANS_BY_GPT3

[xiaozhihong]

I'm not available. How can I add you on WeChat?

john @.***> wrote on Thursday, November 11, 2021, at 9:55 PM:

[...] I tried testing with the latest XCORE-SRS/4.0.195 (Leo) version using the previous method, but it's still very laggy. Sometimes it can play rtmp, but with a delay of more than 10 seconds, and the audio and video are not synchronized. Are you in the SRS WeChat group? If you are, please add me so I can see your setup.

@superyhee sent you an email.

TRANS_BY_GPT3

[superyhee]

Can't see any information from WeChat groups?

TRANS_BY_GPT3

[KevinLoveGitHub]

SRS 4.0.177 version encountered a similar issue. In a local area network environment, webrtc 480p 15-frame pushing has very good RTC pulling effect. However, RTMP pulling has a delay of about 5-10 seconds and often experiences frequent disconnections.

TRANS_BY_GPT3

[bywwcnll]

Encountered similar issues in SRS 4.0.177 version. In a local area network environment, webrtc 480p, 15 fps streaming has a very good RTC playback effect. However, RTMP playback has a delay of approximately 5-10 seconds and frequent disconnections.

Compiled and tested with the latest code because some issues were resolved recently. Currently, apart from the latency, there are basically no more disconnection problems.

TRANS_BY_GPT3

[KevinLoveGitHub]

Encountered similar issues with SRS 4.0.177 version. In a local area network environment, webrtc 480p 15 fps streaming has excellent playback results. However, there is a delay of around 5-10 seconds and frequent stream interruptions when using RTMP playback.

Compiled and tested with the latest code, as some of the issues have recently been resolved. Currently, apart from the delay, there are basically no stream interruption problems.

Thank you for your reply. I have tested it with version 4.0.205 on Ubuntu 18.04, and the situation is similar to before. There are some issues with the DVR recording feature:

1. There are instances where only audio without video, or only one frame of video, appears for the first 10 seconds or so.
2. In the middle of the recorded file, there may be a period of time where only audio is present, and the video freezes.
3. After the recording finishes, the flv file cannot be accessed via HTTP in the browser.

TRANS_BY_GPT3

[winlinvip]

It seems like the issue still exists, Reopen.

TRANS_BY_GPT3

[BlueAirRL]

The same problem with RTC live streaming, DVR-FLV recording is also very choppy, but RTC playback is smooth. SRS version 4.0.177, resolution: 320*180.

TRANS_BY_GPT3

[winlinvip]

You must attach the FLV file for it to work. This is definitely a problem with the file content. Without the file content, it is impossible to troubleshoot.

TRANS_BY_GPT3

[BlueAirRL]

The previous rtc to rtmp memory overflow should be related. #2680 The package cannot be consumed in time, causing stuttering in rtc to rtmp playback and recording.

TRANS_BY_GPT3

[winlinvip]

Please do not guess what the reason is.

If there is no data, this problem closes.

The problems with audio and video cannot be completely solved, and it's not just about audio and video, there are a bunch of other issues as well.

TRANS_BY_GPT3

[superyhee]

It's actually very easy to reproduce. Use 720p streaming and constantly switch the screen. When pulling the stream with ffmpeg, it will show 'Non-monotonous DTS in output stream 0:0', and then it will start lagging.

TRANS_BY_GPT3

[winlinvip]

How often should we switch screens?

TRANS_BY_GPT3

[superyhee]

No need for frequent occurrence, as long as there is significant change in the screen, Doctor Xiao's preliminary diagnosis is that the sender report crosses in the middle of long I frames, resulting in floating-point calculation error, causing a timestamp synchronization error.

TRANS_BY_GPT3

[winlinvip]

That seems to know where the problem lies, just waiting for the fix patch? Why does it feel like the shortest reproducible path has not been found yet?

TRANS_BY_GPT3

[superyhee]

I really didn't find the shortest necessary path, it has a certain randomness, but it appears easily. This bug indeed affects the use of rtc2rtmp, including subsequent streaming or recording operations with ffmpeg. The exact reason for the specific problem is clearer to xiaozhihong.

TRANS_BY_GPT3

[winlinvip]

We generally don't solve occasional problems, and we don't require anyone to solve necessary ones. The rule of open source is to solve one's own problems.

So it's better to spend more time on how to address the necessary problems.

TRANS_BY_GPT3

[superyhee]

Understood, I will take a look when I have time.

TRANS_BY_GPT3

[funbinary]

I also encountered the same situation. Using SRS version SRS/4.0.240, configuration file rtc2rtmp.conf. Steps to reproduce using SRS built-in signaling:

1. Change the shared camera to shared screen, modify the file srs.sdk.js, change getUserMedia to getDisplayMedia, and set constraints. The video width only needs to be greater than 768. My constraints are as follows:

   
   self.constraints = {
       audio: true,
       video: {
   The translation to English is as follows:

//         aspectRatio: 16 / 9, // aspect ratio
//         frameRate: 24, // frame rate
      width: 768, // width
      height: 432 // height

Please note that the commented lines have been omitted in the translation. }

2. Start a call to share a web tab to play videos in 720P or above. I am using Tencent Video to play a 720P video.
3. Then, when using ffplay or potplayer to play the video, it always lags. When using potplayer to play, I noticed that the frame rate drops below 10, as shown in the picture.
![image](https://user-images.githubusercontent.com/44047515/154023212-b3985a65-25a3-488c-8f58-c2571c51c600.png)

`TRANS_BY_GPT3`

[winlinvip]

This pit is not small, hahaha.

TRANS_BY_GPT3

[lipeng19811218]

Can you take a look at the SRS code where the fixed request pli is used? If it is currently every 1 second, can you change it to every 5 seconds and see if it works?

TRANS_BY_GPT3

[winlinvip]

I checked the configuration file, and the default value written in full.conf is 6 seconds.

cat ~/srs/conf/full.conf |grep pli_for_rtmp
#        pli_for_rtmp 6.0;

TRANS_BY_GPT3

[0xhyperdan]

rtc {
        # Whether enable WebRTC server.
        # default: off
        enabled on;
        # Whether support NACK.
        # default: on
        nack on;
        # Whether directly use the packet, avoid copy.
        # default: on
        nack_no_copy on;
        # Whether support TWCC.
        # default: on
        twcc on;
        # The timeout in seconds for session timeout.
        # Client will send ping(STUN binding request) to server, we use it as heartbeat.
        # default: 30
        stun_timeout 30;
        # The strict check when process stun.
        # default: off
        stun_strict_check on;
        # The role of dtls when peer is actpass: passive or active
        # default: passive
        dtls_role passive;
        # The version of dtls, support dtls1.0, dtls1.2, and auto
        # default: auto
        dtls_version auto;
        # Drop the packet with the pt(payload type), 0 never drop.
        # default: 0
        drop_for_pt 0;
        ###############################################################
        # Whether enable transmuxing RTMP to RTC.
        # If enabled, transcode aac to opus.
        # default: off
        rtmp_to_rtc off;
        # Whether keep B-frame, which is normal feature in live streaming,
        # but usually disabled in RTC.
        # default: off
        keep_bframe off;
        ###############################################################
        # Whether enable transmuxing RTC to RTMP.
        # Default: off
        rtc_to_rtmp off;
        # The PLI interval in seconds, for RTC to RTMP.
        # Note the available range is [0.5, 30]
        # Default: 6.0
        pli_for_rtmp 6.0;
    }

Just change 'pli_for_rtmp' to a minimum value of 0.5.

TRANS_BY_GPT3

[superyhee]

I tested changing pli_for_rtmp to 0.5 and also used getDisplayMedia to play a high-definition video. It did improve the smoothness compared to before, but there is still a DTS issue when using ffplay. [flv @ 0x7fd92100ce00] DTS 9969 < 10167 out of order= 0B f=0/0 [flv @ 0x7fd92100ce00] DTS 64824 < 65051 out of order 0B f=3/3 [flv @ 0x7fd92100ce00] DTS 85232 < 85385 out of order 0B f=6/6

TRANS_BY_GPT3

[winlinvip]

What people are saying is that enlarging PLI can solve the problem, but you, on the other hand, feel that shrinking it makes it smoother...

TRANS_BY_GPT3

[zsinba]

Coincidentally, same problem.

TRANS_BY_GPT3

[SuperChrisliu]

What they said is that enlarging PLI can solve the problem, but you did the opposite and made it smaller, thinking it would be smooth...

@winlinvip To share the screen on Chrome, use navigator.mediaDevices.getDisplayMedia to push. For an online stopwatch webpage, use the mini-program RTMP player to play. Every two seconds, the screen moves a bit. Check the network status output on the mini-program side. When the screen freezes, the network speed is 0. When there is a display of network speed, the screen does not freeze. After reducing the pli_for_rtmp parameter by 0.5, it indeed no longer freezes. But when set to 8, it freezes heavily. SRS version: 4.0.251. pli_for_rtmp is set to 0.5.

https://user-images.githubusercontent.com/7054257/187113125-44e59ea9-150a-41d9-b72f-2708c25786be.mp4

pli_for_rtmp is set to 8. https://user-images.githubusercontent.com/7054257/187113217-67d6854e-32d5-48d9-a61f-53f7326fab0b.mp4

TRANS_BY_GPT3

[galaxy-s10]

The same problem, just change the value of pli_for_rtmp inside rtc to a minimum of 0.5.

TRANS_BY_GPT3

[galaxy-s10]

I tested changing pli_for_rtmp to 0.5 and used getDisplayMedia to play a high-definition video. Indeed, it is smoother than before, but there is still a DTS problem when using ffplay [flv @ 0x7fd92100ce00] DTS 9969 < 10167 out of order= 0B f=0/0 [flv @ 0x7fd92100ce00] DTS 64824 < 65051 out of order 0B f=3/3 [flv @ 0x7fd92100ce00] DTS 85232 < 85385 out of order 0B f=6/6

Has this been resolved? Changing pli_for_rtmp to 0.5 indeed reduces the stuttering and makes the playback smoother, but every one or two seconds, there is still a delay of several hundred milliseconds (using navigator.mediaDevices.getDisplayMedia in the browser).

TRANS_BY_GPT3