search

CoSMoSoftware / OBS-studio-webrtc

This is a fork of OBS-studio with generic support for webrtc. It leverages the same webrtc implementation most browsers use.
http://www.cosmosoftware.io
GNU General Public License v2.0
590 stars 131 forks source link

MC - Add simulcast support for millicast #37

Closed murillo128 closed 3 years ago

murillo128 commented 5 years ago

simulcast is not enabled by default on millicast as we use h264 for targeting iOS devices

SCG82 commented 5 years ago

PR #147 appends a=x-google-flag:conference to the SDP as per https://dash.millicast.com/docs.html?pg=how-to-broadcast-in-js

agouaillard-cosmo commented 4 years ago

We contributed H264 simulcast to apple, then to google in 2019. Simulcast has been re-enabled in millicast back-end June 2020. Waiting for Jerry to update JS SDKs and publisher/viewer app, then will finally address this in OBS.

agouaillard-cosmo commented 4 years ago

The JS SDK have been updated, and simulcast works with JS publishers and viewers. It is not based on rid. A little bit more work needs to be done before we enable it in OBS:

agouaillard-cosmo commented 3 years ago

The above items have been executed, but the current implementation is not satisfactory. Needs to be revisited before release.

murillo128 commented 3 years ago

Simulcast is implemented now in OBS using transceiver apis and rids:

https://github.com/CoSMoSoftware/OBS-studio-webrtc/blob/m84-v23.2/plugins/obs-outputs/WebRTCStream.cpp#L291

However it seems that it is causing some bwe issues with millicast and the bitrate sent is to low (around 300kbps iirc).

In order to test new build, just start broadcasting to millicast and check the quality of the stream on the millicast viwer. The actual bitrate should be present on the obs logs and could be double checked on the millicast stats internal mononitoring tool.

agouaillard-cosmo commented 3 years ago

@murillo128 PTAL with m88v26.1.2

murillo128 commented 3 years ago

SDP offer seems to be truncated and does not include the simulcast attributes:

1:43:43.474: CONNECTING TO https://director-dev.millicast.com/api/director/publish
11:43:45.017: WSS url:          wss://live-dev.millicast.com/ws/v2/pub/(removed)
11:43:45.017: JWT (token):      (removed)
11:43:45.017: Connection URL:   wss://live-dev.millicast.com/ws/v2/pub/(removed)?token=(removed)
11:43:45.847: WebRTCStream::onConnected
11:43:45.847: WebRTCStream::onLogged
11:43:45.847: Creating offer...
11:43:45.847: (used_ids.h:55): Duplicate id found. Reassigning from 96 to 127
11:43:45.847: 
11:43:45.847: (used_ids.h:55): Duplicate id found. Reassigning from 102 to 126
11:43:45.847: 
11:43:45.847: (used_ids.h:55): Duplicate id found. Reassigning from 104 to 125
11:43:45.847: 
11:43:45.847: (used_ids.h:55): Duplicate id found. Reassigning from 102 to 124
11:43:45.847: 
11:43:45.847: (used_ids.h:55): Duplicate id found. Reassigning from 97 to 123
11:43:45.847: 
11:43:45.847: (used_ids.h:55): Duplicate id found. Reassigning from 103 to 122
11:43:45.847: 
11:43:45.847: (used_ids.h:55): Duplicate id found. Reassigning from 111 to 121
11:43:45.847: 
11:43:45.847: (used_ids.h:55): Duplicate id found. Reassigning from 109 to 120
11:43:45.847: 
11:43:45.847: (used_ids.h:55): Duplicate id found. Reassigning from 1 to 14
11:43:45.847: 
11:43:45.847: (used_ids.h:55): Duplicate id found. Reassigning from 3 to 13
11:43:45.847: 
11:43:45.847: (used_ids.h:55): Duplicate id found. Reassigning from 5 to 12
11:43:45.847: 
11:43:45.847: (used_ids.h:55): Duplicate id found. Reassigning from 6 to 11
11:43:45.847: 
11:43:45.847: WebRTCStream::OnSuccess
11:43:45.847: 
11:43:45.847: Audio codec:      multiopus
11:43:45.847: Audio bitrate:    160
11:43:45.847: 
11:43:45.847: Video codec:      VP8
11:43:45.847: Video bitrate:    2500
11:43:45.847: 
11:43:45.847: OFFER:
11:43:45.847: 
11:43:45.847: v=0
11:43:45.847: o=- 818475987456736062 2 IN IP4 127.0.0.1
11:43:45.847: s=-
11:43:45.847: t=0 0
11:43:45.847: a=group:BUNDLE 0 1
11:43:45.847: a=msid-semantic: WMS obs
11:43:45.847: m=audio 9 UDP/TLS/RTP/SAVPF 111 96 97 103 104 9 102 0 8
11:43:45.847: c=IN IP4 0.0.0.0
11:43:45.847: a=rtcp:9 IN IP4 0.0.0.0
11:43:45.847: a=ice-ufrag:7WT5
11:43:45.847: a=ice-pwd:aQm9dMsZI3YVRxfdGBCOKCjs
11:43:45.847: a=ice-options:trickle
11:43:45.847: a=fingerprint:sha-256 55:29:15:CC:D4:14:C1:12:65:2A:B4:B7:0B:B2:C4:A0:18:9D:F4:67:EC:7D:65:47:C7:E8:7E:E2:23:80:94:EF
11:43:45.847: a=setup:actpass
11:43:45.847: a=mid:0
11:43:45.847: a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
11:43:45.847: a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
11:43:45.847: a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
11:43:45.847: a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
11:43:45.847: a=extmap:5 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
11:43:45.847: a=extmap:6 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
11:43:45.847: a=sendonly
11:43:45.847: a=msid:obs audio
11:43:45.847: a=rtcp-mux
11:43:45.847: a=rtpmap:111 opus/48000/2
11:43:45.847: a=rtcp-fb:111 transport-cc
11:43:45.847: a=fmtp:111 minptime=10;useinbandfec=1
11:43:45.847: a=rtpmap:96 multiopus/48000/6
11:43:45.847: a=fmtp:96 channel_mapping=0,4,1,2,3,5;coupled_streams=2;minptime=10;num_streams=4;useinbandfec=1
11:43:45.847: a=rtpmap:97 multiopus/48000/8
11:43:45.847: a=fmtp:97 channel_mapping=0,6,1,2,3,4,5,7;coupled_streams=3;minptime=10;num_streams=5;useinbandfec=1
11:43:45.847: a=rtpmap:103 ISAC/16000
11:43:45.847: a=rtpmap:104 ISAC/32000
11:43:45.847: a=rtpmap:9 G722/8000
11:43:45.847: a=rtpmap:102 ILBC/8000
11:43:45.847: a=rtpmap:0 PCMU/8000
11:43:45.847: a=rtpmap:8 PCMA/8000
11:43:45.847: a=ssrc:4150715581 cname:fn2n7H3rK5SNORAY
11:43:45.847: a=ssrc:4150715581 msid:obs audio
11:43:45.847: a=ssrc:4150715581 mslabel:obs
11:43:45.847: a=ssrc:4150715581 label:audio
11:43:45.847: m=video 9 UDP/TLS/RTP/SAVPF 127 123 98 99 100 101 126 122 125 105 106 107 108 109 110 121 112 113 114
11:43:45.847: c=IN IP4 0.0.0.0
11:43:45.847: a=rtcp:9 IN IP4 0.0.0.0
11:43:45.847: a=ice-ufrag:7WT5
11:43:45.847: a=ice-pwd:aQm9dMsZI3YVRxfdGBCOKCjs
11:43:45.847: a=ice-options:trickle
11:43:45.847: a=fingerprint:sha-256 55:29:15:CC:D4:14:C1:12:65:2A:B4:B7:0B:B2:C4:A0:18:9D:F4:67:EC:7D:65:47:C7:E8:7E:E2:23:80:94:EF
11:43:45.847: a=setup:actpass
11:43:45.847: a=mid:1
11:43:45.847: a=extmap:14 urn:ietf:params:rtp-hdrext:toffset
11:43:45.847: a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
11:43:45.847: a=extmap:13 urn:3gpp:video-orientation
11:43:45.847: a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
11:43:45.847: a=extmap:12 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
11:43:45.847: a=extmap:11 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
11:43:45.847: a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
11:43:45.847: a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space
11:43:45.847: a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
11:43:45.847: a=extmap:5 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
11:43:45.847: a=extmap:6 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
11:43:45.847: a=sendonly
11:43:45.847: a=msid:obs video
11:43:45.847: a=rtcp-mux
11:43:45.847: a=rtcp-rsize
11:43:45.847: a=rtpmap:127 VP8/90000
11:43:45.847: a=rtcp-fb:127 goog-remb
11:43:45.847: a=rtcp-fb:127 transport-cc
11:43:45.847: a=rtcp-fb:127 ccm fir
11:43:45.847: a=rtcp-fb:127 nack
11:43:45.847: a=rtcp-fb:127 nack pli
11:43:45.847: a=rtpmap:123 rtx/90000
11:43:45.847: a=fmtp:123 apt=127
11:43:45.847: a=rtpmap:98 VP9/90000
11:43:45.847: a=rtcp-fb:98 goog-remb
11:43:45.847: a=rtcp-fb:98 transport-cc
11:43:45.847: a=rtcp-fb:98 ccm fir
11:43:45.847: a=rtcp-fb:98 nack
11:43:45.847: a=rtcp-fb:98 nack pli
11:43:45.847: a=fmtp:98 profile-id=0
11:43:45.847: a=rtpmap:99 rtx/90000
11:43:45.847: a=fmtp:99 apt=98
11:43:45.847: a=rtpmap:100 VP9/90000
11:43:45.847: a=rtcp-fb:100 goog-remb
11:43:45.847: a=rtcp-fb:100 transport-cc
11:43:45.847: a=rtcp-fb:100 ccm fir
11:43:45.847: a=rtcp-fb:100 nack
11:43:45.847: a=rtcp-fb:100 nack pli
11:43:45.847: a=fmtp:100 profile-id=2
11:43:45.847: a=rtpmap:101 rtx/90000
11:43:45.847: a=fmtp:101 apt=100
11:43:45.847: a=rtpmap:126 H264/90000
11:43:45.847: a=rtcp-fb:126 goog-remb
11:43:45.847: a=rtcp-fb:126 transport-cc
11:43:45.847: a=rtcp-fb:126 ccm fir
11:43:45.847: a=rtcp-fb:126 nack
11:43:45.847: a=rtcp-fb:126 nack pli
11:43:45.847: a=fmtp:126 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f
11:43:45.847: a=rtpmap:122 rtx/90000
11:43:45.847: a=fmtp:122 apt=126
11:43:45.847: a=rtpmap:125 H264/90000
11:43:45.847: a=rtcp-fb:125 goog-remb
11:43:45.847: a=rtcp-fb:125 transport-cc
11:43:45.847: a=rtcp-fb:125 ccm fir
11:43:45.847: a=rtcp-fb:125 nack
11:43:45.847: a=rtcp-fb:125 nack pli
11:43:45.847: a=fmtp:125 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42001f
11:43:45.847: a=rtpmap:105 rtx/90000
11:43:45.847: a=fmtp:105 apt=125
11:43:45.847: a=rtpmap:106 H264/90000
11:43:45.847: a=rtcp-fb:106 goog-remb
11:43:45.847: a=rtcp-fb:106 transport-cc
11:43:45.847: a=rtcp-fb:106 ccm fir
11:43:45.847: a=rtcp-fb:106 nack
11:43:45.847: a=rtcp-fb:106 nack pli
11:43:45.847: a=fmtp:106 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
11:43:45.847: a=rtpmap:107 rtx/90000
11:43:45.847: a=fmtp:107 apt=106
11:43:45.847: a=rtpmap:108 H264/90000
11:43:45.847: a=rtcp-fb:108 goog-re
11:43:45.859: SETTING LOCAL DESCRIPTION
murillo128 commented 3 years ago

Ok, simulcast is hardcoded set to false:

https://github.com/CoSMoSoftware/OBS-studio-webrtc/blob/m88-v26.1.2/plugins/obs-outputs/WebRTCStream.cpp#L314

    bool simulcast = false;

    //Add video track
    webrtc::RtpTransceiverInit video_init;
    video_init.stream_ids.push_back(stream->id());
    video_init.direction = webrtc::RtpTransceiverDirection::kSendOnly;
    if (simulcast) {

@ludocosmo could you take a look please?

ludocosmo commented 3 years ago

Simulcast attributes added to SDP offer:

a=rtpmap:98 VP9/90000
a=rtcp-fb:98 goog-remb
a=rtcp-fb:98 transport-cc
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=fmtp:98 profile-id=0;x-google-min-bitrate=2500;x-google-max-bitrate=2500
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98
a=rid:S send
a=rid:M send
a=rid:L send
a=simulcast:send S;M;L
murillo128 commented 3 years ago

Bandwidth reporting is not working with simulcast:

image

IIRC each simulcast stream creates it's own stats, so we would need to aggregate them, or better, display each of them.

murillo128 commented 3 years ago

I can also confirm that the simulcast streams are set correctly 

0:17:39.163: (video_send_stream_impl.cc:247): VideoSendStreamInternal: {encoder_settings: { experiment_cpu_load_estimator: off}}, rtp: {ssrcs: [1826576249, 2293218106, 2548306445], rids: [S, M, L], mid: '1', rtcp_mode: RtcpMode::kReducedSize, max_packet_size: 1200, extmap-allow-mixed: true, extensions: [{uri: http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01, id: 3}, {uri: http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time, id: 2}, {uri: urn:ietf:params:rtp-hdrext:sdes:mid, id: 4}, {uri: urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id, id: 6}, {uri: urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id, id: 5}], lntf: {enabled: false}, nack: {rtp_history_ms: 1000}, ulpfec: {ulpfec_payload_type: -1, red_payload_type: -1, red_rtx_payload_type: -1}, payload_name: VP8, payload_type: 127, raw_payload: false, flexfec: {payload_type: -1, ssrc: 0, protected_media_ssrcs: []}, rtx: {ssrcs: [1655993933, 1244315501, 180999196], payload_type: 123}, c_name: d+SxvTOaSXT8yPUv}, rtcp_report_interval_ms: 1000, send_transport: (Transport), render_delay_ms: 0, target_delay_ms: 0, suspend_below_min_bitrate: off}

However, something is going wrong inside libwerbtc with the bwe and the bitrate allocation of the simulcast layers:

10:21:38.850: (goog_cc_network_control.cc:688): bwe 419396149 pushback_target_bps=2500000 estimate_bps=2500000
10:21:38.850: 
10:21:38.850: (pacing_controller.cc:232): bwe:pacer_updated pacing_kbps=2750 padding_budget_kbps=2500
10:21:38.850: 
10:21:38.850: (video_stream_encoder.cc:1804): OnBitrateUpdated, bitrate 2159232 stable bitrate = 2078753 link allocation bitrate = 2159232 packet loss 0 rtt 172
10:21:38.850: 
10:21:38.850: (encoder_bitrate_adjuster.cc:225): Utilization factors for spatial index 0: link = 1, media = 1, wanted overshoot = 0 bps, available headroom = 609232 bps, total utilization factor = 1
10:21:38.850: 
10:21:38.850: (encoder_bitrate_adjuster.cc:225): Utilization factors for spatial index 1: link = 1, media = 1, wanted overshoot = 0 bps, available headroom = 609232 bps, total utilization factor = 1
10:21:38.850: 
10:21:38.850: (video_stream_encoder.cc:1140): Adjusting allocation, fps = 60, from VideoBitrateAllocation [
10:21:38.850:   [140000, 70000, 140000],
10:21:38.850:   [480000, 240000, 480000] ], to VideoBitrateAllocation [
10:21:38.850:   [140000, 70000, 140000],
10:21:38.850:   [480000, 240000, 480000] ]
10:21:38.850:

As you can see the bitrate estimation is over 2mbps but the allocation for the highest simulcast layer is of 480kbps+240kbps+480 kbps (which would match the bitrate received by the viewer).

murillo128 commented 3 years ago

This is definetively a weird behaviour on libwebrtc, changing input from 1080p to 720p works fine:

11:03:11.127: (video_stream_encoder.cc:1804): OnBitrateUpdated, bitrate 2167915 stable bitrate = 2164006 link allocation bitrate = 2167915 packet loss 0 rtt 170
11:03:11.127: 
11:03:11.127: (encoder_bitrate_adjuster.cc:225): Utilization factors for spatial index 0: link = 1, media = 1, wanted overshoot = 0 bps, available headroom = 915 bps, total utilization factor = 1
11:03:11.127: 
11:03:11.127: (encoder_bitrate_adjuster.cc:225): Utilization factors for spatial index 1: link = 1, media = 1, wanted overshoot = 0 bps, available headroom = 915 bps, total utilization factor = 1
11:03:11.127: 
11:03:11.127: (encoder_bitrate_adjuster.cc:225): Utilization factors for spatial index 2: link = 1, media = 1, wanted overshoot = 0 bps, available headroom = 915 bps, total utilization factor = 1
11:03:11.127: 
11:03:11.127: (video_stream_encoder.cc:1140): Adjusting allocation, fps = 30, from VideoBitrateAllocation [
11:03:11.127:   [60000, 30000, 60000],
11:03:11.127:   [200000, 100000, 200000],
11:03:11.127:   [607000, 303000, 607000] ], to VideoBitrateAllocation [
11:03:11.127:   [60000, 30000, 60000],
11:03:11.127:   [200000, 100000, 200000],
11:03:11.127:   [607000, 303000, 607000] ]
murillo128 commented 3 years ago

FWIW, doing vp8 simulcast 1080p@60fps via millicast web with chrome publisher works perfectly.

agouaillard-cosmo commented 3 years ago

done