search

Unity-Technologies / com.unity.webrtc

WebRTC package for Unity
Other
739 stars 185 forks source link

[BUG]: Simulcast not working with H264 rid=h;m;l #925

Closed avicarpio closed 8 months ago

avicarpio commented 1 year ago

Package version

3.0.0-pre.5

Environment

* OS: Linux Ubuntu 22.04
* Unity version: 2020.3.35f1

Steps To Reproduce

Create a transceiver before the sdp that has the 3 qualities on the encoding parameters like this:

private IEnumerator addTransceiverSimulcast(List<MediaStreamTrack> mediaStreamTracks){
        RTCRtpTransceiverInit setTransceiver = new RTCRtpTransceiverInit();

        setTransceiver.direction = RTCRtpTransceiverDirection.SendOnly;

        List<RTCRtpEncodingParameters> parameters = new List<RTCRtpEncodingParameters>();

        RTCRtpEncodingParameters encoder = new RTCRtpEncodingParameters();

        encoder.rid = "h";
        encoder.active = true;
        encoder.maxBitrate = ConvertToBits(12);
        encoder.minBitrate = ConvertToBits(11.5f);
        encoder.maxFramerate = 60;
        encoder.scaleResolutionDownBy = 1;

        parameters.Add(encoder);

        encoder = new RTCRtpEncodingParameters();

        encoder.rid = "m";
        encoder.active = true;
        encoder.maxBitrate = ConvertToBits(3.5f);
        encoder.minBitrate = ConvertToBits(3);
        encoder.maxFramerate = 15;
        encoder.scaleResolutionDownBy = 1;

        parameters.Add(encoder);

        encoder = new RTCRtpEncodingParameters();

        encoder.rid = "l";
        encoder.active = true;
        encoder.maxBitrate = ConvertToBits(1.5f);
        encoder.minBitrate = ConvertToBits(1);
        encoder.maxFramerate = 1;
        encoder.scaleResolutionDownBy = 2;

        parameters.Add(encoder);

        setTransceiver.sendEncodings = parameters.ToArray();

        while (publishPeerConnection == null)
        {
            yield return 0;
        }

        MediaStreamTrack videoTrack = null;
        MediaStreamTrack audioTrack = null;

        foreach (var streamTrack in mediaStreamTracks)
        {
            if(streamTrack.Kind == TrackKind.Video){
                videoTrack = streamTrack;
            }
            if(streamTrack.Kind == TrackKind.Audio){
                audioTrack = streamTrack;
            }
        }

        if(videoTrack != null && audioTrack != null){
            var transceiver = publishPeerConnection.AddTransceiver(videoTrack, setTransceiver);
            publishPeerConnection.AddTransceiver(audioTrack);

            // Get all available video codecs.
            var codecs = RTCRtpSender.GetCapabilities(TrackKind.Video).codecs;

            // Filter codecs.
            var h264Codecs = codecs.Where(codec => codec.mimeType == "video/H264");    

            var error = transceiver.SetCodecPreferences(h264Codecs.ToArray());
            if (error != RTCErrorType.None)
                Debug.LogError("SetCodecPreferences failed | " + error);
        }        
    }

Then generate the sdp:

v=0
no=- 3383397139837300255 2 IN IP4 127.0.0.1
ns=-
nt=0 0
na=group:BUNDLE 0 1
na=extmap-allow-mixed
na=msid-semantic: WMS e23ac07d-a5e6-4c2b-a93e-c7ab09c9ffe2
nm=video 9 UDP/TLS/RTP/SAVPF 123 122 121 120
nc=IN IP4 0.0.0.0
na=rtcp:9 IN IP4 0.0.0.0
na=ice-ufrag:Y1Pj
na=ice-pwd:w0W80QYe7Q+ZdOxPw4pyaXJs
na=ice-options:trickle
na=fingerprint:sha-256 38:0C:1B:60:89:94:A4:2D:E9:28:41:5C:A0:99:72:A4:2C:79:36:34:E2:73:4E:9E:C4:93:5A:14:54:F2:61:08
na=setup:actpass
na=mid:0
na=extmap:1 urn:ietf:params:rtp-hdrext:toffset
na=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
na=extmap:3 urn:3gpp:video-orientation
na=extmap:4 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
na=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
na=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
na=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
na=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space
na=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
na=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
na=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
na=sendonly
na=msid:e23ac07d-a5e6-4c2b-a93e-c7ab09c9ffe2 9ead9e35-677b-4a09-a325-9a96cfcd9f39
na=rtcp-mux
na=rtcp-rsize
na=rtpmap:123 H264/90000
na=rtcp-fb:123 goog-remb
na=rtcp-fb:123 transport-cc
na=rtcp-fb:123 ccm fir
na=rtcp-fb:123 nack
na=rtcp-fb:123 nack pli
na=fmtp:123 implementation_name=NvCodec;level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e033
na=rtpmap:122 H264/90000
na=rtcp-fb:122 goog-remb
na=rtcp-fb:122 transport-cc
na=rtcp-fb:122 ccm fir
na=rtcp-fb:122 nack
na=rtcp-fb:122 nack pli
na=fmtp:122 implementation_name=NvCodec;level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=420033
na=rtpmap:121 H264/90000
na=rtcp-fb:121 goog-remb
na=rtcp-fb:121 transport-cc
na=rtcp-fb:121 ccm fir
na=rtcp-fb:121 nack
na=rtcp-fb:121 nack pli
na=fmtp:121 implementation_name=NvCodec;level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=640033
na=rtpmap:120 H264/90000
na=rtcp-fb:120 goog-remb
na=rtcp-fb:120 transport-cc
na=rtcp-fb:120 ccm fir
na=rtcp-fb:120 nack
na=rtcp-fb:120 nack pli
na=fmtp:120 implementation_name=NvCodec;level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=4d0033
na=rid:h send
na=rid:m send
na=rid:l send
na=simulcast:send h;m;l
nm=audio 9 UDP/TLS/RTP/SAVPF 96 97 98 99 102 103 104 9 0 8 100 101 107 108 109 114 110 112 113 126
nc=IN IP4 0.0.0.0
na=rtcp:9 IN IP4 0.0.0.0
na=ice-ufrag:Y1Pj
na=ice-pwd:w0W80QYe7Q+ZdOxPw4pyaXJs
na=ice-options:trickle
na=fingerprint:sha-256 38:0C:1B:60:89:94:A4:2D:E9:28:41:5C:A0:99:72:A4:2C:79:36:34:E2:73:4E:9E:C4:93:5A:14:54:F2:61:08
na=setup:actpass
na=mid:1
na=extmap:14 urn:ietf:params:rtp-hdrext:ssrc-audio-level
na=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
na=extmap:4 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
na=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
na=sendrecv
na=msid:e23ac07d-a5e6-4c2b-a93e-c7ab09c9ffe2 82071518-d86e-40e0-8bac-08fe5d96df7b
na=rtcp-mux
na=rtpmap:96 opus/48000/2
na=rtcp-fb:96 transport-cc
na=fmtp:96 minptime=10;sprop-stereo=1;stereo=1;useinbandfec=1
na=rtpmap:97 red/48000/2
na=fmtp:97 96/96
na=rtpmap:98 multiopus/48000/6
na=fmtp:98 channel_mapping=0,4,1,2,3,5;coupled_streams=2;minptime=10;num_streams=4;useinbandfec=1
na=rtpmap:99 multiopus/48000/8
na=fmtp:99 channel_mapping=0,6,1,2,3,4,5,7;coupled_streams=3;minptime=10;num_streams=5;useinbandfec=1
na=rtpmap:102 ILBC/8000
na=rtpmap:103 ISAC/16000
na=rtpmap:104 ISAC/32000
na=rtpmap:9 G722/8000
na=rtpmap:0 PCMU/8000
na=rtpmap:8 PCMA/8000
na=rtpmap:100 L16/8000
na=rtpmap:101 L16/16000
na=rtpmap:107 L16/32000
na=rtpmap:108 L16/8000/2
na=rtpmap:109 L16/16000/2
na=rtpmap:114 L16/32000/2
na=rtpmap:110 telephone-event/48000
na=rtpmap:112 telephone-event/32000
na=rtpmap:113 telephone-event/16000
na=rtpmap:126 telephone-event/8000
na=ssrc:1720815888 cname:UViF8F+6ZRrj/g6V
na=ssrc:1720815888 msid:e23ac07d-a5e6-4c2b-a93e-c7ab09c9ffe2 82071518-d86e-40e0-8bac-08fe5d96df7b

On the client browser, I can only see the highest quality, changing the substream is ignored because Unity only generates the highest substream. Do you see anything wrong on the code or on the sdp? Thank you.

Current Behavior

Only highest quality is generated.

Expected Behavior

To generate 3 qualities, that is what simulcast should do.

Anything else?

This is the log that Janus (the WebRTC server I am using) dumps: janus.log

Edit: I can confirm that only the highest quality is being generated (this is from the janus.log):

 "in": {
              "packets": 0,
              "bytes": 0,
              "bytes_lastsec": 0,
              "nacks": 0,
              "retransmissions": 0,
              "video-simulcast-2": {
                "packets": 6005196,
                "bytes": 2626644010,
                "bytes_lastsec": 1951136,
                "nacks": 0
              }
            },
karasusan commented 1 year ago

@avicarpio Thank you for sharing the code, is it work with VP8 codec? If so, NvCodec implementation would be caused by the issue.

avicarpio commented 1 year ago

Hi @karasusan, yes, I see it works with VP8, but we can't use it because we need HW Encoding since we work with high quality video and we use AWS machines. Making simulcast with VP8 with the encodings we use is super exhausting and we would need a very expensive machine. Also, the quality is not as flawless as H264. So we should use simulcast with NVCodec, is it possible? Thank you.

avicarpio commented 1 year ago

Btw, I've cloned the project and I found this comment on NvEncoderImpl.cpp:

        // todo(kazuki): Add multiple configurations to support simulcast
        m_configurations[0].width = m_codec.width;
        m_configurations[0].height = m_codec.height;
        m_configurations[0].sending = false;
        m_configurations[0].max_frame_rate = static_cast<float>(m_codec.maxFramerate);
        m_configurations[0].key_frame_interval = m_codec.H264()->keyFrameInterval;
        m_configurations[0].max_bps = m_codec.maxBitrate * 1000;
        m_configurations[0].target_bps = m_codec.startBitrate * 1000;

Is it correct that it is still on todo? Is it that part of the code that does not allow simulcasting? If so, is there any plan to implement it in the short term? Since simulcast is something quite necessary and fundamental in WebRTC. Thank you very much.

karasusan commented 1 year ago

memo: WRS-482

karasusan commented 1 year ago

@avicarpio Yes, we haven't supported simulcasting with NvCodec. I will try to reproduce your issue in my environment.

avicarpio commented 1 year ago

Thank you very much @karasusan! I will be waiting for the news to be able to use it as soon as possible. If you need anything else I will be happy to help you.

karasusan commented 11 months ago

We are fixing this issue on this PR. https://github.com/Unity-Technologies/com.unity.webrtc/pull/955