[BUG]: Simulcast only working stable on 720p or lower qualities

[avicarpio]

Package version

3.0.0-pre.7

Environment

* OS:Ubuntu 22.04
* Unity version: 2022.3.3f1

Steps To Reproduce

I'm repeating the test of https://github.com/Unity-Technologies/com.unity.webrtc/issues/925.

With the latest version 3.0.0-pre.7 I've seen that now simulcast works, but it only takes the 2 lowest quality. Normally, simulcast is configured by setting 3 rids (h;m;l) so only 2 of them can be used.

Is this limitation known of the newest version? Or should work for 3 or more substreams?

Thank you

Current Behavior

Simulcacst not working for 3 substreams.

Expected Behavior

Should work with 3 substreams.

Anything else?

No response

[karasusan]

What parameters did you tried? Is the video resolution possible cause?

[avicarpio]

h: 1080 30FPS 8Mbps
m: 720 30FPS 5Mbps
l: 360 30FPS 1Mbps

The resolutions starts from 1080p and we downscale to get the others. The highest quality is the one that doesn't work. With 2 qualities it works perfectly.

This is the code if you want to test it:

float videoBitrate = 8;

private IEnumerator addTransceiverSimulcast(List<MediaStreamTrack> mediaStreamTracks)
    {
        RTCRtpTransceiverInit setTransceiver = new RTCRtpTransceiverInit();

        setTransceiver.direction = RTCRtpTransceiverDirection.SendOnly;

        List<RTCRtpEncodingParameters> parameters = new List<RTCRtpEncodingParameters>();

        RTCRtpEncodingParameters encoder = new RTCRtpEncodingParameters();

        encoder.rid = "h";
        encoder.active = true;
        encoder.maxBitrate = ConvertToBits(videoBitrate);
        encoder.minBitrate = ConvertToBits(videoBitrate - 0.5f);
        encoder.maxFramerate = videoFramerate;
        encoder.scaleResolutionDownBy = 1;

        parameters.Add(encoder);

        encoder = new RTCRtpEncodingParameters();

        encoder.rid = "m";
        encoder.active = true;
        encoder.maxBitrate = ConvertToBits(videoBitrate / 1.5f);
        encoder.minBitrate = ConvertToBits(videoBitrate / 1.5f - 0.5f);
        encoder.maxFramerate = videoFramerate;
        encoder.scaleResolutionDownBy = 1.5f;

        parameters.Add(encoder);

        encoder = new RTCRtpEncodingParameters();

        encoder.rid = "l";
        encoder.active = true;
        encoder.maxBitrate = ConvertToBits(videoBitrate / 8);
        encoder.minBitrate = ConvertToBits(videoBitrate / 8 - 0.5f);
        encoder.maxFramerate = videoFramerate;
        encoder.scaleResolutionDownBy = 3f;

        parameters.Add(encoder);

        setTransceiver.sendEncodings = parameters.ToArray();

        while (publishPeerConnection == null)
        {
            yield return 0;
        }

        MediaStreamTrack videoTrack = null;
        MediaStreamTrack audioTrack = null;

        foreach (var streamTrack in mediaStreamTracks)
        {
            if (streamTrack.Kind == TrackKind.Video)
            {
                videoTrack = streamTrack;
            }
            if (streamTrack.Kind == TrackKind.Audio)
            {
                audioTrack = streamTrack;
            }
        }

        if (videoTrack != null && audioTrack != null)
        {
            var transceiver = publishPeerConnection.AddTransceiver(videoTrack, setTransceiver);
            publishPeerConnection.AddTransceiver(audioTrack);

            // Get all available video codecs.
            var codecs = RTCRtpSender.GetCapabilities(TrackKind.Video).codecs;

            // Filter codecs.
            var h264Codecs = codecs.Where(codec => codec.mimeType == "video/H264");

            var error = transceiver.SetCodecPreferences(h264Codecs.ToArray());
            if (error != RTCErrorType.None)
                Debug.LogError("SetCodecPreferences failed | " + error);
        }
    }

[avicarpio]

Even with the 2 substreams, sometimes the highest quality stops encoding and only the lowest quality is available. However, the gpu load is always the same, and need to say that it is far from the top load:

A Tesla T4 should be able by far to encode a 1080p 8Mbps and 360p 1Mbps streams at the same time.

[avicarpio]

After a week testing this, I can confirm that simulcast with 1080p quality does not work correctly. It is unstable or even doesn't start the stream. If I set the quality to 720p it is more stable, but the quality is not acceptable for our users nowadays. In fact, it does not make sense to have simulcast with low quality streams because everyone can consume it even with bad internet, where simulcast make sense is when having high quality 1080p/4K for top tier computers/bandwith and lower versions (720p and/or 360p) for the rest of the users with medium/low-end pc/bandwith.

Thank you @karasusan

[karasusan]

@avicarpio Thanks for the clarification.

[karasusan]

@avicarpio I have a question. Can you share me how you test the simulcast?

[karasusan]

@aet Have you got a same issue when using the simulcast?

[aet]

It sounds like with simulcast it's going over the total bitrate budget for the PC and stops sending video

In our use case, we decided to expose SetBitRate method to PeerConnections and do a little fine-tuning to fix it and ensure there's some overhead, https://github.com/aet/com.unity.webrtc/commit/11ed3463b59f42aa3c62fdb78d7a56fcd17b6a26

It's a nice interface to have, but could also try to override the bitrate settings internally for the user by default

[karasusan]

@aet Thank you for your advice! This API looks useful to me.

I read a comment in libwebrtc.

  // SetBitrate limits the bandwidth allocated for all RTP streams sent by
  // this PeerConnection. Other limitations might affect these limits and
  // are respected (for example "b=AS" in SDP).
  //
  // Setting `current_bitrate_bps` will reset the current bitrate estimate
  // to the provided value.
  virtual RTCError SetBitrate(const BitrateSettings& bitrate) = 0;

I think some developers want this API because it looks easier than RTCRtpSender.SetParameters.