[BUG]: Parameters.encodings.count == 0

[avicarpio]

Package version

3.0.0-pre.1

Environment

* OS: Ubuntu 22.04
* Unity version: 2020.3.35f1

Steps To Reproduce

1. Create a Peer Connection
2. Add Tracks (I add 1 video and 1 audio)
3. Get Transceivers of Peer Connection
4. Set the codec
5. Get the Sender.First() of the Peer Connection
6. GetParameters of the sender
7. Set some params like bitrate parameters.encodings[0].maxBitrate = 3000; parameters.encodings[0].minBitrate = 1000;

Current Behavior

IndexOutOfRangeException: Index was outside the bounds of the array.

Unity Crashes because I am attempting to change an empty encodings. If I look at parameters.encondings.Length, what I get is 0.

Expected Behavior

If you look at the documentation and the samples, this should be initialized to be able to change the config. If not, can you explain me or add to the documentation, how can we create an encoding parameter?

I am following this example: https://docs.unity3d.com/Packages/com.unity.webrtc@3.0/manual/videostreaming.html#bitrate-control

Anything else?

Complete function that generates the sdp to create the offer.

public IEnumerator GenerateSdp(List<MediaStreamTrack> mediaStreamTracks)
  {        
      foreach (var mediaStreamTrack in mediaStreamTracks)
      {
          publishPeerConnection.AddTrack(mediaStreamTrack);
      }

      // Get all available video codecs.
      RTCRtpCodecCapability[] codecs = RTCRtpSender.GetCapabilities(TrackKind.Video).codecs;

      // Filter codecs.
      IEnumerable<RTCRtpCodecCapability> h264Codecs = codecs.Where(codec => codec.mimeType == "video/H264");

      RTCRtpTransceiver transceiver = publishPeerConnection.GetTransceivers().ToArray()[0];

      RTCErrorType error = transceiver.SetCodecPreferences(h264Codecs.ToArray());
      if (error != RTCErrorType.None)
          Debug.LogError("SetCodecPreferences failed");

      RTCRtpSender sender = publishPeerConnection.GetSenders().First();
      RTCRtpSendParameters parameters = sender.GetParameters();

      parameters.encodings[0].maxBitrate = 3000;
      parameters.encodings[0].minBitrate = 1000;

      RTCError errorParam = sender.SetParameters(parameters);
      if (errorParam.errorType != RTCErrorType.None)
      {
          Debug.LogErrorFormat("RTCRtpSender.SetParameters failed {0}", errorParam.errorType);
      }

      // Create the SDP.
      var op = publishPeerConnection.CreateOffer();
      yield return op;

      Debug.Log("opCreateOffer.IsError " + op.IsError);
  }

Also, I want to combine this with a simple H264 codec, not NvCodec is needed. Is it possible to use the simpliest Constrained Baseline (level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f) with this config? How can I do it?

Thank you

[avicarpio]

I solved the params.encodings problem. It was a sync problem. It's not possible to modify the encoding config before an Offer/Answer is completed. It would be great if you can add this to the documentation.

But, my problem now is what I said about H264. How can I use some H264 with another level like 1f?

• 42001f: level (1f) 3.1 (720p), profile 42 (AVC baseline)
• 420033: level (33) 5.1 (4K), profile 42 (AVC baseline)

Level 33 is too advanced for the most of current WebRTC Servers.

Thank you

[karasusan]

@avicarpio In my understanding that changing H264 profile level 5.1 to 3.1 wouldn't effects performance. I would like to know the evidence about that.

[karasusan]

@avicarpio Could you reply me if I am misunderstanding your issue? I would like to close the issue if no response.

[avicarpio]

Hi, sorry that I didn't update it. I was able to fix it. My problem was that I thought that I could change the parameters before sending video, but it can only be changed on realtime. So the correct code is:

        var sender = publishPeerConnection.GetSenders().First();

        // Get `RTCRtpSendParameters`
        var parameters = sender.GetParameters();

        // Changing bitrate of all encoders.
        foreach (var encoding in parameters.encodings)
        {
            encoding.active = true;
            encoding.maxBitrate = 6024000;
            encoding.minBitrate = 1024000;
            encoding.maxFramerate = 30;
            encoding.scaleResolutionDownBy = 1.5f;
        }

        // Set updated parameters.
        sender.SetParameters(parameters);

BUT it has to be done when initializing the webrtc streaming, not like the codecs configuration, which have to be configured before the init. That was my misunderstanding, I explain it in case someone is looking for the same problem.

Thank you @karasusan

Edit: About the H264 problem, it was a memory leak with WebRTC.Update() coroutine, also a misunderstanding about how to use it 😅. But I've fixed that too.

[tgerth]

Hi, sorry that I didn't update it. I was able to fix it. My problem was that I thought that I could change the parameters before sending video, but it can only be changed on realtime. So the correct code is:

        var sender = publishPeerConnection.GetSenders().First();

        // Get `RTCRtpSendParameters`
        var parameters = sender.GetParameters();

        // Changing bitrate of all encoders.
        foreach (var encoding in parameters.encodings)
        {
            encoding.active = true;
            encoding.maxBitrate = 6024000;
            encoding.minBitrate = 1024000;
            encoding.maxFramerate = 30;
            encoding.scaleResolutionDownBy = 1.5f;
        }

        // Set updated parameters.
        sender.SetParameters(parameters);

BUT it has to be done when initializing the webrtc streaming, not like the codecs configuration, which have to be configured before the init. That was my misunderstanding, I explain it in case someone is looking for the same problem.

Thank you @karasusan

Edit: About the H264 problem, it was a memory leak with WebRTC.Update() coroutine, also a misunderstanding about how to use it 😅. But I've fixed that too.

Hi :) Now I have exactly the same problem, but I didn't really understand your "solution". Where or when do I need to set the parameters? Even though it's not at the beginning can I still access the updated parameters in the onTrack function on the receiver side?

[tgerth]

Hi, sorry that I didn't update it. I was able to fix it. My problem was that I thought that I could change the parameters before sending video, but it can only be changed on realtime. So the correct code is:

        var sender = publishPeerConnection.GetSenders().First();

        // Get `RTCRtpSendParameters`
        var parameters = sender.GetParameters();

        // Changing bitrate of all encoders.
        foreach (var encoding in parameters.encodings)
        {
            encoding.active = true;
            encoding.maxBitrate = 6024000;
            encoding.minBitrate = 1024000;
            encoding.maxFramerate = 30;
            encoding.scaleResolutionDownBy = 1.5f;
        }

        // Set updated parameters.
        sender.SetParameters(parameters);

BUT it has to be done when initializing the webrtc streaming, not like the codecs configuration, which have to be configured before the init. That was my misunderstanding, I explain it in case someone is looking for the same problem. Thank you @karasusan Edit: About the H264 problem, it was a memory leak with WebRTC.Update() coroutine, also a misunderstanding about how to use it 😅. But I've fixed that too.

Hi :) Now I have exactly the same problem, but I didn't really understand your "solution". Where or when do I need to set the parameters? Even though it's not at the beginning can I still access the updated parameters in the onTrack function on the receiver side?

@avicarpio

[avicarpio]

Can you give more info? Did you follow the documentation? https://docs.unity3d.com/Packages/com.unity.webrtc@3.0/manual/videostreaming.html

[tgerth]

Yes in general I'm following their documentation, but I'm not sure what they mean with "Developers can control properties about video streaming quality in real-time." But yeah I have two problems/questions which are probably correlated:

1. I'm not sure when to set the encodings in the sending parameters. I noticed, that only after setting the local description the encodings array is not empty but filled with default values like 0 or "". I tried to set the parameters before and after setting the local description but calling GetParameters() right after never ended up with those parameters I set. Maybe there is a specific place where I should call SetParameters(..) or I need to do the offer in a specific way, I don't know.

2. When I want to see the encodings in the onTrack function on the receiver side this array is always empty. I'm calling e.Transceiver.Sender.GetParameters().encodings where e is the RTCTrackEvent. What am I doing wrong?

So in general I just want to set the rid value in order to have a string attached to a videostream to identify the stream. Can you help me? @avicarpio

[avicarpio]

I think that you are trying to edit this parameters before the offer/answer is completed. Try to do this communication and send/receive the video without setting any parameters, then, when your are sure that the data is flowing, try to read this parameters (you can use the code I pasted), and not before. What I mean with "realtime" is that the data is actually flowing, so you will have to use first the default parameters (for example 3Mb/s bitrate), and then change it to your needs (for example bitrate 500kb/s).

I hope this works!

[tgerth]

@avicarpio Ah ok I see and how would you then read the updated values on the receiver side?