[1.x] Issues with Subscribing to Simulcast Stream in a Video Room

[mail2mhossain]

I am currently utilizing the 'SipSorcery' library, a .Net Core webRTC solution. However, I'm facing difficulties since this library does not support simulcast. When a device publishes video in a video room without simulcast, I am able to subscribe to and receive the video feed without any issue.

The problem arises when a device publishes a video with simulcast. In this case, I am not receiving any video feed. I am wondering if there are any additional parameters that I need to include in my subscription message so that Janus will provide me with the feed.

Below is my current subscription message for reference:

{
  "handle_id": 3088882397563968,
  "body": {
    "streams": [
      {
        "feed": 1453583828895316,
        "mid": "0"
      }
    ],
    "request": "join",
    "ptype": "subscriber",
    "room": 1167593334359571,
    "muted": false
  },
  "session_id": 5988193796016929,
  "janus": "message",
  "transaction": "6049587d-90b1-42c5-89aa-772f691bcc35"
}

When the 'attached' event is triggered, I receive the following message:

{
   "janus": "event",
   "session_id": 5988193796016929,
   "transaction": "6049587d-90b1-42c5-89aa-772f691bcc35",
   "sender": 3088882397563968,
   "plugindata": {
      "plugin": "janus.plugin.videoroom",
      "data": {
         "videoroom": "attached",
         "room": 1167593334359571,
         "streams": [
            {
               "type": "video",
               "active": true,
               "mindex": 0,
               "mid": "0",
               "ready": false,
               "send": true,
               "feed_id": 1453583828895316,
               "feed_display": "838f3560-0bfb-492b-8614-e8e07e9cd32a",
               "feed_mid": "0",
               "codec": "vp8",
               "simulcast": {
                  "substream": -1,
                  "substream-target": 2,
                  "temporal-layer": -1,
                  "temporal-layer-target": 2
               }
            }
         ]
      }
   },
   "jsep": {
      "type": "offer",
      "sdp": "I intentionally deleted this text."
   }
}

Any guidance or assistance you can provide on this matter would be greatly appreciated.

[lminiero]

The whole team is on vacation, we'll have a look when we're back in a few weeks.

[atoppi]

An offer containing streams with simulcast will have extra attributes for rid negotiation and the RTP stream will also have extensions to support that. Maybe your rtc stack is not capable of handling those extras and will either fail the negotiation or will be unable to decode the RTP stream.

From an API perspective the only thing you need to get a simulcast stream is just adding substream and temporal fields to the streams array elements in the subscribe request.

[mail2mhossain]

So, I need to add substream and temporal fields as follows:

{ "handle_id": 2034943113457475, "body": { "streams": [ { "feed": 3929900983627935, "mid": "0", "substream": 1, "substream-target": 2, "temporal-layer": -1, "temporal-layer-target": 2 } ], "request": "join", "ptype": "subscriber", "room": 2887853390818590, "display": "8801713032885", "muted": false }, "session_id": 6348774939895119, "janus": "message", "transaction": "ef60791a-6f6f-44e0-8532-86ca310496c0" }

Still did not get feed from simulcast enable device.

BTW: Say, one is publishing and subscribing video with simulcast enabled way. Another one is publishing and subscribing without simulcast.

A -> simulcast enabled device: connected to Janus server B-> simulcast disabled device (means its rtc stack do not support simulcast): connected to Janus server C-> simulcast disabled device (means its rtc stack do not support simulcast): connected to Janus server

Scenario 1: A is publishing and subscribing video and successfully getting video of B & C. Scenario 2: B is publishing and subscribing video, getting video of C BUT not getting video of A. Scenario 3: C is publishing and subscribing video, getting video of B BUT not getting video of A.

In scenario 2 or 3, can Janus send video without adding extra attributes so that device B or C can handle RTP stream without doing extra job, means without implementing simulcast support in their rtc stack? Say, Janus will always forward the highest quality video of A to B and C.

[atoppi]

Sorry for the confusion, but the extra simulcast parameters are in the publishing leg, not in the subscribing one. I'll try to be more clear.

When publishing simulcast streams, enabled stacks will add rtp extensions and additional parameters in the negotiation. When subscribing to a simulcast pub, peers just see a single rtp stream (like non-simulcast publishers) and Janus will notify the availability of different simulcast layers through the API. Users can switch simulcast layers through the same API (the parameters substream and temporal I was mentioning before). Please refer to the plugin API here. In case of missing parameters about which layer to receive, Janus just picks a default (I don't remember which one off top of my head).

About your scenarios, I tried with my browser opening 3 tabs on the mvideoroom demo, one with simulcast enabled (add "?simulcast=yes" to the url) and 2 without simulcast. Every tab was able to see the others, so an issue in the sfu is unlikely.

Some troubleshooting steps:

• update Janus to latest version and retry
• share the offer/answer for a peer that is unable to see a simulcast stream
• share the handle info (fetched through Admin API) for a handle that is unable to see a simulcast stream
• for that same handle, do a recording (mjr) of the RTP through the Admin API, convert it to a video with janus-pp-rec tools and try to view it on a player

[mail2mhossain]

Janus message flow for a peer that is unable to see a simulcast stream:

Janus_Message_Flow.txt

[atoppi]

The peer connection is being successfully established. The JSEP answer from sipsorcery looks "too short" with some parameters (often negotiated by rtc stacks) lacking, anyway I'm not sure that might be an issue.

Please try to get an unencrypted dump of the subscriber traffic through the Admin API, that will tell us if the bitstream is playable or not.

[mail2mhossain]

pcap_in_text_with_simulcast.zip

This video we can't process with SipSorcery.

[lminiero]

You need to capture the subscriber side, not the publisher side. That said, I extracted the three video streams from the pcap and, with janus-pp-rec, they all work just fine.

[mail2mhossain]

Attaching pcap_as_subscriber_of_simulcast.txt. I'll send our client tool too so that you can check. pcap_as_subscriber_of_simulcast.txt

[lminiero]

Using the same process (text2pcap, pcap2mjr, janus-pp-rec) I got a playable video out of it, which means the video is fine. The only thing I noticed is that it plays at Benny Hill speed (very fast). Not sure if it's a problem in how the video is encoded by the publisher. But apart from that, it's a working RTP video stream, which again means the problem is in SipSorcery, not Janus. Closing the issue.

[mail2mhossain]

As a simulcast publisher, we provide a range of video qualities, including low, medium, and high, to Janus. If you are a subscriber without simulcast support, which quality of video would you receive from Janus: low, medium, or high?

[lminiero]

If you are a subscriber without simulcast support

All subscribers are without simulcast support. Simulcast support is only needed on the publisher side: on the subscriber side, the VideoRoom plugin automatically only sends a single video stream, choosing one of the available ones.

[mail2mhossain]

FYI: SipSorcery only supports VP8 & H264 Video codecs.