Having issues receiving screenshare-video content (profile-related?)

[SomeGuy87]

As suggested in the mailing list, posting this as an issue here:

I have a mediasoup client that normally streams the webcam. When using this mode, everything is working fine. However, I also have a function that switches the stream to a screensharing instead, streaming the desktop in high resolution.

I do the change by doing the following (can't use replaceTrack since I have to support an Android client as well that doesn't support it):

videoProducer.close(); audioProducer.close(); audioProducer = room.createProducer(stream.getAudioTracks()[0],{simulcast: false}, { source: 'mic' }); (had null instead of the simulcast thing before, was for tests) videoProducer = room.createProducer(stream.getVideoTracks()[0],{simulcast: false}, { source: 'webcam' }); stopTracks(stream); audioProducer.send.... videoProducer.send...

This works as expected when I am already in the session. However, if I join the session with another client while this stream is already active, I do not get any video contents. It works perfectly fine when the stream is still sharing the webcam though, using the same logic to switch the streams. If I continuously disable and enable the stream, I usually get a "effectiveConsumerProfileChanged" to default at some point, after which I get the screen as expected. So I am highly suspecting that I am somehow registering for a wrong profile or did something that causes the profile to be wrong, so that I don't get anything. I'm actually confused why I do have different profiles in the first place, I thought this was only for simulcast. Do I need to disable this differently somehow?

So summarized, the situation is: Client 1: Starts room, sends webcam Client 2: Joins room, receives webcam Client 1: Switches to desktop stream by closing consumer and creating new one Client 2: Receives desktop stream Client 3: Joins, does not receive desktop stream Client 1: Enables and disables stream multiple times Client 3: Gets effectiveConsumerProfileChanged to "default" at some point, receives stream

Screenshot of the whole websocket communication - until the last message, I did not see the video: https://imgur.com/a/2G03Ali

On my Android client, I also have this issue when I already am part of the session, so I'm wondering if there is some sort of option I am missing in general, causing the issues on both sides.

Thanks in advance for any sort of help!

[SomeGuy87]

Probably important and I completely forgot to mention and check: The testing environment was based on version 2.0.9 of the server (current default of the demo server) and 2.0.16/2.1.1 on the client.

I wanted to quickly check if updating the server-side helps with the issue since I wasn't aware it had an older version, but my npm install keeps getting stuck on a Worker.o file, so I currently cannot comment if the changes of 2.0.10 and 2.0.11 help me with this issue unfortunately.

[SomeGuy87]

Finally able to test with 2.1.0 server as well: Still the same behavior.

[SomeGuy87]

Not sure if this is helpful in any way, but it seems like I can kind of force it to work again by changing the video track constraints back and forth whenever another peer joins (or in case of my Android client when the stream is switched initially). So I assume that some sort of condition is not valid that sets the initial effective profile for a consumer? Perhaps missing a keyframe or something. Additionally it seems to be an issue exlusively with desktop streams of getUserMedia, even when starting the calls with desktop sharing the issue applies - so without ever changing the producer and just starting a call normally. Still not sure what makes them so different from webcam streams though.

Snippet of the workaround done whenever a new peer arrives that somehow triggers what's needed:

setTimeout(function() {
   let oldConstraints = {
       width: {max: videoProducer.track.getConstraints().width.max},
       height: {max: videoProducer.track.getConstraints().height.max}
   }
   let newConstraints = {
       width: {max: 320},
       height: {max: 240}
   }

   videoProducer.track.applyConstraints(newConstraints)
   .then(() => {
       setTimeout(function() {
        videoProducer.track.applyConstraints(oldConstraints);
        }, 1500);
    });
}, 1500);

[newinnovations]

Just to inform you that I am having the same issue here. Receiving the desktop stream while in the session works fine, receiving the stream as part of joining the session does not. I am currently in the process of converting from mediasoup v1 to the client based v2. In v1 the desktop shares worked fine.

Have not tried the workaround yet.

[ibc]

I've just tested with Chrome and Android (both Chrome for Android and react-native-webrtc) and it works fine. I mean: both Chrome for Android and the react-native app receive the screen sharing no matter it started before or after they joined the room. Always. No error.

Are you sure you are properly iterating and enabling all the consumers of all the peers at the time the client joins the room? Also, I assume the desktop browser is Chrome, right?

[SomeGuy87]

Are you sure you are properly iterating and enabling all the consumers of all the peers at the time the client joins the room?

Yes, 100% sure. As I mentioned, I can also make it work again by either enabling an disabling the stream multiple times, or by changing its resolution back and forth after the joining client is registered as a consumer. After adding the snippet from above whenever "newPeer" is triggered, the stream immediately appeared again for the joining participant. Hence I had the assumption that it might be related to it waiting for the next key frame that it misses in our cases or something along those lines (racing condition? Hard to imagine why it works for you otherwise).

I also debugged on my Android client and can confirm that I am receiving video content through webrtc, but the resolution is always 0:0 until the effectiveProfileChanged comes in.

Also, I assume the desktop browser is Chrome, right?

Yes, Chrome 67 with getUserMedia "desktop".

[ibc]

When a new Consumer is enabled (or resumed if it was paused) mediasoup requests a video keyframe to the corresponding Producer (in your case, the screen sharer). Before such a keyframe arrives to mediasoup, the Consumer effectiveProfile is "none".

• Can you confirm that, when you do NOT receive the screen track, the consumer.effectiveProfile is "none"?

If so, something weird happens. A way to debug it is by enabling logTags: "rtcp" [*] in mediasoup server and then check the logs when the Consumer is enabled. You should see some logs requesting a keyframe and other when such a keyframe is received by mediasoup so the Consumer`s profile changes from "none" to another value.

[*] https://mediasoup.org/documentation/mediasoup/debugging/#log-tags

[SomeGuy87]

Can you confirm that, when you do NOT receive the screen track, the consumer.effectiveProfile is "none"?

It indeed seems like it is still on null ( so not a "none" string). Debugged on both the opening and the close event, and while the audio stream went from null to "default" between both, the video stream was still on null. Only the preferredProfile updated itself to "low". I can try to do the server-side debugging later.

[SomeGuy87]

Hopefully I did this right - if I see it correctly, it never reacts due to a flood protection? Logs starting the moment the problematic client connects, two participants were in there before for whom things worked fine.

https://pastebin.com/5wj8kHkZ

Also one log from start to finish in case it helps in any way (room created, participant joins, third participant joins who has the issue): https://pastebin.com/ULnQ13DT

[ibc]

Can you please repeat the capture with just the screen sharing producer and the later consumer? Otherwise is too hard to debug.

[newinnovations]

Maybe this is helpful. When I share a single window, and someone enters the session the desktop video does not show. As soon as I resize the window the video appears. Moving the window does nothing and activity in the window does either.

[SomeGuy87]

Hopefully this is the same case, it seemed like the issue didn't happen too often when just using two participants (might be because they pretty much start simultaneously):

https://pastebin.com/eVS9xkZi

Peer "2" should have been the one not receiving the stream.

[jmillan]

@ChrisE87,

In the logs you've pointed there's usually the following log in the latest lines:

mediasoup-demo-server:INFO:Room Consumer "effectiveprofilechange" event [profile:default]

Does the new participant start receiving the Desktop video once after this log?

[SomeGuy87]

Does the new participant start receiving the Desktop video once after this log?

Yes, this is always the point where it works normally again. I'm not quite sure if it was the specific line in my logs though since I was already looking at another computer to pull the logs, so it might also be the profile from another stream. Whenever it works again that line appears 100% guaranteed though.

[jmillan]

Whenever it works again that line appears 100% guaranteed though.

Looking at the logs it seems that the delay between creating the transport for the new consumer and receiving a full frame request from the Desktop sharer producer is 1 second approximately. Is this de delay you are experiencing?

[SomeGuy87]

Is this de delay you are experiencing?

No, it goes more into the range of 1-2 minutes unless I actively force it somehow. In my screenshot from the top post it was roughly 15 seconds, but I think I forced it in that case by enabling and disabling the stream erratically.

[ibc]

@newinnovations said:

Maybe this is helpful. When I share a single window, and someone enters the session the desktop video does not show. As soon as I resize the window the video appears.

Which window do you resize? the sender's window? or the receiver's window? I assume it's the producer's window, right?

If so, it makes sense that, when the shared window is resized, the sender encodes a keyframe that, somehow, reaches the receiver (the consumer) while the keyframe requested by mediasoup in behalf of the consumer (when it is connected) is not reaching the receiver. No idea why right now.

[newinnovations]

Which window do you resize? the sender's window? or the receiver's window? I assume it's the producer's window, right?

That is correct. It is the window that is being shared by the chrome desktopCapture.

[newinnovations]

Changing the constraints back and forth when a peer joins the session as suggested by @ChrisE87 works for me. Probably this is similar in behavior for chrome to changing the size of the shared window.

[ibc]

Most probably, changing the constraints produces a keyframe. The issue here is why the keyframe did not reach the Consumer when it connected.