Description

I'm experiencing a harsh delay of around 20 to 30 seconds on receiver side when calling replaceTrack(oldTrack, newTrack)after creating a local desktop track; interestingly, when changing back from desktop to camera the switch happens in an instant. Im fighting with this issue since 2 weeks now and am kind of lost, so any help or hint would be greatly appreciated 🙏 ❤️

Current behavior

The event events.track.TRACK_VIDEOTYPE_CHANGED is fired instantly on the receiver side, but the desktop stream is not there yet. After 20 - 30 seconds the desktop stream becomes visible; I also checked videoType as well as the track and stream, they seem to be correctly assigned directly after the event is received, just the image itself is not updated until after these 20 seconds.

I'm using the current lib-jitsi-meet (also tried with your cdn version, same behavior).
Tried with our install (latest jitsi) as well as meet.jit.si as a backend. Same behavior.

The code to create the desktop track looks like this:

const createDesktopTrack = (jsmeet) => {
    jsmeet.createLocalTracks({ devices: [ 'desktop' ] })
    .then((tracks) => setTracks(tracks))
    .catch(error => {
        console.log("Error Message", error)
    });
}

const setTracks = (tracks) => {
  const newTrack = tracks[0] // get desktop track

   // add event listener to reset to camera if screenshare is stopped
  if(newTrack.videoType === 'desktop') {
    newTrack.addEventListener(
        window.JitsiMeetJS?.events.track.LOCAL_TRACK_STOPPED,() => createVideoTrack(jsMeet)
    )
  }

  const oldTrack = conference.getLocalVideoTrack()
  if(oldTrack) {
       conference.replaceTrack(oldTrack, newTrack)
        .then(()=>{
            replaceLocalTrack(newTrack)
            conference.removeTrack(oldTrack)
            oldTrack.dispose()  // I think this is already happening in replaceTrack if I saw correclty? 
        })
}

I exped that on receiver sice the active video signal/stream would be simply switched ?
I listen to TRACK_VIDEOTYPE_CHANGED which is correctly and instantanously fired.

track.addEventListener(JitsiMeetJS?.events.track.TRACK_VIDEOTYPE_CHANGED, (e)=>_onVideoTypeChanged(e, id))

const _onVideoTypeChanged = (type:string, id) => produce(newState => {
    user.videoType = type === "desktop" ? "desktop" : "camera" // set videoType directly

    // when adding this update to make sure it's really the latest video track the actual stream still has a 20 sec timeout till desktop is shown
    // user.video = user.user._tracks[1] 
  })

Expected Behavior

I would assume that when events.track.TRACK_VIDEOTYPE_CHANGEDis received, the actual video signal on the receiver side should be updated to the corresponding new media stream

Possible Solution

I really don't know, thought it yould be a server config but since the behavior is similar with meet.jit.si as backend I hope for an helpful tip.

Steps to reproduce

Environment details

latest lib-jitsi-meet from https://meet.jit.si/libs/lib-jitsi-meet.min.js Chrome - 98.0.4758.102

NewEraCracker commented 2 years ago

If you search the minified bundle for {version:" you'll find it is built currently from commit bafd6c7b (Feb 3, 2022) so it lacks the latest bug fixes.

I'd recommend you build latest lib-jitsi-meet version from source and try with that.

dkgrieshammer commented 2 years ago

If you search the minified bundle for {version:" you'll find it is built currently from commit bafd6c7 (Feb 3, 2022) so it lacks the latest bug fixes.

I'd recommend you build latest lib-jitsi-meet version from source and try with that.

So, I just tried using the example in docs folder with a freshly comiled lib-jitsi-meet; remarkably the example doesn't use replaceTrackto switch to desktop but is using dispose() and room.addTrack() - any ideas why that ii?

So I duplicated the switchVideo function in the example to use replaceTrack and I think I found the issue - without explicit call to oldTrack.dispose() the replaceTrack() function takes around the reported 20 - 30 seconds to switch

function switchVideo() { // eslint-disable-line no-unused-vars
    isVideo = !isVideo;
    let oldTrack;

    if (localTracks[1]) {
        oldTrack = localTracks[1];

        // localTracks[1].dispose();
        localTracks.pop();
    }

    JitsiMeetJS.createLocalTracks({
        devices: [ isVideo ? 'video' : 'desktop' ]
    })
        .then(tracks => {
            localTracks.push(tracks[0]);

            // Event added after its in localTracks but should still be same object
            localTracks[1].addEventListener(
                JitsiMeetJS.events.track.TRACK_MUTE_CHANGED,
                () => console.log('local track muted'));
            localTracks[1].addEventListener(
                JitsiMeetJS.events.track.LOCAL_TRACK_STOPPED,
                () => console.log('local track stoped'));
            localTracks[1].attach($('#localVideo1')[0]);

            if (oldTrack) {
                room.replaceTrack(oldTrack, localTracks[1]);
            } else {
                room.addTrack(localTracks[1]);
            }
        })
        .catch(error => console.log(error));

// if oldTrack is not disposed explicitely, the reported delay occurs; TODO: PR on replaceTrack() lib documentation to mention  
    // oldTrack?.dispose();
}

Jacky-Rolo commented 2 years ago

Im fighting with this issue since 2 weeks ...

For now, a word around would be use removeTrack + addTrack (to get new track) instead of replaceTrack and run a infinite while loop to check videoType

await JitsiConference.removeTrack(oldTrack)
await sleep(100) // sleep 100ms to trigger "TRACK_ADDED" event
await JitsiConference.addTrack(newTrack)

// and
while(true) {
  await sleep(800)

  const participants = JitsiConference.getParticipants();

  // then find all video tracks from participants
  // compare track type, if it's different with your application state, then update state
}

Use removeTrack + addTrack (instead of replaceTrack) will have some cost, see this issue to know more detail

https://github.com/jitsi/lib-jitsi-meet/issues/1649

Unfortunately, issue 1649 did NOT fully fixed, so we still need while loop to check correct video type (I will ask re open that issue)

hope this is helpful for you

Edit1: I tested Jitsi-meet, DONT have video freeze issue, dont know what's trick they use

maxired commented 2 years ago

I personally face similar issues with sometimes a screen share being slow to be visible to remote participants. One thing I noticed is the bandwidth estimator is slow to ramp up. This is visible by looking at the JVB logs. Not saying this is your issue here, but that might be worth checking

jitsi / lib-jitsi-meet

20 Second Delay on replaceTrack() #1931