t-mullen / video-stream-merger

Merge multiple video MediaStreams into one composite.
https://t-mullen.github.io/video-stream-merger/
MIT License
357 stars 81 forks source link

Stream stops if the tab is minimized #27

Closed rohitshetty closed 5 years ago

rohitshetty commented 5 years ago

I use screen and media streams and merge them using this library. I use webrtc recorder to record the output stream and download it. Everything works as expected, except when the browser is minimized, the video stream pauses (although audio stream continues) till the browser is minimized, and continues once it's maximized. My initial suspect was issue #9, so I disabled/enabled the chrome flag and tried, it didn't help. To zero in on the source, I just recorded the screen stream, they work fine, even if the browser is minimized, I recorded only the video stream and tested it works fine. The only issue is when I am saving the merged feed.

Please check the code source attached below.

rohitshetty commented 5 years ago

index.html

<button onclick="startCapture()">Start</button>
<button id="stop">stop</button>
<select id="param">
  <option value="0">0</option>
  <option value="0.1">0.1</option>
  <option value="0.2">0.2</option>
  <option value="0.3">0.3</option>
  <option value="0.4">0.4</option>
  <option value="0.5" selected="selected">0.5</option>
  <option value="0.6">0.6</option>
  <option value="0.7">0.7</option>
  <option value="0.8">0.8</option>
  <option value="0.9">0.9</option>
  <option value="1.0">1</option>
</select>
<h2 id="msg">wait</h2>
<video
  playsinline
  id="video"
  autoplay
  controls
  style="object-fit: contain; width: 40%;"
></video>

<script src="./video-merge.js"></script>
<script src="https://cdn.WebRTC-Experiment.com/RecordRTC.js"></script>

<script>
  let videoElem = document.getElementById("video");

  async function startCapture() {
    var CANVAS_MULTIPLIER = 5;
    try {
      var screen = await navigator.mediaDevices.getDisplayMedia({
        video: true
      });
      console.log(screen);

      var cam = await navigator.mediaDevices.getUserMedia({
        audio: true,
        video: true
      });

      const select = document.getElementById("param");
      const param = select.options[select.selectedIndex].value;

      console.log(2 - param);
      var merger = new VideoStreamMerger({
        width: window.screen.width / (2 - param),
        height: window.screen.height / (2 - param)
      });

      setTimeout(function() {
        console.log("trigged");
        merger.addStream(screen, {
          x: 0, // position of the topleft corner
          y: 0,
          width: merger.width,
          height: merger.height,
          mute: true // we don't want sound from the screen (if there is any)
        });
        console.log("trigged2");

        merger.addStream(cam, {
          x: 0,
          y: merger.height - 0.2 * merger.width,
          width: 0.2 * merger.width,
          height: 0.2 * merger.width,
          mute: false
          // draw: function (ctx, frame, done) {
          //   x = 0; y = merger.height - 0.2 * merger.width; width = 0.2 * merger.width; height = 0.2 * merger.width;
          //   // decide where you want your circular image, and what size

          //   ctx.save(); // save canvas context
          //   ctx.beginPath();
          //   ctx.arc(x + width/2, y + height/2, width/2, 0, Math.PI*2, true);   // create an circle centered around frame
          //   ctx.closePath();
          //   ctx.clip(); // clip the context to the circle

          //   ctx.drawImage(frame, x, y, width, height); // draw the image in the clipped context
          //   ctx.restore(); // restore canvas context so you don't clip every other stream
          //   done() // <- you must call this so the merger can continue
          // }
        });

        merger.start();

        // We now have a merged MediaStream!
        // videoElem.srcObject = cam;
        // videoElem.srcObject = screen;

        videoElem.srcObject = merger.result;

        document.getElementById("msg").innerHTML = "Start";
        // replace merger.result with cam or screen
        var recorder = new RecordRTCPromisesHandler(merger.result, {
          type: "video"
        });

        recorder.startRecording();
        var stop = document.getElementById("stop");
        stop.onclick = async () => {
          console.log("stopping recording");
          await recorder.stopRecording();
          let blob = await recorder.getBlob();
          invokeSaveAsDialog(blob);
        };
      }, 1000);
    } catch (err) {
      console.error("Error: " + err);
    }
  }
</script>

and place the latest version of this library in the same folder as video-merge.js to change the sources being recorded, check the line numbers 89-91 (uncomment as required) and line number 95

Tryptophan commented 5 years ago

It has to do with the requestAnimationFrame call when merging a stream, the canvas is throttled when the tab is no longer focused. There currently isn't a way around this, other than maybe https://developer.mozilla.org/en-US/docs/Web/API/OffscreenCanvas. I would look into changing video-stream-merger's code to use that to fix this.

rohitshetty commented 5 years ago

I see. It works fine when the screen is out of focus, that is when I drag another window over the browser. This freezing of stream only happens when the browser is explicitly minimized. Do you think it is consistent with throttling?

Tryptophan commented 5 years ago

Maybe, haven't seen that before.

t-mullen commented 5 years ago

I tested this, it looks like the hack of playing audio to prevent throttling has been fixed in the latest Chrome. Tabbing out also causes the issue.

I'll have to look at what Spotify is doing now to avoid throttling.

t-mullen commented 5 years ago

3.2.1 use setTimeout when the page is hidden. It's not as resource efficient, but the stream will continue merging.

Timebutt commented 5 years ago

@t-mullen I am still experiencing this issue. Having a look at the fix you used, my problem seems fixed when I rewrite _requestAnimationFrame() to this:

// Wrapper around requestAnimationFrame and setTimeout to avoid background throttling
VideoStreamMerger.prototype._requestAnimationFrame = function (callback) {
    setTimeout(callback, 0);
}

Weird how I don't need an explicit requestAnimationFrame() but it just works. I don't know if the latest version of Chrome somehow got more strict, essentially dismantling your fix or if my specific scenario is different from what was originally reported. Could you have a look if what I'm saying makes sense?

t-mullen commented 5 years ago

Chrome might be throttling the minimized tab so much that document.hidden isn't even being run once to switch to the fallback. I'll change this so that we're checking minimization in a setTimeout timer and never relying on a requestAnimationFrame timer returning.

Just using setTimeout works, but requestAnimationFrame has some nice optimizations for canvas that let us get a more stable framerate.

Timebutt commented 5 years ago

Awesome, sounds like a great fix. I'm relying on just the setTimeout through patch-package in my current application but will move to your solution once implemented. Thanks for this!

Tryptophan commented 5 years ago

@Timebutt How are you using setTimeout(callback, 0) without completely blocking the main thread? I've tried this on a local install and the browser seems to crawl to a halt when using this since the callback is being called very quickly repeatedly.

Timebutt commented 5 years ago

@Tryptophan you are absolutely right. While it does work when you alt-tab, it fully loads the main thread. I'm on a powerful 6 core machine and didn't even notice it does, trying this again on a less beefy machine is a whole different story.

Thanks for pointing this out, looks like we'll need the solution @t-mullen is suggesting: using the setTimeout() to do the minimisation check and continue to use requestAnimationFrame(). Any word when you might have a chance to fix this?

t-mullen commented 5 years ago

Still working on a fix. The visibility API seems to lie about tab visibility when Chrome switches tabs...

t-mullen commented 5 years ago

It seems my audio hack no longer works. Spotify is still able to maintain intervals when it is playing... let's hope Chrome hasn't implemented a domain whitelist for timers.

t-mullen commented 5 years ago

Should be fixed in v3.3.1. I expect this issue with budget throttling to come up again, but it works for now.

Tryptophan commented 5 years ago

Just so you're aware, I think setTimeout and setInterval are still throttled in background tabs: https://stackoverflow.com/questions/6032429/chrome-timeouts-interval-suspended-in-background-tabs

t-mullen commented 5 years ago

They ARE supposed to be throttled, but you can violate the budgeting policy by playing audio - which is what we do in this library. It's a hack, but it's the only way to get solid framerates in background tabs.

Timebutt commented 5 years ago

Thanks for this @t-mullen, I'll have a look at how you fixed it and verify if performance is back where it's supposed to be.

FYI: I also had issues with the Visibility API in another problem I was fixing, it has to do with the OS you are on among others it seems.

xiang-valcano commented 4 years ago

I am not sure that the solution has covered the case where OSX users totally minimize (- button) chrome application and the render stop working.

I have solved this long time ago, but I cant remember the actual thing I did. I used some kinds of Worker running in background to call the drawing function. And the drawing function keeps signaling the worker for settimeout. So the render is running smoothly according to the fps without interuption.

Basically, its a settimeout worker running in background. U can also use this worker for other features to trigger a function in the background.

mustafa-toptal commented 2 years ago

This is issue still persist in Safari, seems to work fine in chrome

Ivanca commented 1 year ago

This is issue still persist in Safari, seems to work fine in chrome

The solution is to tell your users to stop using Safari, if you can convince them to do it that would also help many other websites as well; due the millions of other safari-bugs that Apple refuses to address, at this point Safari is the new IE6