Closed rohitshetty closed 5 years ago
index.html
<button onclick="startCapture()">Start</button>
<button id="stop">stop</button>
<select id="param">
<option value="0">0</option>
<option value="0.1">0.1</option>
<option value="0.2">0.2</option>
<option value="0.3">0.3</option>
<option value="0.4">0.4</option>
<option value="0.5" selected="selected">0.5</option>
<option value="0.6">0.6</option>
<option value="0.7">0.7</option>
<option value="0.8">0.8</option>
<option value="0.9">0.9</option>
<option value="1.0">1</option>
</select>
<h2 id="msg">wait</h2>
<video
playsinline
id="video"
autoplay
controls
style="object-fit: contain; width: 40%;"
></video>
<script src="./video-merge.js"></script>
<script src="https://cdn.WebRTC-Experiment.com/RecordRTC.js"></script>
<script>
let videoElem = document.getElementById("video");
async function startCapture() {
var CANVAS_MULTIPLIER = 5;
try {
var screen = await navigator.mediaDevices.getDisplayMedia({
video: true
});
console.log(screen);
var cam = await navigator.mediaDevices.getUserMedia({
audio: true,
video: true
});
const select = document.getElementById("param");
const param = select.options[select.selectedIndex].value;
console.log(2 - param);
var merger = new VideoStreamMerger({
width: window.screen.width / (2 - param),
height: window.screen.height / (2 - param)
});
setTimeout(function() {
console.log("trigged");
merger.addStream(screen, {
x: 0, // position of the topleft corner
y: 0,
width: merger.width,
height: merger.height,
mute: true // we don't want sound from the screen (if there is any)
});
console.log("trigged2");
merger.addStream(cam, {
x: 0,
y: merger.height - 0.2 * merger.width,
width: 0.2 * merger.width,
height: 0.2 * merger.width,
mute: false
// draw: function (ctx, frame, done) {
// x = 0; y = merger.height - 0.2 * merger.width; width = 0.2 * merger.width; height = 0.2 * merger.width;
// // decide where you want your circular image, and what size
// ctx.save(); // save canvas context
// ctx.beginPath();
// ctx.arc(x + width/2, y + height/2, width/2, 0, Math.PI*2, true); // create an circle centered around frame
// ctx.closePath();
// ctx.clip(); // clip the context to the circle
// ctx.drawImage(frame, x, y, width, height); // draw the image in the clipped context
// ctx.restore(); // restore canvas context so you don't clip every other stream
// done() // <- you must call this so the merger can continue
// }
});
merger.start();
// We now have a merged MediaStream!
// videoElem.srcObject = cam;
// videoElem.srcObject = screen;
videoElem.srcObject = merger.result;
document.getElementById("msg").innerHTML = "Start";
// replace merger.result with cam or screen
var recorder = new RecordRTCPromisesHandler(merger.result, {
type: "video"
});
recorder.startRecording();
var stop = document.getElementById("stop");
stop.onclick = async () => {
console.log("stopping recording");
await recorder.stopRecording();
let blob = await recorder.getBlob();
invokeSaveAsDialog(blob);
};
}, 1000);
} catch (err) {
console.error("Error: " + err);
}
}
</script>
and place the latest version of this library in the same folder as video-merge.js
to change the sources being recorded, check the line numbers 89-91 (uncomment as required) and line number 95
It has to do with the requestAnimationFrame
call when merging a stream, the canvas is throttled when the tab is no longer focused. There currently isn't a way around this, other than maybe https://developer.mozilla.org/en-US/docs/Web/API/OffscreenCanvas. I would look into changing video-stream-merger's code to use that to fix this.
I see. It works fine when the screen is out of focus, that is when I drag another window over the browser. This freezing of stream only happens when the browser is explicitly minimized. Do you think it is consistent with throttling?
Maybe, haven't seen that before.
I tested this, it looks like the hack of playing audio to prevent throttling has been fixed in the latest Chrome. Tabbing out also causes the issue.
I'll have to look at what Spotify is doing now to avoid throttling.
3.2.1
use setTimeout when the page is hidden. It's not as resource efficient, but the stream will continue merging.
@t-mullen I am still experiencing this issue. Having a look at the fix you used, my problem seems fixed when I rewrite _requestAnimationFrame()
to this:
// Wrapper around requestAnimationFrame and setTimeout to avoid background throttling
VideoStreamMerger.prototype._requestAnimationFrame = function (callback) {
setTimeout(callback, 0);
}
Weird how I don't need an explicit requestAnimationFrame()
but it just works. I don't know if the latest version of Chrome somehow got more strict, essentially dismantling your fix or if my specific scenario is different from what was originally reported. Could you have a look if what I'm saying makes sense?
Chrome might be throttling the minimized tab so much that document.hidden
isn't even being run once to switch to the fallback. I'll change this so that we're checking minimization in a setTimeout timer and never relying on a requestAnimationFrame timer returning.
Just using setTimeout
works, but requestAnimationFrame
has some nice optimizations for canvas that let us get a more stable framerate.
Awesome, sounds like a great fix. I'm relying on just the setTimeout
through patch-package
in my current application but will move to your solution once implemented. Thanks for this!
@Timebutt How are you using setTimeout(callback, 0)
without completely blocking the main thread? I've tried this on a local install and the browser seems to crawl to a halt when using this since the callback is being called very quickly repeatedly.
@Tryptophan you are absolutely right. While it does work when you alt-tab, it fully loads the main thread. I'm on a powerful 6 core machine and didn't even notice it does, trying this again on a less beefy machine is a whole different story.
Thanks for pointing this out, looks like we'll need the solution @t-mullen is suggesting: using the setTimeout()
to do the minimisation check and continue to use requestAnimationFrame()
. Any word when you might have a chance to fix this?
Still working on a fix. The visibility API seems to lie about tab visibility when Chrome switches tabs...
It seems my audio hack no longer works. Spotify is still able to maintain intervals when it is playing... let's hope Chrome hasn't implemented a domain whitelist for timers.
Should be fixed in v3.3.1
. I expect this issue with budget throttling to come up again, but it works for now.
Just so you're aware, I think setTimeout and setInterval are still throttled in background tabs: https://stackoverflow.com/questions/6032429/chrome-timeouts-interval-suspended-in-background-tabs
They ARE supposed to be throttled, but you can violate the budgeting policy by playing audio - which is what we do in this library. It's a hack, but it's the only way to get solid framerates in background tabs.
Thanks for this @t-mullen, I'll have a look at how you fixed it and verify if performance is back where it's supposed to be.
FYI: I also had issues with the Visibility API
in another problem I was fixing, it has to do with the OS you are on among others it seems.
I am not sure that the solution has covered the case where OSX users totally minimize (- button) chrome application and the render stop working.
I have solved this long time ago, but I cant remember the actual thing I did. I used some kinds of Worker running in background to call the drawing function. And the drawing function keeps signaling the worker for settimeout. So the render is running smoothly according to the fps without interuption.
Basically, its a settimeout worker running in background. U can also use this worker for other features to trigger a function in the background.
This is issue still persist in Safari, seems to work fine in chrome
This is issue still persist in Safari, seems to work fine in chrome
The solution is to tell your users to stop using Safari, if you can convince them to do it that would also help many other websites as well; due the millions of other safari-bugs that Apple refuses to address, at this point Safari is the new IE6
I use screen and media streams and merge them using this library. I use webrtc recorder to record the output stream and download it. Everything works as expected, except when the browser is minimized, the video stream pauses (although audio stream continues) till the browser is minimized, and continues once it's maximized. My initial suspect was issue #9, so I disabled/enabled the chrome flag and tried, it didn't help. To zero in on the source, I just recorded the screen stream, they work fine, even if the browser is minimized, I recorded only the video stream and tested it works fine. The only issue is when I am saving the merged feed.
Please check the code source attached below.