Add a new version of requestAnimationFrame for use with OffscreenCanvas

[junov]

The fundamental problems we need to solve are:

1. Window.requestAnimationFrame is not exposed in workers.
2. commit() may not be synchronized with a browsing context graphics update, so we need a version of requestAnimationFrame that is independent from the browsing context.

I have prepared an initial proposal that we can use as a starting point for discussion: https://wiki.whatwg.org/wiki/OffscreenCanvas.requestAnimationFrame

Any thoughts?

[junov]

Summoning @kenrussell @mephisto41 @grorg @toji

[kenrussell]

Nice work putting that together Justin. On an initial reading it sounds like a great start, and I don't immediately have any suggestions for improvement.

[domenic]

Given the open questions, what about an API like requestAnimationFrameAndCommit()?

[junov]

Given the open questions, what about an API like requestAnimationFrameAndCommit()?

Yes, I was also thinking we should merge the two, but I am not sure what the right form would be. An alternative would be to have only one commit method (as opposed to rAF vs non-rAF versions), and it would take an optional animation callback argument. Another possibility is to use promises: ctx.commit().then(drawNextFrame);

These are all equivalent in terms of functionality. I'm just wondering which embodiment would be preferable.

[domenic]

It seems to me like merging them would be good to avoid the issues in open questions. I don't have a strong opinion on the particular design (or even on merging them; I'm really just an interested and hopefully helpful observer).

But I guess I am confused whether you want to run the code then commit, or commit then run the code. requestAnimationFrameAndCommit(codeToRun), or commit(codeToRunBeforeCommiting), would support the former. Whereas commit().then(codeToRun), or commit(codeToRunAfterCommiting) would work for the latter.

[domenic]

Is this issue a dupe of https://github.com/whatwg/html/issues/2051 ?

[junov]

Is this issue a dupe of #2051 ?

D'Oh! I closed the other one.

[junov]

I guess I am confused whether you want to run the code then commit, or commit then run the code.

Hmmm. I was thinking commit then run the code for the next frame, which is sort of consistent with how window.rAF works. But both ways would work.

[junov]

Reposting comment by @toji here:

I don't think we want commit() to throw an exception, since one of the expected use cases is rendering to the headset AND mirroring to a PC monitor. rAF() is maybe a bit different, though I could maybe see an argument for having a main logic loop that looks like:

canvas.requestAnimationFrame(drawMirroredView); // Draws to main display at 60Hz
vrDisplay.requestAnimationFrame(drawVRView); // Draws to HMD at 90Hz

In a case like this the page would be drawing a different view of the scene (a third person view or "demo driver" view) to the main display, and drawing the first person view to the HMD. Thus far I've been coding such examples to draw to both within the single rAF, but that means that we're producing frames for the external display that are getting dropped on the floor. That's a somewhat odd pattern, though, and will produce an uneven GPU workload across frames, so I'm not sure if it's worth it.

[junov]

So in the case where we want the canvas to push a different view to the page than it does to the vr display, we would need separate commit mechanisms. I see that the WebVR spec already has its own commit-like thing: VRDisplay.submitFrame(). So I suppose OffscreenCanvas.commit() should just push a frame to the placeholder canvas, and should have no effect on any connected VRDisplays?

[toji]

That would work well for us. (And FWIW I'm proposing that we rename our submitFrame to commit as well, since they're really the same concept and just push the buffer to different places)

[domenic]

I guess I am confused whether you want to run the code then commit, or commit then run the code.

Hmmm. I was thinking commit then run the code for the next frame, which is sort of consistent with how window.rAF works. But both ways would work.

Well, requestAnimationFrame(cb) actually adds cb to the list of animation frame callbacks, and then the event loop first runs all those callbacks, then renders. So RAF is run code, possibly multiple times, then render.

Or did I misunderstand what you were thinking?

---

This brings up another important point. requestAnimationFrameAndCommit(cb), or some variation, is not very "composable" in the sense that multiple parts of the program cannot use it to add different callbacks. Whereas a separated requestAnimationFrame(cb) + commit() flow allows multiple calls to requestAnimationFrame, and thus multiple parts of the program to do that kind of work.

Is that kind of coordination aspect of RAF useful? Or do we envision people having a single loop? This kind of makes me lean back toward separate functions and just being sure to design the interaction correctly.

[junov]

After thinking more about the idea of merging rAF and commit, I think @domenic's first idea of requestAnimationFrameAndCommit is the right thing to do because it provides a reasonable solution for use cases where multiple animation callbacks are queued. Basically, when the OffscreenCanvas is ready to process a new frame, it would run all the queued animation callbacks, and it would invoke commit() only once, after the last animation callback has finished.

[junov]

With separated rAF + commit, we have a big problem with multiple calls to rAF: where do we call commit? if each animation callback calls commit, the we'd be wastefully pumping out many frames per animation cycle. Not good/

[domenic]

Oh, so is the idea really requestAnimationFrameAndAlsoCommitIfYouAreTheLastAnimationFrameCallback? :)

[junov]

Yes. I think that behavior would be quite sane from the dev's standpoint.

[bfgeek]

cc/ @esprehn

+1 to the implicit commit() on the last animation frame callback if we go for the requestAnimationFrame api. Needing to explicitly call commit at the end of the raf feels slightly broken.

That said, I agree it would be nice merge the two APIs. Having two different ways of making a canvas draw feels clunky.

I think that we should strongly consider the promise returning commit() that @junov suggested.

The only downsides of this approach are:

• Lack of coordination for multiple callbacks drawing to the same canvas. My gut feeling is that this is a non-issue, but would like to hear other peoples thoughts. The primary reason that rAF supports this type of coordination is because people draw to different canvas', not the same on each rAF.

• First frame is a bit weird with the commit() API? The initial draw calls wouldn't be frame aligned? Or you have to commit a blank canvas?

• The final downside is that it breaks the rAF path which everybody already knows how to use. I think this is also a non-issue as the rAF here is on the canvas, and typically inside a worker which is sufficiently different.

@junov Are there any non-obvious advantages to the commit() type API? Does this allow us to do fancy things regarding gpu back-pressure? Add additional arguments for scheduling hints? E.g. await commit('low-priority').

[junov]

I too kind of like the promise-returning commit(). One of its advantages is that it is a single API entry point that can be used for both tight rendering loops, as well as use cases that only do occasional updates. For occasional updates, one would just ignore the promise and call commit() whenever they need to.

You are right that the first frame would not be vsync-aligned (at least the JS part would not be) but I think that is fine. Sub-frame jank on the first frame is not really noticeable.

For cases where the user calls commit too often (not waiting for the promise to resolve), the user agent may intervene in order to keep performance smooth. This is a problem we've solved before on the main (browsing context) thread. The same mitigation techniques could be used in a worker. This means we could:

1. Pause the Worker to relieve gpu back-pressure.
2. Drop a frame. This means that when commit() is called while the frame from the previous commit() call is still queued. We just squash the previous frame (it will never be displayed). This method is especially useful if the implementation uses a form of display list for deferring rasterization, which makes it possible to skip a frame before it has been rasterized.

I concede that these interventions are a bit ugly, but they only ever kick in on outlier webpages (e.g. unmaintained legacy pre-requestAnimationFrame demos).

Anyways, I think that the problems of dealing with gpu-backpressure and overdraw do not need to be exposed in the spec. These are implementation issues. As long as devs follow best practises, and use rAF (or a commit promise) to schedule animation loops, this problem never arises anyways.

Regarding the scheduling hints, I think that is an orthogonal problem that should be discussed separately, and that feature can be added to any form of API we chose by just adding an argument, likely a dictionary. For those reading this who do not not know what we mean by "scheduling hints". The problem is that when it is not possible to render frames at the same rate as the monitor's vsync, a hint would be useful to determine whether rAF should render at the highest possible rate, to minimize latency; or whether it should drop to a lower but regular frame rate (e.g. 30fps on a 60Hz monitor), to maximize smoothness.

[junov]

So I landed an experimental implementation of promise-returning commit() in blink. As I wrote tests for it, I found this form of API to be interestingly quite readable. I also had to work out the details of making recurrent vs occasional animation updates function correctly using a single API. It is a pretty simple and elegant solution. I am going formalize this proposal a bit more and write back to this thread shortly for further discussion.

[bfgeek]

We talked about this mode in WebVR F2F in Seattle yesterday. https://github.com/w3c/webvr/issues/174

^ was the proposal and this removes VRDisplay.prototype.requestAnimationFrame and VRDisplay.prototype.submitFrame in favour of using the commit() API. +1 from me for this approach generally.

[junov]

Interesting use of the async/await syntax for driving an animation loop in w3c/webvr#174. In the case where you have multiple OffscreenCanvases that you want to animate in lock step, you could do something like this:

async function drawLoop() {
  do {
    (...)
  } while (await Promise.all([ctx1.commit(), ctx2.commit()]));
}

[bfgeek]

Yeah, the other thing this allows which came up is the case where you might want to do some post after committing the frame but before the next loop, e.g.

async function drawLoop() {
  let p;
  do {
    // draw calls.
    p = ctx.commit();

     // post draw calls which does other interesting things. 
  } while (await p);
}

[junov]

Updated proposal: https://wiki.whatwg.org/wiki/OffscreenCanvas.requestAnimationFrame

That page contains both competing proposals (rAF vs. commit+promise). Feedback on the commit+promise processing model would be much appreciated. In particular, if any one can think of use cases and edge cases that may work...

[junov]

At the WebGL working group F2F today, it was suggested that having rAF in the WorkerGlobalScope is the minimum viable API for getting animations to work. The vsync it should align with (in the case of multiple monitors) is the same one as the window.rAF of the scope that owns the worker. The only way this would be the wrong vsync is if we someday exposed OffscreenCanvas in SharedWorker, in which case it would become possible to obtain an OffscreenCanvas that was transferred from another window.

[bfgeek]

I don't really think this will be forwards compatible with how we want to use canvas in the future.

For example: 1) This isn't a forwards looking API in regards to multiple display, e.g. WebVR being able to run at a high frame rate inside a worker. 2) Canvas contexts being instantiated on different GPUs which may have different back pressure. The promise-returning-commit solves this case neatly.

This also brings in issues; e.g. 1) If you create a Worker inside a ServerWorker what is rAF rate tied to?

Were there any specific objections against the promise returning commit?

[junov]

Were there any specific objections against the promise returning commit?

Not that I recall, it was mostly about identifying the MVP. Will cycle back on this when the minutes or recordings are published.

[junov]

Discussion moved to WICG Discourse, as requested by people who cannot participate here. https://discourse.wicg.io/t/offscreencanvas-animations-in-workers/1989

[annevk]

What's the status here?

[junov]

This has been stalled for a while. I am going to capture the proposal made by @toji on the TAG review into a working document (.md file) so that we can iterate on it to get things moving again. Members of the W3C TAG expressed concerns over the lack of unity around different APIs that do essentially the same thing (window.rAF, WebVR's rAF, and OffscreenCanvas). What we're gonna try to do is define a single mix-in interface that can provide rAF everywhere. This implies possibly dropping a use case that commit() was designed to support: a never ending task (tight animation loop) that continually commits frames. This is based on the discussion in the TAG review here: https://github.com/w3ctag/design-reviews/issues/141

[trusktr]

I still see commit() in the spec, here:

https://html.spec.whatwg.org/multipage/canvas.html#offscreencontext-commit

Should that be removed?

[kenrussell]

It should; at this point requestAnimationFrame on workers is working well as a way of driving animations in OffscreenCanvas. cc @fserb @Juanmihd

[Kaiido]

This is https://github.com/whatwg/html/pull/3872 isn't it?