whatwg / html

HTML Standard
https://html.spec.whatwg.org/multipage/
Other
8.07k stars 2.65k forks source link

Event loop does not account for worklets #4213

Closed annevk closed 4 years ago

annevk commented 5 years ago

https://drafts.css-houdini.org/worklets/#the-event-loop claims worklets have an event loop, which make sense, but they don't integrate with its processing model the way workers and documents do.

cc @bfgeek

annevk commented 5 years ago

This can be fixed more easily once #4342 is done.

The main problem here is that a worklet does not always run in parallel to all other event loops, so it needs to either be able to share an event loop or do some other trickery to make this clear. On the other hand, I think for all worklets we want to keep the ability to run them on a different thread if desired, so ensuring no synchronous access assumptions are/can be made in this setup would be good.

gterzian commented 5 years ago

There might also be the "mirror" issue of this, which is how the event-loop of the WorkletGlobalScope integrates with an eventual "processing loop" in which the worklet is involved with.

For example the AudioWorkletProcessor runs in two contexts:

  1. "The process() method is called synchronously by the audio rendering thread at every render quantum"
  2. A "normal" distinct WorkletGlobalScope event-loop, in this case a AudioWorkletGlobalScope

Calling the process method of the processor(effectively the "inside" of the worklet), is defined as part of the "rendering loop", which consists of a few main steps:

  1. Process the control message queue.
  2. Process a render quantum.
  3. Perform a microtask checkpoint.

Next to this, the WorkletGlobalScope would presumably be running it's own event-loop, handling tasks related to incoming messages of the MessagePort of each AudioWorkletProcessor, handling tasks from calls to addModule.

However, the idea is that the AudioWorkletGlobalScope would be running on the rendering thread:

This special execution context is designed to enable the generation, processing, and analysis of audio data directly using a script in the audio rendering thread. https://webaudio.github.io/web-audio-api/#audioworkletglobalscope

So somehow the event-loop of the AudioWorkletGlobalScope should interleave with the "rendering loop", and probably not run "in-parallel".

For example I guess you wouldn't want an attribute of a AudioWorkletProcessor to change because of an incoming message having been handled on its port, right in the middle of a call to process.

Here is an example of interplay between onmessage and process: https://github.com/GoogleChromeLabs/web-audio-samples/blob/master/audio-worklet/design-pattern/shared-buffer/shared-buffer-worklet-processor.js


I guess this could be Web Audio specific, however I think it's worth thinking at the level of Worklet about how to spec such a situation.

I can think of a few potential solution:

A. Don't let the event-loop of a WorkletGlobalScope "run" itself. Instead, spec a hook to run it as part of an outside "processing loop"(or even a window loop, for example for paint worklets whose paint function wouldn't be running in parallel). (Basically a kind of reverse (micro-)task checkpoint, which would handle all currently enqueued tasks, inteleaved with microtasks, for a given worklet global scope, via a step on a processing loop). B. Let the event-loop of a WorkletGlobalScope "run" an outside processing loop, similar to update-the-rendering

https://drafts.css-houdini.org/css-paint-api-1/#draw-a-paint-image seems to take the A approach, where the window event-loop drives "running" a paint-worklet(although that's easier since it doesn't come with a port), stating:

Run invoke a paint callback given name, inputArguments, snappedConcreteObjectSize, workletGlobalScope optionally in parallel.

Note: If the user agent runs invoke a paint callback on a thread in parallel, it should select a paint worklet global scope which can be used on that thread.

Although that's similar to how a rendering loop would call the process method of a AudioWorkletProcessor. What makes that one a bit more hairy is that it also comes with a MessagePort, whose message-queue will have to be run somewhere as well, outside of the call to `process.