Multiple real-time AudioWorklet processing threads?

[juj]

Currently there exists exactly one AudioWorkletGlobalScope per an AudioContext. All AudioWorkletProcessors share that same global scope, and so all AudioWorkletNodes on a given audio context are executed sequentially on that one audio thread.

To get more audio threads, it seems that an application currently would have to instantiate multiple AudioContexts on the page - but that seems like an abuse of the spirit of how Web Audio API in general would like to be used (and will have the limitation that the audio processing graphs will then be completely separate)

Have there been discussions about other ways to enable multiple real-time audio threads?

[padenot]

This is not an easy problem. Web Audio API (like most audio rendering systems) assumes, for all intents and purpose, a single audio rendering thread. Sometimes, fork/join approaches are used, within an audio callback: at the beginning of the audio rendering (i.e. when the callback is called), work is distributed to threads of the right scheduling class. This also can only happen if the audio rendering graph topology lends itself to some parallelization. All those are things that are hard to do on the Web.

In Web Audio API, the graph has an arbitrary number of sources (i.e. AudioWorkletProcessor, AudioBufferSourceNode, etc.), sinks (AudioDestinationNode, MediaStreamAudioDestinationNode), and the topology can have cycles and all sorts of weirdness, so it's going to be fairly hard to parallelize.

Even if we're only talking about AudioWorkletProcessors being potentially parallelizable, it's easy to construct a graph where there are data dependencies between Processors, and this means a bunch of context switches during the rendering.

Regular workers can be used to render some audio, but they are not real-time, so there's going to be a priority inversion from the get go. AudioWorkletProcessors also cannot use Atomics.wait, but can probably implement something like a work-stealing scheme. Maybe we could advance the platform to this direction though.

The "multiple AudioContext connected by MediaStreams" approach is something that could be made working, but it's fairly limiting, and if we use the same audio output device, there might be a linearization point inside the OS's audio stack (or a mixer in the browser, etc.).

I'm not sure that there is a good answer here, in the current state of things.

[juj]

Thanks @padenot for the information!

Hopefully when Wasm Audio Worklets feature becomes more widely used, we'll hear more about usage feedback, and then can conclude whether this will be an important future feature need, or something that applications will be able to avoid.

Btw does Audio Worklet thread have Atomics.waitAsync?

[padenot]

Btw does Audio Worklet thread have Atomics.waitAsync?

No, the functions of an AudioWorkletProcessor (or really anything in an AudioWorkletGlobalScope), it's on a real-time thread) are supposed to "really" return to stable state frequently.

Maybe in the future we can implement fork/join for audio, but we'd need real-time workers.

[guest271314]

@padenot @juj

Btw does Audio Worklet thread have Atomics.waitAsync?

Yes, Atomics.waitAsync() is defined in `AudioWorkletGlobalScope on Chrome and Chromium 107.

[padenot]

We need to check what Chromium does here.

When process(...) returns, audio need to be available in the buffers passed in as arguments, that's how it works. But the waitAsync's Promise's thenable will only be called during a microtask checkpoint, which is when process(...) returns. At this point, it's too late: the audio is gone, so the worklet can't react to the memory change.

The best way is probably to write a WPT and to talk with ES folks, and ask them to get us a way to disable this method in particular scopes.

[guest271314]

Since AudioWorket supports transferables, on Chromium, technically we can transfer MediaStreamTrackProcessor or MediaStreamTrackGenerator readable and/or writable to AudioWorket then do whatever we want with the data in processor. Why are multiple threads needed?

[hoch]

2023 TPAC Audio WG Discussion:

Rather than spawning additional threads from AudioWorkletGlobalScope, using the Web Worker with configurable QoS would be more general and scalable solution. The WG will not be pursuing this capability as an active project.

[juj]

I disagree that Web Workers with QoS would solve the problem, or otherwise if it does, we might as well deprecate and obsolete AudioWorklets and replace them with Worker-QoS altogether?

[hoch]

At least on its current proposal, the Worker-QoS is not connected to the audio infrastructure. The expectation is to use it with AudioWorklet for an audio-specific use case. If Worker-QoS becomes more capable in the future with built-in media I/O, AudioWorklet may become unnecessary for certain use cases.

Given the current scope of the group's charter, this is not something we can focus on at the moment. When we open a new charter for the group, we can revisit this idea and include as a V2 project.