Building wasm to run in AudioWorklet environment which is controlled by JS

[stefanholmer]

I have a use-case where I want to run a wasm module from within an audio worklet in a similar way to what's described here: https://developer.chrome.com/blog/audio-worklet-design-pattern/#setting-up

this relies on the environment being detected as SHELL, which is a bit odd. See also https://github.com/emscripten-core/emscripten/issues/6230#issuecomment-951094323.

Now there's an AUDIO_WORKLET environment available, but this seems to be heavily tied to the Wasm Audio Worklets API and in turn Wasm Workers, and doesn't allow for the pattern described above. A problem with Wasm Audio Worklets is that they rely on shared memory, which may not always be available. https://emscripten.org/docs/api_reference/wasm_audio_worklets.html#wasm-audio-worklets-api

It would be nice if it was possible to build for and specify the audio worklet environment without having to use Wasm Audio Worklets API, and not have to specify the SHELL environment which, I guess, may not be guaranteed to work indefinitely?

[kripken]

cc @juj for thoughts.

the SHELL environment which, I guess, may not be guaranteed to work indefinitely?

I think we'll keep supporting that for the foreseeable future. It is important to be able to test in JS environments like the V8 and SpiderMonkey shells, as the very latest engine features are there.

[sbc100]

I think it might be better to extend the AUDIO_WORKLET environment so that it supports the pattern you need.

@kripken I believe the problem with ENVRIONMENT=shell is not that its going away but that its not designed to mean "can run in an audio worklet" and we don't test that case. The only way to tell emscripten that you want to run in an audio worklet to today is -sAUDIO_WORKLET.

[stefanholmer]

I believe the problem with ENVRIONMENT=shell is not that its going away but that its not designed to mean "can run in an audio worklet" and we don't test that case.

This captures the problem well

[juj]

The needed changes could be expanded to a -sENVIRONMENT=audio_worklet if passing -sENVIRONMENT=shell is not desirable.

If you want to contribute a test for that, I don't see why it couldn't be added as a tested configuration in the CI.

[juj]

Alternatively, we could take the command line -sAUDIO_WORKLET without -sWASM_WORKERS to denote such an environment.

[sbc100]

Do we really want to support yet another build mode though? @juj did you think it makes sense to support running in audio worklets without -sAUDIO_WORKLET? It would be a yet another configuration to support and test.. :(

[juj]

did you think it makes sense to support running in audio worklets without -sAUDIO_WORKLET

Originally I did not anticipate Audio Worklets without Shared Memory (i.e. -sAUDIO_WORKLET without -sWASM_WORKERS) being a thing. However there are developers who have expressed frustration about Emscripten having tied its Audio Worklets support with shared Wasm Memory, and they would like to not have the shared Memory part of it at all.

My impression of what they mean by that is that these developers would like to use Emscripten only to write the Audio Worklet audio processor node code in C/C++ for that better Wasm performance than JS provides, and otherwise they would not be using Emscripten/Wasm at all for any other parts of their web page. (this is a bit of a guess at this point, I am not 100% sure) I.e. the main page would be in JS, they are not using Emscripten to develop their page, and all the Emscripten generated code would only run inside an Audio Worklet. And then their own web site manages the message passing between the worklet and their main page.

This would not necessarily be a full new configuration to test and maintain, since large parts of it would be shared by the existing AUDIO_WORKLET code. That is, we already must support loading the generated .js inside a Wasm Audio Worklet scope, so this kind of -sAUDIO_WORKLET without -sWASM_WORKERS mode would then be the same, just that it would not have a shared Wasm Memory.

I think it probably would work out already in existing codegen by running that -sINVOKE_RUN=0 -sENVIRONMENT=shell -sSINGLE_FILE=1 -o a.js (or by adding -sAUDIO_WORKLETS and removing the -sWASM_WORKERS build time check), and then integrating the generated a.js file into an existing AudioWorklet-based web page that a developer already has.

So if that is useful to some use cases, I don't see why we couldn't drop a test for that scenario in the browser suite. Probably will need someone to champion an example of a compelling use case of how exactly they would like their site integration with Emscripten worklets in such a scenario to work.

[sbc100]

Sounds reasonable to me. Would anyone like to volunteer to create simple / minimal test case for this in the browser test suite?

I think -sENVIRONMENT=audio_worklet is probably a good option for how to opt into this mode.

[stefanholmer]

Are you looking for something different than, e.g., this?

What I'm doing right now is essentially to build with -sMODULARIZE=1 -sENVIRONMENT=SHELL -sINVOKE_RUN=0, and then I prepend the glue code onto my AudioWorkletProcessor implementation. The wasm module is fetched and compiled on the main thread and then transferred over to the worklet.

[sbc100]

Using https://github.com/GoogleChromeLabs/web-audio-samples/tree/main/src/audio-worklet/design-pattern/wasm/ sounds reasonable to me.

[sbc100]

The main thing is the keep the test as simple as possible.

[juj]

Are you looking for something different than, e.g., this?

Thanks, this would be a good start. I posted a draft PR at https://github.com/emscripten-core/emscripten/pull/20630 that shows how that code could be integrated into the interactive browser test suite in Emscripten, and the existing -sAUDIO_WORKLET build mode.

It is not complete, and I am afraid I might not have the time to finish working on this, but I wanted to poke into this mainly to see how well it could integrate.

I would be curious to know how many people are more interested in using such a -sAUDIO_WORKLET=2 mode rather than the -sAUDIO_WORKLET=1 + -sWASM_WORKERS=1 mode, i.e. do people who look into WebAssembly for Audio Worklets generally just want to use Wasm in their Audio Worklets while the rest of their site (incl. the main thread) doesn't have any Wasm content at all?

[hoch]

A few folks who are interested in this topic just had a discussion: https://github.com/GoogleChromeLabs/web-audio-samples/issues/348

Also Re:juj@

i.e. do people who look into WebAssembly for Audio Worklets generally just want to use Wasm in their Audio Worklets while the rest of their site (incl. the main thread) doesn't have any Wasm content at all?

Yes. Many existing audio apps are using vanilla JS for their UI/control logic. Using a compiled WASM blob for sound synthesis/processing on the WebAudio render thread is definitely an established pattern.

[hoch]

It is not complete, and I am afraid I might not have the time to finish working on this, but I wanted to poke into this mainly to see how well it could integrate.

I don't have expertise on this project, but would like to support/help. I glanced the change and am happy to see that one of the existing examples from our repository is being used as a test case.

[stefanholmer]

do people who look into WebAssembly for Audio Worklets generally just want to use Wasm in their Audio Worklets while the rest of their site (incl. the main thread) doesn't have any Wasm content at all?

I don't know the general use of wasm in audio worklets, but in my case there is likely to be other wasm binaries doing other things as well, but they may be independent from the audio worklet wasm and without a need to share memory.

[stefanholmer]

Another peculiarity of Audio Worklets is that performance.now() is undefined in the AudioWorkletGlobalScope, but when building for the SHELL environment emscripten is implementing some functionality using performance.now(), so the user will have to apply a workaround (polyfill or changing the code to avoid calls to performance.now()).

Adding an AUDIO_WORKLET build option that doesn't enforce WASM_WORKERS would be useful also to fix this issue.