Implement AudioWorklet driver for CsoundWebAudio

[gogins]

See:

• https://developers.google.com/web/updates/2017/12/audio-worklet.
• https://www.webaudiomodules.org/wamsynths/.but this seems to have a polyfill for AudioWorklet that falls back to a Web Worker.

It seems clear that Csound has to run on the audio rendering thread, which has advantages, but must then communicate with the browser or JavaScript context thread by message passing through a port. Otherwise, it seems pretty straightforward.

[gogins]

Seems to show how to load WASM in rendering thread:

https://github.com/kripken/emscripten/issues/6230

[gogins]

The AudioWorklet design is classic upper half/lower half. In Csound's Start method, first the upper half has to be done in the CsoundAudioWorklet, then the lower half has to be done in the CsoundAudioProcessor.

Errors have to be sent asynchronously to an error callback.

Some functions can wait for return values using promises.

[gogins]

This claims to work but I'm having trouble building/running it:

https://github.com/madChopsCoderAu/WASMAudio

I can get it to configure with:

emconfigure ./configure  CPPFLAGS="-I/usr/include/eigen3" EIGEN_CFLAGS="-I/usr/include/eigen3"

And the build runs to completion... but the example does not run, looks like there are missing parts.

[gogins]

This https://github.com/madChopsCoderAu/WASMAudio project was updated while I was in Buenos Aires. I am trying it again. These are the steps I took to get it to build and run:

source ~/emsdk/emsdk_env.sh
./configure CPPFLAGS="-I/usr/include/eigen3"
emconfigure ./configure CPPFLAGS="-I/usr/include/eigen3"
make VERBOSE=1
emmake make VERBOSE=1

Edit config.h to #define HAVE_EMSCRIPTEN and #define HAVE_EIGEN.

Install Chrome 66 (first version with AudioWorklet enabled by default).

npm install -g polymer
bower install Polymer/polymer#^2.0.0
polymer install

Copy all in webApp/bower_components up one level to webApp. Then

polymer serve

Sheesh, but it works! The WASM code is embedded in the libwasmaudio.js script.

[gogins]

I'm now working on modularizing csound-extended's WebAssembly build. Now that I understand how this works:

        csound_extended(Module).then(function(Module) {
            console.log("Module: " + Module + " has been loaded.\n");
        });

it builds and runs fine.

To use -s SINGLE_FILE=1 builds and runs, but the csound_extended.js file is about 12 (!!) times bigger than the WASM files and takes forever to load... not a good way to go.

[gogins]

According to https://github.com/WebAudio/web-audio-api/issues/778, the processor must be defined in a separate file from the node. I'm not sure what is best here as the CMask code is WASM as well as the processor code.

I will have to modularize things but that should solve it.

[gogins]

https://github.com/google/schism

[gogins]

I have both CsoundAudioWorklet and CsoundAudioProcessor working now. The critical steps were:

1. Passing these options to em++: em++ -v -O1 -std=c++11 --bind --pre-js ../src/CsoundAudioProcessor_prejs.js --post-js ../src/CsoundAudioProcessor_postjs.js -DINIT_STATIC_MODULES=1 -s WASM=1 -s ASSERTIONS=0 -s "BINARYEN_METHOD='native-wasm'" -s LINKABLE=1 -s RESERVED_FUNCTION_POINTERS=1 -s TOTAL_MEMORY=268435456 -s ALLOW_MEMORY_GROWTH=1 -s BINARYEN_ASYNC_COMPILATION=0 -s NO_EXIT_RUNTIME=0 -s SINGLE_FILE=1 -s EXTRA_EXPORTED_RUNTIME_METHODS='["ccall", "cwrap"]' csound_web_audio.bc libcsound.a ../deps/libsndfile-1.0.25/libsndfile-wasm.a -o CsoundAudioProcessor.js. In particular, building a single file does the base64 encoding automatically, and turning off asyc compilation is essential.
2. Understanding that my port does not use Transferable arguments, just lists of simple values.

[gogins]

Works! Scrims now plays. On stopping and restarting, I get this:

Uncaught RuntimeError: memory access out of bounds
    at wasm-function[3785]:216
    at wasm-function[3737]:5
    at wasm-function[608]:796
    at wasm-function[5888]:13
    at invoke_ii (http://localhost:5103/CsoundAudioProcessor.js:10737:32)
    at wasm-function[623]:652
    at wasm-function[209]:31
    at wasm-function[394]:114
    at wasm-function[5892]:15
    at dynCall_iii_2293 (eval at makeDynCaller (http://localhost:5103/CsoundAudioProcessor.js:8543:19), <anonymous>:4:12)
<WASM UNNAMED> @ wasm-00c041fa-3785:108
<WASM UNNAMED> @ wasm-00c041fa-3737:4
<WASM UNNAMED> @ wasm-00c041fa-608:371
<WASM UNNAMED> @ wasm-00c041fa-5888:8
invoke_ii @ CsoundAudioProcessor.js:10737
<WASM UNNAMED> @ wasm-00c041fa-623:342
<WASM UNNAMED> @ wasm-00c041fa-209:16
<WASM UNNAMED> @ wasm-00c041fa-394:58
<WASM UNNAMED> @ wasm-00c041fa-5892:9
dynCall_iii_2293 @ VM54:4
Csound$PerformKsmps @ VM173:8
process @ CsoundAudioProcessor.js:11797

Seems to be fixed by properly calling csound.Stop and csound.Reset.

[gogins]

TODOs include:

1. Setting up real-time/offline rendering is done before connecting the audio graph in Web Audio, but needs to be done after that, in the CsoundAudioProcessor.Start function. Perhaps I can push down the graph startup until after the CsoundAudioProcessor actually starts. Without pthreads this is problematic. Done by rendering to soundfile entirely within one call to AudioWorkletProcessor.process. Needs testing in NW.js to see if the soundfile is actually created.
2. Many getters and other lightweight functions should return a value from CsoundAudioProcessor to CsoundAudioNode synchronously. Not doing, as not necessary.
3. Messages from CsoundAudioProcessor need to be printed by CsoundAudioNode. Done in a kind of hackish way (replacing console.log with a closure, this will work only for one instance of Csound at a time).

[gogins]

I can now do some performance testing and profiling using Scrims, as I can increase the tempo until I get breakups.

1. csound_node:
2. csound_web_audio:
3. csound_audio_node:

[gogins]

Too many uncontrolled variables with Scrims, adapting thecmask.html example instead. This test works in NW.js if it uses Chrome 66 or later. The instances build up and the time is displayed.

[gogins]

Couldn't figure out what to do to throw exceptions mandated by the spec, used RangeError instead.

[gogins]

There is a bug where restarting with the audio worklet doesn't work. The problem is CsoundEmbind::Start hangs the second time. I think the processor is still running.

[gogins]

Anyway it is clear enough that the AudioWorklet implementation runs more smoothly than the ScriptProcessorNode implementation. I need to get it to restart...

[gogins]

Ok, just adding simple FM notes up to see how many instances can run without breaking up:

1. csound.node: 243
2. ScriptProcessorNode: 274
3. AudioWorklet: GUI got logy around 500 plus.

[gogins]

Trying again with faster note generation, no logging to TextArea, and instance count at dropouts:

1. csound.node: Occasional glitches, becoming a real problem ~1,200.
2. ScriptProcessorNode: ~1,200.
3. AudioWorklet: ~950.

I repeated these tests several times with consistent results.

[gogins]

Conclusion: CsoundAudioNode, the AudioWorklet + WebAssembly + Embind implementation of Csound for WebAssembly, is done for now.

The AudioWorklet implementation is not actually quite as performant as the other implementations, but does seem run more steadily and to be more resistant to dropouts caused by events in the browser thread. I wonder if it is possible to speed up the AudioWorkletProcessor.process function by not looking at the input unless necessary. I am testing that now. No help.

[gogins]

These are the only obvious TODOs:

1. Refactor creation, add separate constructor, etc. to enable running without any input node.
2. Implement some synchronous getters.