gogins / csound-extended

Extensions for Csound including algorithmic composition, Android app, and WebAssembly.
GNU Lesser General Public License v2.1
40 stars 1 forks source link

Implement AudioWorklet driver for CsoundWebAudio #18

Closed gogins closed 6 years ago

gogins commented 6 years ago

See:

It seems clear that Csound has to run on the audio rendering thread, which has advantages, but must then communicate with the browser or JavaScript context thread by message passing through a port. Otherwise, it seems pretty straightforward.

gogins commented 6 years ago

Seems to show how to load WASM in rendering thread:

https://github.com/kripken/emscripten/issues/6230

gogins commented 6 years ago

The AudioWorklet design is classic upper half/lower half. In Csound's Start method, first the upper half has to be done in the CsoundAudioWorklet, then the lower half has to be done in the CsoundAudioProcessor.

Errors have to be sent asynchronously to an error callback.

Some functions can wait for return values using promises.

gogins commented 6 years ago

This claims to work but I'm having trouble building/running it:

https://github.com/madChopsCoderAu/WASMAudio

I can get it to configure with:

emconfigure ./configure  CPPFLAGS="-I/usr/include/eigen3" EIGEN_CFLAGS="-I/usr/include/eigen3"

And the build runs to completion... but the example does not run, looks like there are missing parts.

gogins commented 6 years ago

This https://github.com/madChopsCoderAu/WASMAudio project was updated while I was in Buenos Aires. I am trying it again. These are the steps I took to get it to build and run:

source ~/emsdk/emsdk_env.sh
./configure CPPFLAGS="-I/usr/include/eigen3"
emconfigure ./configure CPPFLAGS="-I/usr/include/eigen3"
make VERBOSE=1
emmake make VERBOSE=1

Edit config.h to #define HAVE_EMSCRIPTEN and #define HAVE_EIGEN.

Install Chrome 66 (first version with AudioWorklet enabled by default).

npm install -g polymer
bower install Polymer/polymer#^2.0.0
polymer install

Copy all in webApp/bower_components up one level to webApp. Then

polymer serve

Sheesh, but it works! The WASM code is embedded in the libwasmaudio.js script.

gogins commented 6 years ago

I'm now working on modularizing csound-extended's WebAssembly build. Now that I understand how this works:

        csound_extended(Module).then(function(Module) {
            console.log("Module: " + Module + " has been loaded.\n");
        });

it builds and runs fine.

To use -s SINGLE_FILE=1 builds and runs, but the csound_extended.js file is about 12 (!!) times bigger than the WASM files and takes forever to load... not a good way to go.

gogins commented 6 years ago

According to https://github.com/WebAudio/web-audio-api/issues/778, the processor must be defined in a separate file from the node. I'm not sure what is best here as the CMask code is WASM as well as the processor code.

I will have to modularize things but that should solve it.

gogins commented 6 years ago

https://github.com/google/schism

gogins commented 6 years ago

I have both CsoundAudioWorklet and CsoundAudioProcessor working now. The critical steps were:

  1. Passing these options to em++: em++ -v -O1 -std=c++11 --bind --pre-js ../src/CsoundAudioProcessor_prejs.js --post-js ../src/CsoundAudioProcessor_postjs.js -DINIT_STATIC_MODULES=1 -s WASM=1 -s ASSERTIONS=0 -s "BINARYEN_METHOD='native-wasm'" -s LINKABLE=1 -s RESERVED_FUNCTION_POINTERS=1 -s TOTAL_MEMORY=268435456 -s ALLOW_MEMORY_GROWTH=1 -s BINARYEN_ASYNC_COMPILATION=0 -s NO_EXIT_RUNTIME=0 -s SINGLE_FILE=1 -s EXTRA_EXPORTED_RUNTIME_METHODS='["ccall", "cwrap"]' csound_web_audio.bc libcsound.a ../deps/libsndfile-1.0.25/libsndfile-wasm.a -o CsoundAudioProcessor.js. In particular, building a single file does the base64 encoding automatically, and turning off asyc compilation is essential.
  2. Understanding that my port does not use Transferable arguments, just lists of simple values.
gogins commented 6 years ago

Works! Scrims now plays. On stopping and restarting, I get this:

Uncaught RuntimeError: memory access out of bounds
    at wasm-function[3785]:216
    at wasm-function[3737]:5
    at wasm-function[608]:796
    at wasm-function[5888]:13
    at invoke_ii (http://localhost:5103/CsoundAudioProcessor.js:10737:32)
    at wasm-function[623]:652
    at wasm-function[209]:31
    at wasm-function[394]:114
    at wasm-function[5892]:15
    at dynCall_iii_2293 (eval at makeDynCaller (http://localhost:5103/CsoundAudioProcessor.js:8543:19), <anonymous>:4:12)
<WASM UNNAMED> @ wasm-00c041fa-3785:108
<WASM UNNAMED> @ wasm-00c041fa-3737:4
<WASM UNNAMED> @ wasm-00c041fa-608:371
<WASM UNNAMED> @ wasm-00c041fa-5888:8
invoke_ii @ CsoundAudioProcessor.js:10737
<WASM UNNAMED> @ wasm-00c041fa-623:342
<WASM UNNAMED> @ wasm-00c041fa-209:16
<WASM UNNAMED> @ wasm-00c041fa-394:58
<WASM UNNAMED> @ wasm-00c041fa-5892:9
dynCall_iii_2293 @ VM54:4
Csound$PerformKsmps @ VM173:8
process @ CsoundAudioProcessor.js:11797

Seems to be fixed by properly calling csound.Stop and csound.Reset.

gogins commented 6 years ago

TODOs include:

  1. Setting up real-time/offline rendering is done before connecting the audio graph in Web Audio, but needs to be done after that, in the CsoundAudioProcessor.Start function. Perhaps I can push down the graph startup until after the CsoundAudioProcessor actually starts. Without pthreads this is problematic. Done by rendering to soundfile entirely within one call to AudioWorkletProcessor.process. Needs testing in NW.js to see if the soundfile is actually created.
  2. Many getters and other lightweight functions should return a value from CsoundAudioProcessor to CsoundAudioNode synchronously. Not doing, as not necessary.
  3. Messages from CsoundAudioProcessor need to be printed by CsoundAudioNode. Done in a kind of hackish way (replacing console.log with a closure, this will work only for one instance of Csound at a time).
gogins commented 6 years ago

I can now do some performance testing and profiling using Scrims, as I can increase the tempo until I get breakups.

  1. csound_node:
  2. csound_web_audio:
  3. csound_audio_node:
gogins commented 6 years ago

Too many uncontrolled variables with Scrims, adapting thecmask.html example instead. This test works in NW.js if it uses Chrome 66 or later. The instances build up and the time is displayed.

gogins commented 6 years ago

Couldn't figure out what to do to throw exceptions mandated by the spec, used RangeError instead.

gogins commented 6 years ago

There is a bug where restarting with the audio worklet doesn't work. The problem is CsoundEmbind::Start hangs the second time. I think the processor is still running.

gogins commented 6 years ago

Anyway it is clear enough that the AudioWorklet implementation runs more smoothly than the ScriptProcessorNode implementation. I need to get it to restart...

gogins commented 6 years ago

Ok, just adding simple FM notes up to see how many instances can run without breaking up:

  1. csound.node: 243
  2. ScriptProcessorNode: 274
  3. AudioWorklet: GUI got logy around 500 plus.
gogins commented 6 years ago

Trying again with faster note generation, no logging to TextArea, and instance count at dropouts:

  1. csound.node: Occasional glitches, becoming a real problem ~1,200.
  2. ScriptProcessorNode: ~1,200.
  3. AudioWorklet: ~950.

I repeated these tests several times with consistent results.

gogins commented 6 years ago

Conclusion: CsoundAudioNode, the AudioWorklet + WebAssembly + Embind implementation of Csound for WebAssembly, is done for now.

The AudioWorklet implementation is not actually quite as performant as the other implementations, but does seem run more steadily and to be more resistant to dropouts caused by events in the browser thread. I wonder if it is possible to speed up the AudioWorkletProcessor.process function by not looking at the input unless necessary. I am testing that now. No help.

gogins commented 6 years ago

These are the only obvious TODOs:

  1. Refactor creation, add separate constructor, etc. to enable running without any input node.
  2. Implement some synchronous getters.