CsoundWebAudio examples not working properly

[gogins]

• [x] Redo csound_loader.js to actually work for Android, csound.node, CsoundWebAudio, and CsoundAudioNode. Opening as new issue.
• [x] ALSA is great, but there is choppiness and buzziness in Xanadu with AudioWorklet (CsoundAudioNode) with PulseAudio,
• [x] Dropouts in Xanadu with ScriptProcessorNode (CsoundWebAudio). Nothing I can do about this but it is sorta usable.
• [x] Csound messages not printing with AudioWorklet (CsoundProcessorNode). It turns out the AudioWorklet was probably not even working. I have fixed this.
• [x] Csound is not restartable with ScriptProcessorNode without refreshing the page. I have fixed this by using the default message callback, not the per instance callback.
• [x] The native AudioContext audio format needs to be printed for the user not just the developer.
• [x] Print loading and startup messages from WebAssembly to the Csound message console.

[gogins]

If I break at Start and then resume, the AudioWorklet works, though it's buzzy and noisy. If I don't break, it doesn't work. That suggests Start returns before things are actually working.

[gogins]

Trying to reorder connecting and processing in AudioWorklet.

[gogins]

Things to look at:

• [x] Priority of AudioWorker thread. Enabled chrome://flags/#enable-experimental-webassembly-features and chrome://flags/#audio-worklet-realtime-thread. No obvious help.
• [x] Bindings in onprocessaudio going out of scope thanks to garbage collection. Don't think so, this method is a class member and the class is not going out of scope.
• [x] Use Float32Array buffers. ScriptProcessorNode has Float32Array buffers. AudioWorkletProcessor has Float32Array buffers. CsoundEmbind uses Float32Array buffers. This is already optimal.
• [x] Use AudioBuffer.copy* functions in ScriptProcessorNode. Can't do that as Csound buffers are frame major and WebAudio buffers are channel major. This is perhaps a problem, along with implicit float32 to float (64) conversions.
• [x] Does Scrims.html in fact use AudioWorklet? Yes.
• [x] Are the message ports in AudioWorklet synchronous or asynchronous? They are asynchronous.
• [x] If asynchronous, how to serialize calls to the API? There is no standard way to do this. I can create a promise in the call and resolve it with the return message. I will look to see what CsoundObj does about this. Turns out it is very similar to my AudioWorklet code The alternative is a shared buffer with atomics. Before processing a block of audio, the node reads input audio and state, after processing the node writes output audio and state. Another alternative is to push operations onto a stack in the processor and only execute when all dependencies are satisfied.
• [x] Use let not var in CsoundWebAudio.js. Using let ran, but provided no better and possibly worse performance. Reverted to var. According to https://stackoverflow.com/questions/21467642/is-there-a-performance-difference-between-let-and-var-in-javascript let and var are about equally performant.

[gogins]

This https://developers.google.com/web/updates/2018/06/audio-worklet-design-pattern provides context for these issues. This, frankly, is a real mess. The basic recommendation is to run CsoundEmbind in a Worker not in the Processor, and use the Processor only for synchronously and efficiently transferring audio and control data using shared memory rather than a MessagePort. It's not clear to me that this would provide any substantial advantage over the ScriptProcessorNode.

However see this: https://github.com/hlolli/csound-wasm. It follows the whole pattern above and exposes most of the "C" Csound API. Not sure about C++.

Actually, now that it's working with ALSA, it's clear that the AudioWorklet is indeed significantly more performant then ScriptProcessorNode.

[gogins]

OK, things are getting clearer. I will need to make at least Csound.CompileCsdText, Csound.Stop, and Csound.Cleanup behave as though they are synchronous so that the sequencing always works.

I will not make the module loading be synchronous because the page might freeze.

The logic in csound_loader.js is very tricky. It was wrong. I think I have mostly fixed it.

CsoundAudioNode is now rendering. But it is still buzzy and choppy.

[gogins]

Profiler total times over several seconds of Xanadu:

• CsoundWebAudio: 98.9% in AudioProcessNode.onaudioprocess, 82.3% in Csound$PerformKsmps.
• CsoundAudioNode: 87.3% in the AudioWorkletThread process call, 86.3% in Csound$PerformKsmps.

This is hasty, but the implication is that the choppiness is not down to Csound or even to Embind, but to the AudioWorklet infrastructure. As suggested by the worklet design pattern, this may be down to using MessagePort instead of shared memory.

That being so, the only advantage I see to AudioWorklet at this time is that ScriptProcessorNode is deprecated. After reviewing the documents, and after my experience with other implementations of Csound using AudioWorklet which are all just as choppy as mine, I am going to get AudioWorklet to work with calls that need to be serialized being made synchronous, and let ScriptProcessorNode be preferred to AudioWorklet.

[gogins]

Try:

• [x] Both with and without sync/await pattern.
• [x] Both with and without shared memory audio buffers. No need to do this, already using heap buffers.

[gogins]

See: https://stackoverflow.com/questions/26150232/resolve-javascript-promise-outside-function-scope.

[gogins]

Finally got Csound.CompileCsdText to act as though it is synchronous. This was not easy to understand. First the event handler for the Play button has to be async. Then the CsoundAudioNode.CompileCsdText has to be async with a Promise that assigns its resolve callback to CsoundAudioNode.resolveCompileCsdText.

I'm not sure how to make this consistent with the other JavaScript interfaces to Csound.

[gogins]

Using emscripten_console_log does not help.

[gogins]

Regarding the double run to build... first run:

/usr/bin/cmake -E cmake_progress_start /home/mkg/csound-extended/WebAssembly/cmask/CMakeFiles 0
rm: cannot remove 'CMakeCache.txt': No such file or directory
Packaging some resources...
python: can't open file '/tools/file_packager.py': [Errno 2] No such file or directory
Configuring csound-static...
configure: cmake -DCMAKE_VERBOSE_MAKEFILE=1 -DBUILD_PLUGINS_DIR=plugins -DUSE_COMPILER_OPTIMIZATIONS=0 -DWASM=1 -DINIT_STATIC_MODULES=1 -DUSE_DOUBLE=NO -DBUILD_MULTI_CORE=0 -DBUILD_JACK_OPCODES=0 -DEMSCRIPTEN=1 -DCMAKE_TOOLCHAIN_FILE=/cmake/Modules/Platform/Emscripten.cmake -DCMAKE_MODULE_PATH=/cmake -DCMAKE_BUILD_TYPE=Release -GUnix Makefiles -DHAVE_BIG_ENDIAN=0 -DCMAKE_16BIT_TYPE=unsigned short -DHAVE_STRTOD_L=0 -DBUILD_STATIC_LIBRARY=YES -DHAVE_ATOMIC_BUILTIN=0 -DHAVE_SPRINTF_L=NO -DUSE_GETTEXT=NO -DLIBSNDFILE_LIBRARY=../deps/libsndfile.a -DSNDFILE_H_PATH=../deps/libsndfile-1.0.25/src ../../dependencies/csound -DCMAKE_CROSSCOMPILING_EMULATOR="/home/mkg/emsdk/node/12.9.1_64bit/bin/node"
CMake Error at /usr/share/cmake-3.16/Modules/CMakeDetermineSystem.cmake:99 (message):
  Could not find toolchain file: /cmake/Modules/Platform/Emscripten.cmake
Call Stack (most recent call first):
  CMakeLists.txt:6 (project)

CMake Error: CMake was unable to find a build program corresponding to "Unix Makefiles".  CMAKE_MAKE_PROGRAM is not set.  You probably need to select a different build tool.
CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
-- Configuring incomplete, errors occurred!
Making csound-static...
make: make csound-static -j6
make: *** No rule to make target 'csound-static'.  Stop.

Second run:

Packaging some resources...
Remember to build the main file with  -s FORCE_FILESYSTEM=1  so that it includes support for loading this file package
Configuring csound-static...
configure: cmake -DCMAKE_VERBOSE_MAKEFILE=1 -DBUILD_PLUGINS_DIR=plugins -DUSE_COMPILER_OPTIMIZATIONS=0 -DWASM=1 -DINIT_STATIC_MODULES=1 -DUSE_DOUBLE=NO -DBUILD_MULTI_CORE=0 -DBUILD_JACK_OPCODES=0 -DEMSCRIPTEN=1 -DCMAKE_TOOLCHAIN_FILE=/home/mkg/emsdk/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake -DCMAKE_MODULE_PATH=/home/mkg/emsdk/upstream/emscripten/cmake -DCMAKE_BUILD_TYPE=Release -GUnix Makefiles -DHAVE_BIG_ENDIAN=0 -DCMAKE_16BIT_TYPE=unsigned short -DHAVE_STRTOD_L=0 -DBUILD_STATIC_LIBRARY=YES -DHAVE_ATOMIC_BUILTIN=0 -DHAVE_SPRINTF_L=NO -DUSE_GETTEXT=NO -DLIBSNDFILE_LIBRARY=../deps/libsndfile.a -DSNDFILE_H_PATH=../deps/libsndfile-1.0.25/src ../../dependencies/csound -DCMAKE_CROSSCOMPILING_EMULATOR="/home/mkg/emsdk/node/12.9.1_64bit/bin/node"

and configuration then succeeds with many warnings.

[gogins]

Looks like EMSCRIPTEN_ROOT is undefined or empty on the first run. Fixed by setting up the Emscripten SDK environment in a more consistent manner.

[gogins]

Configure the build to ignore or fix certain warnings:

• [x] -Wimplicit-int-float-conversion.
• [x] -Wno-dev for CMake.
• [x] Fix emcc:WARNING: Wasm source map won't be usable in a browser without --source-map-base.
• [x] Fix /home/mkg/csound-extended/dependencies/cmask/source/parser.cpp:1057:21: warning: 'delete' applied to a pointer that was allocated with 'new[]'; did you mean 'delete[]'? [-Wmismatched-new-delete]. Nope, I'm leaving that in case it causes a memory leak and I want to know why.

[gogins]

I will try an actual callback for the Csound messages. Emscripten supports this with a C API. See:

https://stackoverflow.com/questions/20421002/passing-a-c-function-to-a-javascript-function-in-emscripten https://emscripten.org/docs/porting/connecting_cpp_and_javascript/Interacting-with-code.html#interacting-with-code-call-function-pointers-from-c

[gogins]

An actual callback is easy with emscripten::val as a parameter in the CsoundEmbind::SetMessageCallback function.

But again, it works only on the first run. No, now it works on all runs. I used csoundSetDefaultMessageCallback instead of csoundSetMessageCallback.

Trying to pass the actual text message now from Csound::Embind....

[gogins]

Uncaught TypeError: Illegal invocation
    at shim (http://localhost:5103/CsoundAudioProcessor.js:20248:13)
    at __emval_call (http://localhost:5103/CsoundAudioProcessor.js:8851:23)
    at wasm-function[210]:0x7f03f

Is this caused by an invalid function, or by an invalid parameter, or both? The message parameter is OK, the function object is invalid. I will try variations on the function object.

Works now. Workaround: create global var for the MessagePort that will be in scope of the callback shim. But this limits the instances of Csound on a page to 1.

[gogins]

I have looked at the design pattern HeapAudioBuffer and inspected the current AudioBuffer in the debugger. I do not understand the motivation for the HeapAudioBuffer unless it is simply to skip range checking or to adapt to changes in buffer shape (not an issue here).

Wait, I do get it. Elementwise access to the buffers is thus pushed down into C code compiled to WASM. However, this should not be a problem for my code because spin and spout are obtained before process is ever called, and these are just Float32Array allocated from the WASM heap like the buffers in the HeapAudioBuffer.

In other words, instead of pushing heap buffers down into C code before calling process, I am pulling heap buffers up from C code before calling process. Should be a wash.

[gogins]

Regarding the separate Worker pattern, I do not see any reason to do this unless it isolates the audio processing from interruption by other things happening in the browser. But that may be the motivation. I think I will look again at @hlolli's code again and see if it uses this pattern. If so, it confers no significant advantage.

[gogins]

Where is this coming from? audio buffered in 256 sample-frame blocks. Maybe this is an issue. I would have thought this would be 128.

This message comes from Csound, and probably should not be an issue, because we are directly using spin and spout. However, I have set this size to 128.

[gogins]

Closing the processor's MessagePort when performance begins does not reduce audio problems.

[gogins]

Profiled again. Auto-scrolling the message window is a performance load. I will try not using a message callback at all.

This does not help.

[gogins]

Nothing learned from the profile, as it is not easy to interpret the data.

I tried Faust's online Faust compiler and although it's not as bad as our stuff, it is still bad. There is always a repetitive glitch.

Looking through this, and I will see if I can find a more current meeting: https://www.w3.org/2019/09/16-audio-minutes.html

I will look through issues here, both V1 and V2: https://github.com/WebAudio/web-audio-api.

[gogins]

This issue https://github.com/WebAudio/web-audio-api/issues/2008 discusses the shared memory design pattern, and implies that garbage collection in the rendering task is the cause of my woes. The expert says "never ever create objects [he means JS objects] on this rendering thread."

Because Csound itself creates objects during rendering, this may not be possible.

It is clear that at least as of July 2019, the official WebAudio spec for AudioWorklet was not complete.

I will try my player.html in Firefox. Oops, doesn't run at all, can't create AudioWorklet.

[gogins]

The experts expect that the worker will listen on the MessagePort just for a start message, then use a SharedArrayBuffer to manage all further communications with the worklet.

[gogins]

csound_loader.js (embedded in player.html doesn't work on Android, as it presupposes the WebAssembly code has been loaded. I can try just loading these files, that should work, but better if they're not required. Can .js files be loaded programmatically? Yes, but only asynchronously.

[gogins]

CsoundAudioNode just started working perfectly, but I am not sure why. I had assumed it would not work and changed several things at the same time. Check:

• [x] Use console.log vs. message callback in csound_loader.js. The callback works fine.
• [x] NW.js and Chrome fight about NODE vs. WEB causing _dirname to blow up when loading csound_extended in NW.js. Fixed.
• [x] The Chrome browser appears to have been updated, look at release notes. But I don't have the most recent version. I don't think this is it.
• [x] The WebAudio sampleRate was 44100, now is 48000. I rebooted and perhaps I am currently using only ALSA and that works much better.

Indeed, I am now using ALSA without PulseAudio.

The sampleRate mismatch is a red herring. The fix is simply using just ALSA instead of PulseAudio.

The implication also is that for Csound, it is not necessary to used the shared memory IPC mechanism and the MessagePort appears to be working fine.

[gogins]

If Csound completes its performance, it will not restart.

Changed the order of calls. Did not help.

Changed AudioWorkletProcessor.process to always return true unless buffer shapes don't match. That fixed the restart problem. Returning false takes the node out of the signal flow graph, i.e. permanently turns off Csound.

[gogins]

I can't run csound.node in NW.js without PulseAudio. I can run Csound on the command line or in WebAssembly without PulseAudio. This probably is the reason: https://github.com/nwjs/nw.js/issues/3573.

I changed /etc/pulse/daemon.conf to use float samples and a sampling rate of 48000 Hz. This has enabled NW.js to run Csound just fine with PulseAudio. But with PulseAudio, CsoundAudioNode is still choppy and buzzy, though I think not as much as before.