AudioWorklet example?

[JorenSix]

Hi,

First of all thanks for this work: it works perfectly for my use-case which is resampling audio within an AudioWorklet. I think this is a logical place for audio resampling. However I am wondering which is the best way to do this? The import statements - which work in a 'normal' worklet - do not seem to work in the audio context.

I have managed to get it working by building a 'shell' version of the library and including the code directly in an Audioworklet. I am wondering if this is the best way and if you could provide an AudioWorklet example showing the best way to do it.

See here for my hacked resampler: https://github.com/JorenSix/Olaf/tree/master/wasm

[aolsenjazz]

Happy to provide it! I think I never provided support for/examples of using in an AudioWorklet because AudioContext by default has a static sample rate. Correct me if I'm wrong: you want to be able to do the following:

import { create, ConverterType } from '@alexanderolsen/libsamplerate-js'; // this isn't working

class ResampleProcessor extends AudioWorkletProcessor {
    constructor(options){
        super(options); 
        // create the resampler at some point
    }

    process(inputs,outputs,parameters){
        // resample with libsamplerate js
    }
}

registerProcessor('resample-processor', ResampleProcessor);

is this correct?

[JorenSix]

Hi, Thanks for considering it.

Indeed the code above is what I am looking for. Some rationale below :)

Resampling within an AudioContext is especially relevant for "analysis nodes". Say e.g. you want to detect pitch in audio or extract some other feature but the analysis algorithm only supports a single sample rate. If the analysis algorithm is computationally expensive you might want to resample to 8000Hz and use that as input.

My specific case is similar: to reuse a C library with an acoustic fingerprinting system. In that case the algorithm expects 16kHz audio.

As you probably know, when requesting a microphone stream of a certain sample rate the Web Audio API only allows configurations your hardware supports. Ideally there should be an option to resample the incoming stream to a requested sample rate (and format) independent of hardware.

On macOS and Chrome the issue becomes even more confusing: when using multiple AudioContexts they can only have the same sample rate. E.g. starting a microphone on 16kHz by itself is possible but not when there is also audio playback on the same page, then everything switches over to 48kHz. There even seems to be an effect of different browser tabs. Other browsers and platforms have similar issues. This is problematic when you need audio in a fixed sample rate.

The solution is to resample incoming audio samples in your code or use the OfflineAudioContext as a resampler. The OfflineAudioContext way needs a lot of code and, crucially, only works on the main browser thread and not in an AudioWorklet. The AudioWorklet should be the place for computationally intensive audio processing like resampling and seems like a good place for this library.

See here for a working resampling demo with this libary: https://0110.be/posts/Resampling_audio_via_a_Web_Audio_API_Audio_Worklet

See here for some info on Olaf, an acoustic fingerprinting system: https://0110.be/posts/Acoustic_fingerprinting_in_the_browser_with_Olaf

[aolsenjazz]

Thanks for the extra info and context! I see why you'd want to use it this way. After some digging into current limitations of AudioWorklets, it looks like the only way to do this would be the following:

1) Modify libsamplerate-js to expose it to the global scope:

globalThis.LibSampleRate = {
  create: create(),
  ConverterType: ConverterType
}

2) Load libsamplerate-js into the isolated audio context scope:

const audioContext = new AudioContext();
await audioContext.audioWorklet.addModule('@alexanderolsen/libsamplerate-js');
await audioContext.audioWorklet.addModule('my-processor.js');

3) Access libsamplerate-js in the processor via the global object:

constructor() {
  super();
  globalThis.LibSampleRate.create(...);
}

I was unable to load libsamplerate-js into the audio context via addModule when I wrote up a trivial example - will likely require some reconfiguring of webpack and/or emscripten. Stack trace:

libsamplerate.js:1 Uncaught TypeError: Cannot set properties of undefined (setting 'LibSampleRate')
  at libsamplerate.js:1:202
  at libsamplerate.js:1:207

The alternative here is basically what you've done - add all of the code directly in the worklet or use a bundler to do it for you. Would be nice to support using the system outlined above, but will require some finagling of compilation process.

I've added trivial scaffolding to examples/worklet for testing in d65fdec21ee7adcaaf051891037cd51878a96b0d

[JorenSix]

Thanks again for your work.

This looks promising, that global context is new to me and could indeed be an elegant solution. Not sure if it is relevant but my hack involved building libsamplerate-js for the shell target which did change the emscripten build.

[aolsenjazz]

Looking into this now. Would you provide the emcc flags that you compiled with in your example repo?

[aolsenjazz]

At first convenience, would you please check out 4e89f7163fb6c72b06d188b378fb934570ceded3 and see if this both works on your end and suits your use case? You'll find a simple overview of how to use with AudioWorklets in the readme, but it's nontrivial so check out the example at examples/worklet for all the info you'll need.

Key changes required were:

• compile using 2 webpack targets, one for UMD (all non-AudioWorklet), one for AudioWorklet
• expose via globalThis (gross, but no choice given the current state of API)

Note that Typescript typing will not work in the context of the AudioWorklet processor. I'm certain it's possible to declare the types on the global object somehow, but I couldn't figure it out quickly, and I don't love the idea of globally declaring its existence when it's only actually be loaded onto the global object in the context of AudioWorkletGlobalScope anyways.

Either way, if all checks out, will move forward with publishing to NPM.

[JorenSix]

Hi, this looks perfect for my use case.

I have tried out the example and it works fine here. It did complain about the lack of a file. The npm run compile-wasm step expects a file at src/post.js. At least in my config (ubuntu 22.04, emcc from apt):

emcc: error: '--extern-post-js': file not found: 'src/post.js'

Having an empty file there was enough to fix this issue.

The globalThis solution may not be perfect but is much cleaner than the hack of including the whole build in a processor (audio worklet) . The build process also gives some inspiration for another WASM audio worklet I have been using in a similarly hacky way.

Thanks for the work!

[aolsenjazz]

Thanks for pointing out that error - that was a straggler from testing. Added Worklet support in 50e864f39ddc069a7d94b9f11373f58c2f34a30f, and publish v2.1.0 to NPM.

A small oversight/update from my last comment: loading the worklet at the '@alexanderolsen/libsamplerate-js/dist/libsamplerate.worklet.js' won't work in web contexts, so users will need to load from a CDN or bundle it in their projects. I've included the link to the CDN download in the readme - that'll be, in some ways, the simplest approach.

Thanks for helping make this library better!