Example of using audioWorklets (createAudioWorkletNode) with React

[stevebarakat]

Can someone please create an example of using audioWorklets (createAudioWorkletNode) with React. Preferably and example of running Effects on audio worklet nodes.

[marcelblum]

Not sure what would be different about using audioWorklet in React vs vanilla js. Maybe you can be more specific about what you're trying to do or code you've already tried?

[stevebarakat]

That's fine. It doesn't have to be React. Is there a vanilla example anywhere? Using Tone's audioWorklet api, (not web audio api).

[marcelblum]

Tone's AudioWorklet api is pretty basic, createAudioWorkletNode() just returns a standardized-audio-context-wrapped native AudioWorkletNode rather than a full-featured ToneAudioNode, meaning that it lacks Tone conveniences like toDestination() and requires a bit of careful handling to interact with other Tone components. This could use improvement in Tone.js, but I think the thinking behind it being implemented this way is because Tone can't infer much about the details of the custom processor being imported, so it's left to the user to handle specifics. Most of the heavy lifting is done inside the worklet js which has its own structure and syntax that is not Tone-specific (see https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API/Using_AudioWorklet).

Here's an example using the sample noise generator worklet from here:

await Tone.getContext().addAudioWorkletModule("https://cdn.jsdelivr.net/gh/GoogleChromeLabs/web-audio-samples/src/audio-worklet/basic/noise-generator/noise-generator.js");
const noiseWorkletNode = Tone.getContext().createAudioWorkletNode("noise-generator");
const gain = new Tone.Gain(2).toDestination();
noiseWorkletNode.connect(gain.input); // gain.input is necessary here because noiseWorkletNode is not a ToneAudioNode, so noiseWorkletNode.connect(gain) would fail

[stevebarakat]

Thank you for the example. That was easy to implement with React but I'm still a bit confused... Most of the examples of audioWorklets are sound generators similar to that example but I'm interested in using audioWorklets with Tone.js components and effects like Tone.Player, Tone.Meter, Tone.Delay, etc–in order to improve the performance of my app (a multi-track audio mixer with fx busses). Is that possible?

[marcelblum]

If you check the other chromelabs examples you'll find input processing examples like bitcrusher and one pole filter. Also Tone.js has its own simpler version of a bitcrusher implemented with a worklet, source here. I'm not sure if your question is Tone-specific or more about AudioWorklet in general. Yes it's possible to integrate your own custom AudioWorklet with Tone objects, for example in a chain of effects mixing Tone objects and custom AudioWorkletNodes. But it's important to keep in mind what AudioWorklet can and can't do well, for example it would never be more performant than what native web audio nodes already provide, and unless it uses WebAssembly its speed is still limited by JS. It's meant for extending the web audio api, for example this one provides high quality realtime timestretching pitch shift but can't really be used in more than 4 or 5 nodes at once at high quality settings due to processing speed limitations.

[stevebarakat]

Okay. I guess I misunderstood what audioWorklets are used for. Are you saying they're strictly for creating custom audio nodes? I thought their main purpose was improving performance by providing additional threads to process any audio your app is using.

For example, I have an audio mixer app created with Tone that starts to get buggy when I add too many effects at once. I thought I could create an audioWorklet to pass a bunch of Tone effects to, to process on a separate thread. Theoretically, that would improve performance, WebAssembly or not, wouldn't it? Is there a way to implement the patten I'm describing, even if I have to use a web worker or something else?

Here's a link to an unfinished buggy version of the app, if that helps. You can login with fake credentials if you want to. https://remixer.fly.dev/

Thanks for you insight. It's greatly appreciated.

[marcelblum]

Cool project, hey how did you get the stems to those songs?! lol

The history of AudioWorklet is that it was developed as a faster replacement for ScriptProcessorNode, which didn't run on a separate thread. AudioWorklet can certainly be used strategically for optimization (although overuse can have a negative effect especially on low core count devices that don't have threads to spare). When you say

they're strictly for creating custom audio nodes? I thought their main purpose was improving performance by providing additional threads to process any audio your app is using.

both of those sentences are true. AudioWorklet is the best way to access raw audio output buffers in realtime or generate audio in realtime, and it's pretty cool we can do this in the browser. But there is no magic way that AudioWorklets can instantly make your project faster, they need to be deployed strategically. AudioWorklets can be input-only (for analysis or external output routing), output-only (for procedural sound generation), or input+output (for realtime processing). They're perfect for extending the web audio api with a novel effect, or offloading fancy analysis without blocking the main audio thread.

Tone.js is already pretty well optimized and automatically uses AudioWorklets in places. In your demo the bottleneck is probably coming from specific effects and their handling. To my ears it sounded like some effects weren't being disposed of/disconnected when I expected them to be. Also different effects can bring dramatically varying processing expense, for example Chebyshev is super cheap, FrequencyShifter can be surprisingly expensive, and I don't know what you're using for PitchShift but that can also be expensive. Reverb is always expensive but typically Convolver is the cheapest since web audio offers that natively, sounded to me like your reverb is maybe not using Convolver? EQ can also be surprisingly expensive - I assume you're using Tone's EQ3? - and you might want to only instantiate those nodes lazily as needed - for example if all EQ bands are flat then the node should be disposed. Also there could be bottlenecks on the visual rendering side as well, depending on how your animations are implemented, keep in mind web audio has to compete with page rendering/reflows/drawing/etc. for cpu cycles. React is known to bring overhead in that department.

Overall there is no reason you shouldn't be able to do what you're trying to do in a modern browser - mix a bunch of synced tracks through mostly shared effect nodes (at least I hope the effects are paraphonically shared in your implementation?) - so I think you probably just have some optimizing to do, particularly better cleanup and lazy loading, and also see recommendations here if you haven't already. Maybe the Madonna one is slightly pushing it with 17 tracks I think I counted. I mean even in a standalone DAW you'd need a powerful machine to render a 17 track session with effects.

[stevebarakat]

Also there could be bottlenecks on the visual rendering side as well, depending on how your animations are implemented, keep in mind web audio has to compete with page rendering/reflows/drawing/etc. for cpu cycles. React is known to bring overhead in that department.

Would the app have better performance if it used Vanilla JS instead of React? React might be part of the problem. I'm using requestAnimationFrame recursively to trigger the function that animates the meters (600 times per second). So the meters and the transport clock are constantly re-rendering. Here's a sandbox demonstrating that. https://stackblitz.com/edit/github-5ujywu?file=src/components/Channels/ChannelStrip.tsx Maybe there's a better way to automate the meters? I tried using Tone.Draw + Tone.Loop instead of requestAnimationFrame but they were too slow.

Do you use a framework or Vanilla JS with Tone? I'm using React/Remix—which is a significant paradigm shift from other React frameworks, in that you don't really need to use useState and useContext because you use routes and action/loader functions on the server instead–allowing you to delete lots of code you needed before to keep track of local state and context. yada yada yada. If React is causing performance issues with page rendering/reflows/drawing/etc, I wouldn't mind trying out qwick with Tone.

Also, currently, all the parameters (vol, pan, fx, etc) are static. Is there a way to save/record MIDI events in real time using Tone.Transport as MIDI clock? ..so the audio parameters and their UI controls automate dynamically on playback? I mapped my Korg nanoKontrol to the faders, knobs, and buttons of the mixer and log their values to the console. I can see the values change in real time but don't know how to save or play them back in realtime.

[stevebarakat]

there is no magic way that AudioWorklets can instantly make your project faster...

Not sure if I'm getting caught up in semantics here, but my intention isn't for AudioWorklets to make the project faster, it's to free up memory (cpu cycles) to prevent it from crashing.

[marcelblum]

Would the app have better performance if it used Vanilla JS instead of React?

I can't answer that for sure in your case, generally the answer to this question is often yes 😅, but I'd recommend doing some testing and benchmarking and getting some hard evidence before any major refactor. Chrome dev tools like Layers and Rendering go a long way in helping to identify UI bottlenecks. Just from a quick glance at your code looks like you're mostly using canvas draws and requestAnimationFrame, all good practice for ensuring optimized gpu-based animations, no glaring red flags there. I suggest adding a framerate setting so you can really separate visual rendering from audio for testing purposes, and check if you're still getting the same performance issues at a lower framerate. One easy trick is to simply minimize the browser while playing audio which will effectively halt page paints and throttle requestAnimationFrame in most browsers, and keep an eye on browser cpu usage so you can see how much is coming from audio processing alone.

600 times per second

I hope you mean 60 not 600? IMHO 60fps is overrated for most use cases, personally I like to cap browser animations to 40fps in audio projects to reduce cpu/gpu use.

Do you use a framework or Vanilla JS with Tone?

I'm not using any frameworks with Tone, but these days I am mostly using Tone in Electron which gives me the luxury of focusing on tuning performance for a single browser engine. That said, you can get even more vanilla by not using Tone at all and using vanilla web audio api calls instead, Tone brings its own overhead on some level, there are always tradeoffs...!

Is there a way to save/record MIDI events in real time using Tone.Transport as MIDI clock?

Not that I know of but maybe someone else on here has more experience doing this? Sounds like you're already pretty close if you've gotten as far as mapping the controls and logging the results, from there it's just a matter of storing and retrieving the data using the audio context clock timestamps which is a very high resolution clock.

free up memory (cpu cycles) to prevent it from crashing

Wait what you're getting full on crashes? I thought we were just talking about audio glitching which would be more suggestive of an audio processing bottleneck. Crashing suggests a memory hole or other bug. AudioWorklets wouldn't reduce memory usage, just make your app more multithreaded and help prevent blocking the main thread.

[stevebarakat]

Wait what you're getting full on crashes?

It doesn't crash. That's the wrong word. The audio cuts off until the app is reloaded... whatever that word is. Saving memory isn't the right word either.. My idea was that I could process the useVuMeter hook as an audioWorklet to help prevent bottlenecks (the hook that recursively calls requestAnimationFrame–yes 60, 600 was a typo). But if that's not going to work, no big deal. There's other things I can do to improve performance.

Thanks for all your help. I really appreciate it!

[stevebarakat]

What is Offline Context used for?

[braebo]

I was hoping to use it for periodically rendering a few minutes of an ever-growing, generated array of Parts.

[stevebarakat]

What's the difference between Offline Context and Context?

[braebo]

You can read about that here.

[tmhglnd]

@marcelblum , thanks for your example on how to create the audioWorkletProcessor with the white noise processor example. I got the noise-processor to work as output. But could you (or someone else) provide some insight in how to use a processor with input? I think I've tried everything I could think of and have searched all over the place for a clear example but it is not very well documented as far as I could find. I've tried both the onepole-filter and the bit-crusher as example. I thought of using a dummy Gain input and output and connect the effect between there like so:

let gainIn = new Tone.Gain(1);
let gainOut = new Tone.Gain(1);
let fx = Tone.getContext().createAudioWorkletNode('one-pole-processor');

gainIn.connect(fx);
fx.connect(gainOut.input);

But this gives me an error:

Uncaught TypeError: Cannot read properties of undefined (reading '0') at OnePoleProcessor.process (

I've also tried to connect with fx.input (similar to how the fx is connected to gainOut.input). But this gives me another error:

Uncaught (in promise) Error: Cannot connect to undefined node

Any hints or resources I might've overlooked? Thanks in advance!

[marcelblum]

Hey so looks like 2 different issues there @tmhglnd.

The first error is an issue within the one-pole-processor code. I don't know if you're using an identical one-pole-processor.js taken straight from the GoogleChromeLabs repo but it would be helpful to see what exact code you're using there. Looks like as is that's a simplified example that lacks some safety checks to make sure that the processor is actively receiving audio data. For example inputs[0] can be empty sometimes. So typically an input-processing worklet needs to have a check like if (inputs[0]?.length) before assuming inputs is actually populated.

Also note that addAudioWorkletModule() is async and you can't do createAudioWorkletNode() until the module has been loaded e.g. using await or then. Hopefully you already knew that but just wanted to mention as that can yield some errors if not.

In your example fx.input would be expected to be undefined because createAudioWorkletNode() returns a native AudioWorkletNode not a ToneAudioNode (explained in my comment). A native AudioWorkletNode lacks the input property unique to Tone-created nodes. gainIn.connect(fx) should just work. Mixing native web audio nodes with Tone nodes can get a little tricky because of these differences. Sometimes the easiest thing to do is create your own custom ToneAudioNode to play nicer with other Tone nodes. It's not documented/easily exposed to the user (probably should be) but you can do something like this as an alternative, wrapping the AudioWorkletNode with Tone scaffolding:

await Tone.getContext().addAudioWorkletModule("https://cdn.jsdelivr.net/gh/GoogleChromeLabs/web-audio-samples/src/audio-worklet/basic/noise-generator/noise-generator.js");
const newNode = new Tone.ToneAudioNode();
newNode.noiseWorkletNode = Tone.getContext().createAudioWorkletNode("noise-generator");
newNode.input = new Tone.Gain();
newNode.output = new Tone.Gain();
newNode.input.chain(newNode.noiseWorkletNode, newNode.output);
newNode.toDestination();

Then you can just use connect, disconnect, toDestination etc. as with other Tone nodes and not have to worry about additional hacking using input etc.

[tmhglnd]

Hi @marcelblum, Thanks for the very quick reply! I was indeed using the identical effects from GoogleChromeLabs. But I discarded the one-pole and bitcrushes examples, because was not planning on using them anyways. So went on with coding my own effect including your suggestion to check for the inputs[0].length. I've also used ToneAudioNode() and all together this now seems to work very well! It really is a bit of puzzle with no documentation on this, but now that I've got it working for a simple downsampling effect I think I can continue exploring/expanding. Below the code for others who are struggling with this in the future.

The processor stored in a separate processor.js file

// A downsampling/degrading processing effect
// Sample and holds the signal based on an incrementing counter
// written by Timo Hoogland, (c) 2023, www.timohoogland.com
//
class DownSampleProcessor extends AudioWorkletProcessor {
    static get parameterDescriptors() {
        return [{
            name: 'down',
            defaultValue: 8,
            minValue: 1,
            maxValue: 1024
        }];
    }

    constructor(){
        super();
        // the frame counter
        this.count = 0;
        // sample and hold variable array
        this.sah = [];
    }

    process(inputs, outputs, parameters){
        const input = inputs[0];
        const output = outputs[0];

        // if there is anything to process
        if (input.length > 0){
            // for the length of the sample array (generally 128)
            for (let i=0; i<input[0].length; i++){
                // for every channel
                for (let channel=0; channel<input.length; ++channel){
                    // if counter equals 0, sample and hold
                    if (this.count % parameters.down === 0){
                        this.sah[channel] = input[channel][i];
                    }
                    // output the currently held sample
                    output[channel][i] = this.sah[channel];
                }
                // increment sample counter
                this.count++;
            }
        }
        return true;
    }
}
registerProcessor('downsampler-processor', DownSampleProcessor);

Usage with ToneJS

await Tone.getContext().addAudioWorkletModule('./processor.js');

const fx = new Tone.ToneAudioNode();
fx.input = new Tone.Gain(1);
fx.output = new Tone.Gain(1);
fx.workletNode = Tone.getContext().createAudioWorkletNode('downsampler-processor');
fx.input.chain(fx.workletNode, fx.output);
// Then just connect some source to this with source.connect(fx). 
// And connect the FX output to something with fx.connect(), or output it to main with .toDestination()

// When you want to set a parameter
let param = fx.workletNode.parameters.get('down');
param.setValueAtTime(32, time);

Thanks again!