AudioWorkletProcessors cannot process music (if you have more than one)

[carlsmith]

Currently (in Chrome 86 and FireFox 82 on macOS), large arrays of data (like a recording) cannot be passed to an AudioWorkletProcessor, even during instantiation. Otherwise, any existing nodes will stutter, as the audio thread blocks to handle the data.

In my case, I implemented a Deck node (like a DJ turntable). Naturally, it needs to copy minutes of stereo samples from the main thread, which means that instantiating a new Deck instance causes any existing one to stutter.

I tried using a ShardArrayBuffer, but it made no noticeable difference, so I tried using setTimeout to send a chunk of samples every ten or twenty milliseconds (I've tried many combinations of block size and time delta). I even tried calling the main thread from the process method to tell the main thread to send a chunk over, but nope. Nothing works. I simply cannot load recorded music to the audio thread without it stuttering.

I think it may actually be initializing megabytes of memory (during initialization of the node) that's blocking the thread (not copying data to the memory). I've checked now, and that works fine.

Passing a track to the audio thread, in whole or incrementally, causes it to drop out or stutter.

[padenot]

This repository is home to the Web Audio API specification.

Questions on how to do something with the Web Audio API are better suited to either Stack Overflow or the Web Audio API slack.

I'll add that people have been implementing this kind of thing for some time in the Web Audio API, without issues and with rather high performance (limited by WASM speed), and that what you're seeing is likely to be (a) bug(s) in your implementation.

[carlsmith]

@padenot - Would you be kind enough to point me to any example of an AudioWorkletProcessor subclass that takes a track (even a few minutes long) from the main thread (by any means), and creates a memory large enough to store the track, while another instance of that class is already actively playing some other track?

You could potentially create two decks up front, with tons of memory each, and load data (in blocks) onto them on the fly, so you'd reuse the same two nodes for the lifetime of the application. That would waste a lot of memory though (and many people use four decks).

[rtoy]

I believe the general approach is not to store the entire clip in the worklet, Use a SharedArrayBuffer that you fill from the main thread (or worker thread) and which can be read from in the worklet processor, possibly using a ring buffer to handle differences in timing.

It was also mentioned that you could possibly transfer the buffer from the main thread to the worklet. (Not sure how that's done; I'd have to look that up or find some examples.)

[carlsmith]

@rtoy - thanks for considering this.

The issue with a ring buffer is that the audio processor never knows which part of the track is going to be played next. For example, digital turntables universally provide the ability to instantly jump to hotcues, which can be set by the user on the fly, and the jump cannot be latent.

You could potentially store a short buffer for the start of each hotcue (as they're assigned), but there are other features, like drop (takes an arbitrary stylus position and jumps to it instantly) that would still be a problem.

As far as I know, all DJ software makes the entire track available (gradually, so it takes a lot longer than the WebAudio API to load a track onto a deck, but without stalling the audio thread).

It was also mentioned that you could possibly transfer the buffer from the main thread to the worklet. (Not sure how that's done; I'd have to look that up or find some examples.)

Transferring a buffer (even a shared array buffer) that's large enough to hold even a few minutes of 44.1KHz stereo audio blocks the audio thread long enough that it drops out badly.

[carlsmith]

I tried creating a shared array buffer on the audio thread, then posting it to the main thread, then copying the data from one buffer to the other. It works better, but still audibly drops out sometimes.

I thought I'd fixed it, but there's still no way to share any kind of buffer or memory between the main thread and audio thread.

[rtoy]

I think @padenot's comment in https://github.com/WebAudio/web-audio-api/issues/2268#issuecomment-718865572 still holds. There are lots of knowledgeable people on the webaudio slack channel and also on stack overflow.

I'm sorry I can't be more helpful; I rarely write worklet code and even less code using shared array buffers.

[carlsmith]

I think @padenot's comment in #2268 (comment) still holds

@rtoy - seriously mate. I appreciate you're genuinely trying to be helpful, and am grateful for that, but you're wrong.

I asked @padenot for an example, and he didn't/couldn't provide one. Then, I posted a demo that showed that this is clearly a limitation of the API (not a bug in my code, as he assumed), but nobody cared to even look at it, so I gave up.

Ultimately, this is not my problem. Other people will have the same issue, and it will eventually need to be fixed. I have a workaround I can live with till then (it wastes tens of megabytes of RAM, but that's not the end of the world).

The API provides no way to share tens of megabytes of data (a typical recording) without the audio thread dropping out. You cannot pass arrays, buffers, shared array buffers or shared Wasm memories that large, in either direction, without the audio thread choking.

You cannot chop the data into blocks and send them one at a time, without the audio thread stuttering, unless you really send small blocks at a relatively low rate, so it takes forever to sync an entire track.

You cannot use ring-buffers or anything like that, unless the application is able to preempt which data the audio thread will require, which massively complicates the implementation of some features (hotcues, scratching), and makes others impossible to implement without completely unacceptable latencies (drops, spinbacks).

This is all easily demonstrable. I'd rewrite the demo and post it again, if I thought it would make a difference.

[skratchdot]

@carlsmith - can you post a working example on https://codesandbox.io/ so we can hear the stutter you are talking about and view the implementation?

[padenot]

I asked @padenot for an example, and he didn't/couldn't provide one. Then, I posted a demo that showed that this is clearly a limitation of the API (not a bug in my code, as he assumed), but nobody cared to even look at it, so I gave up.

This is a spec bug tracker, not a help forum. I explained that you could ask your question to support forums, pointing to two communities where you would have had an answer very quickly. I finally wrote two pages that show how to do this, including workaround for browser bugs.

Please limit your contribution in this issue tracker to specification matters, thanks. I'll be happy to answer any question you have in the appropriate venue.

[carlsmith]

This is a spec bug tracker, not a help forum.

I don't need help. My code is working fine. The API has a limitation that I've explained many times, and would have proven, if anyone had looked at the example code.

@skratchdot - I did post a demo, but pulled it down since, as the consensus seemed to be that I was mistaking bugs in all of my code for limitations of the API, and should ask Stack Overflow.

[carlsmith]

@padenot posted a broken demo and closed my issue again. I give up.

[padenot]

You're probably seeing Chrome not using real time threads for audio and using a buffer size that is too low for your machine's spec, this works well on Windows, Mac and Linux for me. Testing this is how I found the Firefox bug and offered a workaround.

Please open a bug on the vendor's bug tracker.

On Friday, November 6, 2020, Carl Smith notifications@github.com wrote:

@padenot posted a broken demo and closed my issue again. I give up.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.< https://ci5.googleusercontent.com/proxy/OwMeMWdRtOIyPt8oS_KzeFMXVyvlHcFF-kdltZw7qf87Qf5nNJbzuwTz3DsZQFZdS8MzI4xKZIIXK_VdajhTWHT4PLrE5JK-RBEhBtOzN1fqzVF8zQkSn0N6emc9mMT74KA4s3lIvDJOK5NL7IEUr678XkFJmRT0ucsCPBDmAesbBSgtwhXuLI2QhHu-7iEHfdd20qYn2GyjcptomRgUGIFqYGbcUt6LDSfjjiVe4A=s0-d-e1-ft#https://github.com/notifications/beacon/AACCBFOJSTRPPHVUYDJ4VX3SONBZTA5CNFSM4TDAOMSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOFMKAP7Q.gif>