Audio Workers

[olivierthereaux]

Originally reported on W3C Bugzilla ISSUE-17415 Tue, 05 Jun 2012 12:43:20 GMT Reported by Michael[tm] Smith Assigned to

Audio-ISSUE-107 (JSWorkers): JavaScriptAudioNode processing in workers [Web Audio API]

http://www.w3.org/2011/audio/track/issues/107

Raised by: Marcus Geelnard On product: Web Audio API

https://dvcs.w3.org/hg/audio/raw-file/tip/webaudio/specification.html#JavaScriptAudioNode

It has been discussed before (see [1] and [2], for instance), but I could not find an issue for it, so here goes:

The JavaScriptAudioNode should do its processing in a separate context (e.g. a worker) rather than in the main thread/context. It could potentially mean very low overhead for JavaScript-based audio processing, and seems to be a fundamental requirement for making the JavaScriptAudioNode really useful.

[1] http://lists.w3.org/Archives/Public/public-audio/2012JanMar/0225.html [2] http://lists.w3.org/Archives/Public/public-audio/2012JanMar/0245.html

[olivierthereaux]

That said, as suggested by several people, most recently by Joe (http://lists.w3.org/Archives/Public/public-audio/2013OctDec/0177.html), the preferred solution may or may not be workers.

[cwilso]

Upon review of Joe's comments:

Shared workers, while not having a per-node overhead problem, seem too global in scope.

Yep, I think we should use dedicated workers.

If there is one Dedicated worker per node, it's likely that Dedicated workers will have to be created in great quantity since Web Audio in many use cases is a very node-intensive system. The setup overhead for dedicated workers may be too high.

Just because the system should act like nodes are created this way, doesn't mean it shouldn't be optimized past that point (in actual implementation). In other words - although one could model a GainNode with a dedicated worker, it's unlikely an implementation would actually choose to implement it that way.

Dedicated workers imply postMessage-style communication via a MessagePort, which suggests unnecessary communication overhead since we don't need to allow passing of arbitrary data structures between node workers and the rest of the environment. We can focus on audio events and their results.

Actually, it's NECESSARY communication overhead (and with transferables, it shouldn't be too bad).

Going forward, Web Audio workers do not even need to communicate with arbitrary other pieces of the browser environment. They only need to communicate with the machinery running the audio graph.

Yes. And most of that communication should be through the i/o on onaudioprocess events.

[cwilso]

One enhancement to the proposal above from TAG: transfer sampleRate attribute to AudioWorkerGlobalScope. I'm not sure if we don't additionally need some AudioContext there, in order to be able to create other AudioBuffers, but I think we can avoid it. Eventually, we'll be able to create audio contexts in workers anyway.

[Constructor(DOMString scriptURL, optional unsigned long bufferSize)]
interface AudioWorker : Worker {
};

interface AudioProcessingEvent : Event {
    readonly attribute double playbackTime;
    transferrable attribute AudioBuffer buffer;
};

interface AudioWorkerGlobalScope : DedicatedWorkerGlobalScope {
  attribute EventHandler onaudioprocess;
      readonly attribute float sampleRate;
};

interface ScriptProcessorNode : AudioNode {
  attribute EventHandler onaudioprocess;
  readonly attribute long bufferSize;
};

partial interface AudioContext {
  ScriptProcessorNode createScriptProcessor(
    DOMString scriptURL,
    optional unsigned long bufferSize = 0,
    optional unsigned long numberOfInputChannels = 2,
    optional unsigned long numberOfOutputChannels = 2);
}

[karlt]

If the aim here is not to have ScriptProcessorNode JS run synchronously with the audio thread, so that it can process input to produce output without additional delay, then having the entire AudioContext in a worker (issue #16) is an alternative solution that will also provide more precise scheduling of timers for actions (see comment #24244648).

If AudioContexts will eventually be available in workers anyway, then is there any need to also have a worker ScriptProcessorNode?

[opera-mage]

How about the ability to run several ScriptProcessorNodes in parallel (to better utilize multi-core architectures)? Not sure if anyone is planning to do that, but I think that it's definitely an option.

[noteflight]

Another way to look at this is to ask if simply having JS audio processing (a la ScriptProcessorNode) take place in a Worker mitigates most of the problems with latency and "third-class citizenship" that result from allowing this type of processing to occur in the main thread. I do not believe we need to have full AudioContext access for this to work, although it might be beneficial. Most of the important scenarios for such processing do not require inter-node communication.

While a "shader language" type solution would be faster and better, Workers alone (even running in normal-priority threads) would be a huge step ahead of the current problems with main thread execution of JS.

[cwilso]

The more I've thought about this, the more convinced I've become that we should remove main-thread ScriptProcessors, and have ONLY audio-thread scriptprocessors. In response to opera-mage: the inter-thread communication causes so much POTENTIAL overhead, it's more likely to utilize multi-core WORSE. At the very least, you need to be trading latency for cores.

[jussi-kalliokoski]

have ONLY audio-thread scriptprocessors

:+1:

What about something like this:

partial interface AudioContext {
  AudioWorker createAudioWorker(String url);
  ScriptProcessorNode createScriptProcessor(AudioWorker worker);
}

partial interface AudioProcessingEvent {
  void finish();
}

The finish() method on the event would let us embrace the inherently asynchronous nature of JS, and let the worker say when it finished processing the data in the event. The audio engine can then ignore that event if it fails to deliver in time. This would enable the audio worker to spawn subworkers to split the work and what have you async stuff, but most of the time finish() would be called synchronously at the end of the event.

The reason that the AudioWorker is created separately from the script processor is that it allows the worker to be shared between multiple ScriptProcessorNode's, and also gives a reference to the worker for communication, such as initial setup or state changes (such as a sample upload).

For simplicity's sake I ignored a few things such as identifying for which node the event is, defining the inputs and outputs (and AudioParams), how to expose the data in the event and such, which we can discuss if this sounds at all sensible otherwise. :)

[cwilso]

I think we need to declare a lot of those things. And I'm not convinced sharing the worker across scriptprocessors (i.e. such that they can communicate) is a good idea.

On Sun, May 4, 2014 at 1:38 AM, Jussi Kalliokoski notifications@github.comwrote:

have ONLY audio-thread scriptprocessors

[image: :+1:]

What about something like this:

partial interface AudioContext { AudioWorker createAudioWorker(String url); ScriptProcessorNode createScriptProcessor(AudioWorker worker);} partial interface AudioProcessingEvent { void finish();}

The finish() method on the event would let us embrace the inherently asynchronous nature of JS, and let the worker say when it finished processing the data in the event. The audio engine can then ignore that event if it fails to deliver in time. This would enable the audio worker to spawn subworkers to split the work and what have you async stuff, but most of the time finish() would be called synchronously at the end of the event.

The reason that the AudioWorker is created separately from the script processor is that it allows the worker to be shared between multiple ScriptProcessorNode's, and also gives a reference to the worker for communication, such as initial setup or state changes (such as a sample upload).

For simplicity's sake I ignored a few things such as identifying for which node the event is, defining the inputs and outputs (and AudioParams), how to expose the data in the event and such, which we can discuss if this sounds at all sensible otherwise. :)

— Reply to this email directly or view it on GitHubhttps://github.com/WebAudio/web-audio-api/issues/113#issuecomment-42127725 .

[domenic]

@jussi-kalliokoski I'd be curious about some example code written with that proposal?

[jussi-kalliokoski]

@domenic I made two examples as gists:

• https://gist.github.com/jussi-kalliokoski/4e50f671d6796add4ce5 This one shows how you could share a stateless worker between multiple ScriptProcessorNode instances. It also shows how you could create subworkers to fill the buffers in parallel (asynchronously, of course).
• https://gist.github.com/jussi-kalliokoski/9cfb5661948a93a278e1 This one shows how you can use the worker instance to do cross-thread messaging with the audio worker.

P.S. Do note that in about 99.9% of the cases it would a pretty bad idea to parallelize noise generation like this. :)

[cwilso]

I think we need to have a discussion about what goals we have with ScriptProcessor node. The only way to eliminate latency is to eliminate asynchronicity, imco.

[russellmcc]

If the topology of the audio graph itself is very parallel, then adding parallel processing will probably reduce latency. Most professional audio programs utilize all cores available for this reason. That's not to say that parallelism is the best choice here, but it's something to consider.

As a developer (i.e., not an API expert by any means), I like jussi-kalliokoski's finish() proposal, but I'd prefer it to be "opt-in" (I'm not sure how this could work - maybe a makeAsync() call that would prevent finish() from being automatically called when the handler returns), just in case I forget to call finish in the normal case of just processing the event immediately.

[cwilso]

Most pro audio programs do exactly what we're doing - move UI and rendering to a different core than audio. The current ScriptProcessor design puts script-based audio processing back into the UI thread, which of course is a bad thing to do; however, any time you want to hop across thread boundaries it's going to cost you in latency, due to asynchronous design and ITC.

If your scenario really calls for parallel processing threads of audio (more than one core processing audio), you can still do that - fire up a worker thread, and use transferables to move data across.

The consensus on the call yesterday was to go with the in-audio-thread design, where zero latency is possible.

[russellmcc]

First of all, let me say that I'm really excited that the committee is getting the ScriptProcessorNodes off the main thread so soon.

I'm not sure exactly what the design you are proposing is, but afaik there is nothing in the current spec that would prevent implementations from running different nodes on different threads if the shape of the audio graph was quite parallel. I'm thinking in particular about the "Online music production tool" use case from the requirements doc - you'll likely have a large number of parallel tracks, each with their own processing graph. The lowest latency solution would be to spread the work across the cores so that everything happens in time. This holds true regardless if the nodes are native or script. I understand that currently web audio implementations use a single audio thread, but there's no reason this should always be the case. As a developer, I'd be less excited about an improved ScriptProcessorNode that would forbid such an optimization in the future.

[cwilso]

Well, it would be possible to arbitrarily fork nodes to different threads; however, the latency would likely radically increase because of such a design.

There's a (potentially significant) cost to traversing a thread boundary. At the very least, it turns a "give me the next processed block" operation into an asynchronous operation, and that could potentially be devastating to overall latency. Adding more cores to solve a problem only works if the cost of sharing the task (and recombining the answer) is less than the work that needs to be done. In particular, I disagree with your statement "the lowest latency solution would be to spread the work across cores so that everything happens in time." The lowest latency solution in most cases is to get the processing to all happen, in real time, in a single core, and not have to bounce audio buffers across threads asynchronously more than you have to.

Keep in mind you can ALWAYS fork your own worker threads and do your own processing in a separate thread, and postmessage the results back to an audio scriptprocessor node. We are not preventing that; we're just trying to recognize that for what it is; a not-incredibly-common scenario, and of much less benefit than getting zero-additional-latency JS-executing nodes.

[adelespinasse]

Keep in mind you can ALWAYS fork your own worker threads and do your own processing in a separate thread, and postmessage the results back to an audio scriptprocessor node. We are not preventing that; we're just trying to recognize that for what it is; a not-incredibly-common scenario, and of much less benefit than getting zero-additional-latency JS-executing nodes.

I have some questions about how this would work.

In an OfflineAudioContext, since the ScriptProcessorNode callback/event handler is asynchronous, I assume it could take as much time as it wanted to produce an output, and the audio engine would just wait for it.

Now, let's say you've decided to spin some of the processing off to another worker thread. You might do something like this (this is the audio worker's code):

var subworker = new Worker('subworker.js');
var subworkerResults;

subworker.onmessage = function(event) {
  subworkerResults = event.data;
};

onaudioprocess = function(event) {
  useSubworkerResultsSomehow(subworkerResults);
  doMoreProcessing(event);
};

The problem with the above is that there's no way to guarantee that the subworker sends the data before onaudioprocess needs it. Unless there's a graceful way to handle a lack of data, you'd really want onaudioprocess to block and wait for it. I'd like it to look more like this:

var subworker = new Worker('subworker.js');

onaudioprocess = function(event) {
  var subworkerResults = subworker.waitForMessage().data;
  useSubworkerResultsSomehow(subworkerResults);
  doMoreProcessing(event);
};

But nothing like waitForMessage() exists, as far as I can tell. Seems to me there should be... not just for AudioWorkers, but in the Worker interface and/or WorkerGlobalScope. (Where would I go to advocate such a thing? It wouldn't be part of the Web Audio API.)

Or is there some other way that I'm missing?

(For my own current purposes, alluded to in my comments on issue #308, I would actually want the audio worker to wait for a message from the main UI thread, not a sub-worker, but I think it would work the same way. It would just call the waitForMessage() method of AudioWorkerContext instead of on the Worker. I know that's weird, but I really do have a use for it. Unless possibly I can get access to DOM objects in the worker thread, but that seems less likely...)

[cwilso]

No, you're not missing something. But WaitForMessage() wouldn't solve this problem - it only makes it worse. Time - REAL time - keeps proceeding in a linearly progressing fashion. :) The best you can do is choose (based on knowing the environment) how much to keep as a latency buffer to avoid glitching - but if you don't have enough buffered up, glitches are going to happen.

You can't wait for a message to unblock onaudioprocess, because that would block the upstream calls - and at some point, you're just going to glitch. Instead, you need to make sure the pipeline keeps enough buffer latency to keep the pump primed, so to speak.

On Wed, Jul 30, 2014 at 4:40 PM, adelespinasse notifications@github.com wrote:

Keep in mind you can ALWAYS fork your own worker threads and do your own processing in a separate thread, and postmessage the results back to an audio scriptprocessor node. We are not preventing that; we're just trying to recognize that for what it is; a not-incredibly-common scenario, and of much less benefit than getting zero-additional-latency JS-executing nodes.

I have some questions about how this would work.

In an OfflineAudioContext, since the ScriptProcessorNode callback/event handler is asynchronous, I assume it could take as much time as it wanted to produce an output, and the audio engine would just wait for it.

Now, let's say you've decided to spin some of the processing off to another worker thread. You might do something like this (this is the audio worker's code):

var subworker = new Worker('subworker.js'); var subworkerResults;

subworker.onmessage = function(event) { subworkerResults = event.data; };

onaudioprocess = function(event) { useSubworkerResultsSomehow(subworkerResults); doMoreProcessing(event); };

The problem with the above is that there's no way to guarantee that the subworker sends the data before onaudioprocess needs it. Unless there's a graceful way to handle a lack of data, you'd really want onaudioprocess to block and wait for it. I'd like it to look more like this:

var subworker = new Worker('subworker.js');

onaudioprocess = function(event) { var subworkerResults = subworker.waitForMessage().data; useSubworkerResultsSomehow(subworkerResults); doMoreProcessing(event); };

But nothing like waitForMessage() exists, as far as I can tell. Seems to me there should be... not just for AudioWorkers, but in the Worker interface and/or WorkerGlobalScope. (Where would I go to advocate such a thing? It wouldn't be part of the Web Audio API.)

Or is there some other way that I'm missing?

(For my own current purposes, alluded to in my comments on issue #308 https://github.com/WebAudio/web-audio-api/issues/308, I would actually want the audio worker to wait for a message from the main UI thread, not a sub-worker, but I think it would work the same way. It would just call the waitForMessage() method of AudioWorkerContext instead of on the Worker. I know that's weird, but I really do have a use for it. Unless possibly I can get access to DOM objects in the worker thread, but that seems less likely...)

— Reply to this email directly or view it on GitHub https://github.com/WebAudio/web-audio-api/issues/113#issuecomment-50695322 .

[adelespinasse]

I was specifically talking about NON-real-time. OfflineAudioContext. Of course you can't block indefinitely in a real-time AudioContext. I don't see a reason why you couldn't in an OfflineAudioContext, especially since it sounds like all implementations are expected to be single-threaded.

[cwilso]

Hmm. This sounds like an issue for Web Workers, but I expect they'll be a bit aghast at the potential for deadlocking.

On Mon, Aug 4, 2014 at 7:07 AM, adelespinasse notifications@github.com wrote:

I was specifically talking about NON-real-time. OfflineAudioContext. Of course you can't block indefinitely in a real-time AudioContext. I don't see a reason why you couldn't in an OfflineAudioContext, especially since it sounds like all implementations are expected to be single-threaded.

— Reply to this email directly or view it on GitHub https://github.com/WebAudio/web-audio-api/issues/113#issuecomment-51064041 .

[andremichelle]

I admit I completely lost track on this issue. There seems to be discussions all over the place basically asking for the same thing. Excuse me if I am asking again for things that might already been reasonably discussed or even going to the next step of implementation. I currently have the feeling that having AudioContext exposed to Workers or having a ScriptNodeProcessor acting as a Worker is still out of the question. From my point of view this is still a very bad decision.

I am the lead developer of http://www.audiotool.com/app an online audio workstation developed in Flash. We are currently rewriting the entire application with http://www.defrac.com which compiles down to several platforms including Javascript. We managed to run the entire application (desktop plugins view, cables and sequencer) running with 60fps in WebGL. The only bottleneck we are facing is the audio processing running in the the main-thread. Obviously we could run the dsp code in a worker transferring the results back to the main-thread for the onaudioprocess event. But this adds at least one buffer-size of latency to the equation. Low latency is crucial for us. We will allow Midicontrols where people expect instantly to listen to the synthesiser sound when pressing a key or a modification of a sound when they turn a knob. The problem is that some operations are and always will be costly in the UI and cannot be easily outsourced to a worker as DOM changes (impossible), WebGL rendering (nearly impossible) and Data (Format) Modification. The current worker concept is just good for very simple number crushing where no complex object structure is involved.

@cwilso "Most pro audio programs do exactly what we're doing - move UI and rendering to a different core than audio." Name one. Seriously. Pro audio software do not work with workers rather than threads where data can be shared easily. I cannot think of an application structure in JS where we can move the UI and rendering to a different worker. This would mean that we have to send all javascript events as window, mouse and keyboard events to the worker, do some rendering (?) and return a Transferable containing all informations to alter the DOM, WebGL state and do Data Modification. Not to forget that we need to synchronise the format therefor having the same object structure available for each worker (which we were already able to do to a reasonable degree). Again: There are(!) ui tasks that will block the main thread for a even rather short time and it is nearly impossible to let a worker to it. This is where the audio becomes glitchy.

I am open to any of the proposals I have read so far. The easiest is to expose the AudioContext to the worker. The flashplayer can do it, so is it really such a big deal? Also having a ScriptNodeProcessor running as a worker is a solution but might blow up the API to communicate with that node. I have not thought it through. However definitely doable. The remaining proposals as transferring audio back and forth is just a hack, adds latency and even makes the entire implementation weak since the browser might ask for two audio renderings consecutively within a frame where we actually need more buffers to fix that, hence more latency.

To sum it up We can already synchronise the format of a song by JSON commands at two different locations as workers (even windows and live collaboration) to render the ui and do the audio processing. As long as we do not have real threads, that is the way to implement it. More complex ui tasks that need some time to process would be send later to the audio worker later but this is just for longer operations that can always happen. The important thing is that the audio NEVER becomes glitchy even the main thread becomes very busy for any reason. If this is a problem to you, you can always stop the audio-worker with an error when it takes way too long which might be an argument for a ScriptNodeProcessor running as a worker.

Just saying: It would be a shame if the html5 version is more glitchy than the flash version even we managed to use every feature that html5 gives us.

[cwilso]

I think you're completely misreading the situation.

I currently have the feeling that having AudioContext exposed to Workers or having a >ScriptNodeProcessor acting as a Worker is still out of the question.

Incorrect, on both accounts. We've continued this thread (ha!) on the public list: http://lists.w3.org/Archives/Public/public-audio/2014JulSep/0049.html is the last message I sent on AudioWorker, with a distinct proposal. Exposing AudioContext on Worker is a separate issue (https://github.com/WebAudio/web-audio-api/issues/16), but reading through the thread we had there, and remembering discussions, I don't recall any dissent to making this happen, it just isn't done yet.

Yes, it's true that in native programming you can utilize threads without memory protection; we can't do that in JS. But using a separate thread for audio moves audio processing to a different core (which is what I said) - and the design I've been espousing for ScriptProcessors would put them in that process, too. I was not telling you to do your UI/rendering in a Worker thread; I can see from how I put it that you might have taken it as "move your rendering out of the main thread where audio is, that's audio's thread" but that was certainly not my intent. UI and rendering happen in the main thread; even the initial design of web audio was intended to keep (most) audio processing OUT of the main UI thread that everything else happens in. ScriptProcessorNodes were the exception.

[cwilso]

Current proposal:

[Constructor(DOMString scriptURL, optional unsigned long bufferSize)]
    interface AudioWorker : Worker {
};

interface AudioWorkerGlobalScope : DedicatedWorkerGlobalScope {
      attribute EventHandler onaudioprocess;
      readonly attribute float sampleRate;
};

interface ScriptProcessorNode : AudioNode {
      readonly attribute double latency;
      readonly attribute AudioWorker worker;
};

interface AudioProcessingEvent : Event {
        readonly    attribute double      playbackTime;
        readonly    attribute AudioBuffer inputBuffer;
        readonly    attribute AudioBuffer outputBuffer;
};

partial interface AudioContext {
      ScriptProcessorNode createScriptProcessor(
        DOMString scriptURL,
        optional unsigned long bufferSize = 0,
        optional unsigned long numberOfInputChannels = 2,
        optional unsigned long numberOfOutputChannels = 2);
}

[andremichelle]

Thanks for clarifying! So there is hope after all :)

[joeberkovitz]

I propose that the new ScriptProcessorNode be renamed AudioWorkerNode, so that we do not have an incompatible change in an existing interface name. The two nodes operate quite differently and the new one's features are not a superset of the old one's features.

I don't have a position on retaining the existing ScriptProcessorNode implementation for a compatibility period. I note that there are good arguments for removing it immediately, but also that many developers have relied on it as the only means to an end despite its inadequacies so a change would break some apps in the world.

[srikumarks]

This is great to hear overall. Perhaps we should change the issue title to "A New Hope" :)

Some questions/comments regarding Chris Wilson's current proposal. Apologies if this duplicates what's already been discussed. -

1. I presume the "scriptURL" argument would be required to support all URL forms referring to a script? In particular, Object URLs? That would be needed to generate scripts on the fly.
2. How would lifetime of the node be managed in this case? If this is expected to be garbage collected like other nodes, then perhaps the "terminate()" method exposed in the Worker interface should be hidden? If not, then would terminate() be expected to get rid of the node? What would it do? When should Object URLs passed to nodes be "revoked"?
3. While it is a welcome change to do the js audio processing independent of the main thread, it has always seemed excessive to me to dedicate a worker to each node, especially if browsers are expected to run workers in separate threads, and especially for low cpu usage nodes. (1) It seems like one shared worker per audio context or per script node type (2) may be better from a system resource perspective.
4. For a script node to function as a usable source node type, instantiation/takedown should be light weight - ex: can be instantiated in predictably short time in response to a key press. Does this design choice limit how light weight it can be?
5. I presume the spec will detail how this node is expected to work with offline audio contexts. While I don't see any conceptual issues there right away, I just want to pop into view that these are expected to effectively run synchronously in the case of offline contexts. Lifetime management will also need to be detailed for this case. (3)
6. Lastly, it looks like “how to write an onaudioprocess” won’t change with this proposal. So we’re expected to fill the outputBuffer as usual? Is ownership of this buffer is automatically "transferred" to the audio thread afterwards then?

(1) It is unclear to me from the WebWorkers spec whether user agents are allowed to share a thread between multiple workers (wording: "or similar construct") and whether current browsers do such things.

(2): By "script node type", I mean script nodes that run a common onaudioprocess function (ex: compressor). Script nodes that depend on common source code can use a shared worker.

(3): Lifetime management of normal nodes for offline contexts still remains an open issue in the general case, but what we have now is acceptable.

[Theodeus]

Hello!

I'll also readily admit that I've lost track on this issue as well, but in the light of the recent email about the working group coming to a decision on this in a really near feature I just felt the need to raise two questions.

1. Are all the issues in Srikumar's list of concerns addressed? I think all of them are relevant.
2. One thing that I haven't had (and won't have in the near future) a chance to check myself yet is whether you can access and interact with flash objects from a worker thread? If you can, all is peachy, otherwise removing scriptProcessor from the main thread completely will make it harder (not impossible though) to pollyfill audio input for those browsers that doesn't support getUserMedia. The only solution I've been able to come up with in order to do this is to have a scriptProcessor poll audio data from a flash component whenever onaudioprocess happens. Something like this,

onaudioprocess -> poll input audio data from flash -> do processing, if any -> write to processor buffers

I guess this could be done in a worker environment with some proxying, but it doesn't really seem sane (super error prone with tons of overhead);

(worker) onaudioprocess -> poll JS proxy in main thread for data (main thread, proxy) poll input audio data from flash component -> post audio data to worker (worker) do processing, if any -> write to processor buffers

That said, I don't understand all the underlying architectural challenges in making this change to the API, but it seems to me like keeping the option for main thread processing would be a good idea, at least until all implementations has finished developing the worker based solution (and preferably then some..). Otherwise we'd have a period of total chaos for us developers where one implementation requires the worker approach while another don't support it at all. For those of us who's already put out a lot of projects using the old scriptProcessor, it will also mean a substantial amount of work to port existing applications to the new format, so we will probably see lots of broken sites for a while which might be bad for the cause of Web Audio in the long run.

Personally I've seen clients thinking twice about using Web Audio in production due to the recent breaking changes in Chrome where we first had the situation where it started throwing errors if stop was called twice on the same buffers, and then a few months later when support for legacy names and synchronous decoding of audio data etc. was removed.

I do understand and appreciate the need for the API to evolve and change, but from a purely self serving I-don't-know-what's-going-on-under-the-hood perspective I'd ask you to think twice about making breaking changes without a transition period for us developers to have a chance to deal with it. (Granted, the legacy names etc. was supported over a transition period, so thumbs up for that!)

[jussi-kalliokoski]

whether you can access and interact with flash objects from a worker thread?

I don't think that will happen. I don't even know which working group would be responsible for making that change. And, speccing and landing support for flash in workers would probably take far longer than implementing getUserMedia() across all browsers.

pollyfill audio input for those browsers that doesn't support getUserMedia

I'm quite positive that getUserMedia() will be supported in all the browsers that will support the web audio workers before the support for web audio workers will land.

I'd ask you to think twice about making breaking changes without a transition period for us developers to have a chance to deal with it

I think we can safely deprecate the ScriptProcessor from the spec without it affecting the transition period offered by the browsers. It's up to the browser vendors to provide a proper transition time based on their usage statistics, and I don't think any Web Audio-supporting browser will let the ScriptProcessor go without a proper transition time. :) Although, the ScriptProcessor has probably seen the worst transition period treatment of the things in Web Audio API due to its perceived low importance in the past, but I think we're far past those times today. :)

[joeberkovitz]

I can offer a few opinions and thoughts, based on the discussion so far, of course deferring to Chris Wilson where the nature of his proposal is concerned…

I presume the "scriptURL" argument would be required to support all URL forms referring to a script? In particular, Object URLs? That would be needed to generate scripts on the fly.

This hasn’t been discussed but I also presume/hope this would be OK as it is allowed for regular Web Workers.

How would lifetime of the node be managed in this case? If this is expected to be garbage collected like other nodes, then perhaps the "terminate()" method exposed in the Worker interface should be hidden? If not, then would terminate() be expected to get rid of the node? What would it do? When should Object URLs passed to nodes be "revoked"?

Great question, and I wonder the same. I’d like to suggest that the same node lifetime apply to such nodes as to any other AudioNodes, based on the audio processing graph properties.

I think that terminate() perhaps does not make sense in this situation and should throw an exception.

I don’t see that Object URLs should be revoked by anything except the application code that created them. I guess I don’t understand why there would be interaction between Object URLs and Web Audio.

While it is a welcome change to do the js audio processing independent of the main thread, it has always seemed excessive to me to dedicate a worker to each node, especially if browsers are expected to run workers in separate threads, and especially for low cpu usage nodes. () It seems like one shared worker per audio context or per script node type (*) may be better from a system resource perspective

The proposal is that each worker will be run directly in the audio thread to do its work. So there will not be thread-related overhead related to numbers of workers.

For a script node to function as a usable source node type, instantiation/takedown should be light weight - ex: can be instantiated in predictably short time in response to a key press. Does this design choice limit how light weight it can be?

I presume the spec will detail how this node is expected to work with offline audio contexts. While I don't see any conceptual issues there right away, I just want to pop into view that these are expected to effectively run synchronously in the case of offline contexts. Lifetime management will also need to be detailed for this case. (Lifetime management of normal nodes for offline contexts still remains an open issue.)

I agree that these should run synchronously for offline contexts, it is the cleanest interpretation. However “synchronously” should refer to whatever thread manufactures the output of an offline context, and until we clean up the offline audio context spec I don’t know if we can completely understand what the consequences of this position are.

Lastly, it looks like “how to write an onaudioprocess” won’t change with this proposal. So we’re expected to fill the outputBuffer as usual? Is ownership of this buffer is automatically "transferred" to the audio thread afterwards then?

I believe so; Chris, can you clarify?

(*) It is unclear to me from the WebWorkers spec whether user agents are allowed to share a thread between multiple workers (wording: "or similar construct") and whether current browsers do such things.

I think the fact that audio workers run in the audio thread makes this issue moot.

(**): By "script node type", I mean script nodes that run a common onaudioprocess function (ex: compressor). Script nodes that depend on common source code can use a shared worker.

Same as above — since they all run (synchronously) within the audio thread, simply having different audio workers running the same script should do the trick, shouldn’t it? Will that not be efficient enough?

. . . . . ...Joe

Joe Berkovitz President

Noteflight LLC Boston, Mass. phone: +1 978 314 6271 www.noteflight.com "Your music, everywhere"

[NorbertSchnell]

From our current experience with Web Audio API here at Ircam and by reading the above (and apart from all kinds of hacky workarounds due to missing but forthcoming features) there seem to be two major motivations for using a ScriptProcessorNode: (1.) synchronous (no-latency, "node-like") processing, adding functionalities that cannot be implemented with the current nodes (e.g. sample-wise recursive and spectral processing) (2.) asynchronous (sometimes just unpredictable and/or potentially heavy) processing that you don't want to interrupt the rest of your audio (nor anything else)

As a matter of fact, the current ScriptProcessorNode serves neither of these purposes very well and I would like to suggest these motivations as requirement for future solutions.

It very much seems like that the proposed "AudioWorker" would surely pull the ScriptProcessorNode out of its current dilemma and make it a useful well defined tool for requirement (2.). If I understood right, Chris (cwilso) even suggested that "AudioWorkers" could fit both requirements, but I still have a hard time to believe this. In fact, I am even not sure if I even would like to mix both requirements too much, but it obviously wouldn't harm to push "AudioWorkers" as far as possible in terms of low/no-latency as well as compile and runtime optimizations (JIT compilation, no-garbage, flushing denormals, etc).

However, concerning the synchronous requirement (1.), I cannot stop dreaming of a "shader-like" solution that would have everything to be heavily JIT compiled. Cycling 74' "gen~" in Max/MSP (http://cycling74.com/products/gen/) and GRAME's Faust (http://faust.grame.fr/) are two examples of possible formalisms. Apart from the possibility of radical optimization, the big advantage of such an approach would be that also the syntax can be well adapted to the task(s) of audio processing. Especially, Faust is a very good example of how a community of DSP specialists can adopt a formalism to contribute their knowledge to an open community of programmers – many of which just practice copy-and-pasting of free code snippets. I would like to have these specialists around Web Audio API.

On a more concrete note, I would have a little idea for the "AudioWorker" interface: What about renaming "playbackTime" (anyway a weird name) to "inputTime" and add another value named "outputTime" for the (envisaged :-) time when the produced frames would meet again the rest of the Web Audio graph (probably playbackTime + latency). Wouldn't that make things even clearer?

[joeberkovitz]

On Mon, Aug 11, 2014 at 8:16 AM, Joseph Berkovitz joe@noteflight.com wrote:

1.

I presume the "scriptURL" argument would be required to support all URL forms referring to a script? In

particular, Object URLs? That would be needed to generate scripts on the fly.

This hasn’t been discussed but I also presume/hope this would be OK as it is allowed for regular Web Workers.

Yes, this would work.

1.

How would lifetime of the node be managed in this case? If this is expected to be garbage collected like other nodes, then perhaps the "terminate()" method exposed in the Worker interface should be hidden? If not, then would terminate() be expected to get rid of the node? What would it do?

Great question, and I wonder the same. I’d like to suggest that the same node lifetime apply to such nodes as to any other >AudioNodes, based on the audio processing graph properties.

I think that terminate() perhaps does not make sense in this situation and should throw an exception.

I don't think we should change the model in that way. We should make terminate() kill the worker node, and it should be replaced with silence.

1.

When should Object URLs passed to nodes be "revoked"?

I don’t see that Object URLs should be revoked by anything except the application code that created them. I guess I don’t understand why there would be interaction between Object URLs and Web Audio.

Indeed. It could be revoked once the instantiation is finished (i.e. the contents is "downloaded"), just like a regular worker. Although as a side investigation, I note that Workers don't have an "onload" event, which is going to make "I'm ready to process now" challenging.

1.

While it is a welcome change to do the js audio processing independent of the main thread, it has always seemed excessive to me to dedicate a worker to each node, especially if browsers are expected to run workers in separate threads, and especially for low cpu usage nodes. ( ) It seems like one shared worker http://www.w3.org/TR/workers/#shared-workers-introduction per audio context or per script node type (*) may be better from a system resource perspective

The proposal is that each worker will be run directly in the audio thread to do its work. So there will not be thread-related overhead related to numbers of workers.

Precisely.

1.

For a script node to function as a usable source node type, instantiation/takedown should be light weight - ex: can be instantiated in predictably short time in response to a key press. Does this design choice limit how light weight it can be? 2.

I presume the spec will detail how this node is expected to work with offline audio contexts. While I don't see any conceptual issues there right away, I just want to pop into view that these are expected to effectively run synchronously in the case of offline contexts. Lifetime management will also need to be detailed for this case. (Lifetime management of normal nodes for offline contexts still remains an open issue.)

I agree that these should run synchronously for offline contexts, it is the cleanest interpretation. However “synchronously” should refer to whatever thread manufactures the output of an offline context, and until we clean up the offline audio context spec I don’t know if we can completely understand what the consequences of this position are.

They WILL run synchronously; that's the beauty here. It's all in one thread, and the callbacks are synchronous, so we don't have the same problem we have today (where the audio thread, while processing offlineaudiocontexts, would need to continuously block and spin its wheels until new data comes back from a script processor callback in the main thread).

1.

Lastly, it looks like “how to write an onaudioprocess” won’t change with this proposal. So we’re expected to fill the outputBuffer as usual? Is ownership of this buffer is automatically "transferred" to the audio thread afterwards then?

I believe so; Chris, can you clarify?

It's not transferred; it's in the same thread. It's just passed around. In fact, a major benefit is that you CAN reuse the same buffers, eliminating the need to allocate and deallocate new JS objects for every onaudioprocess callback; this will decrease GC pressure significantly.

(**): By "script node type", I mean script nodes that run a common onaudioprocess function (ex: compressor). Script nodes that depend on common source code can use a shared worker http://www.w3.org/TR/workers/#shared-workers-introduction.

No, shared workers are different; these would all be dedicted workers.

[srikumarks]

The proposal is that each worker will be run directly in the audio thread to do its work.

All I can say is this is AWESOME!! In this case, offline contexts would also work correctly automatically, as stated. Does this mean that the 128 frame delay that's currently forced upon script nodes no longer exists?

One of the previously stated reasons (by Chris Rogers, iirc) to not put the script processor into the audio thread is that unpredictable stuff like JIT/GC can result in audio break up. Is this now considered to be unimportant? Another reason stated (again iirc) was that it may not be secure to let JS code run in a high priority thread. That's now changed too? If these are no longer considered issues, it would be nice if someone on the team can summarize the reasons for the benefit of others. I've so far accepted these arguments with a "hmm, maybe", but have harboured a doubt that these considerations may become dated at some point.

[cwilso]

It will be true that GC COULD cause audio breakup. As the only JS code running in that thread is the audio worker, however, the author is in control of this. Don't allocate and release objects. :)

JIT shouldn't cause significant issues here.

This will have the side effect of demoting audio thread from high-pri to regular priority. However, 1) we can do this dynamically, and 2) it's on the Mac, apparently, that we could do this anyway.

On Mon, Aug 11, 2014 at 11:37 AM, Srikumar notifications@github.com wrote:

The proposal is that each worker will be run directly in the audio thread to do its work.

All I can say is this is AWESOME!! In this case, offline contexts would also work correctly automatically, as stated. Does this mean that the 128 frame delay that's currently forced upon script nodes no longer exists?

One of the previously stated reasons (by Chris Rogers, iirc) to not put the script processor into the audio thread is that unpredictable stuff like JIT/GC can result in audio break up. Is this now considered to be unimportant? Another reason stated (again iirc) was that it may not be secure to let JS code run in a high priority thread. That's now changed too? If these are no longer considered issues, it would be nice if someone on the team can summarize them for the benefit of others. I've so far accepted these arguments as "hmm, maybe", but have harboured a doubt that these considerations may become dated at some point.

— Reply to this email directly or view it on GitHub https://github.com/WebAudio/web-audio-api/issues/113#issuecomment-51821666 .

[adelespinasse]

Well, I posted a question to public-webapps about synchronous message passing. It re-started an ongoing, sometimes heated debate about synchronous APIs. Pretty interesting.

http://lists.w3.org/Archives/Public/public-webapps/2014JulSep/0226.html

It sounds like it's likely to happen at some point, although there could be restrictions designed to prevent deadlock.

Of course another way to solve the same problem (making sure an OfflineAudioContext waits for data it needs from other threads) would be to have an explicit way to pause the entire audio processing engine. (OfflineAudioContext only, of course! Not for real-time audio.) But I'm not going to push for that now, since I have some hope that it won't be necessary.

[NorbertSchnell]

The more I think of it, the more I think this is a great proposal that can really work out pretty well.

For me one of the main questions concerning the ScriptProcessorNode is: Could we implicitly flush denormals in this worker?

Secondly (and even if I slightly mixed up the playbackTime in my last post), I'd like to propose once more to include an explicit input and output time into the AudioProcessingEvent. That would make:

interface AudioProcessingEvent : Event { readonly attribute double inputTime; readonly attribute AudioBuffer inputBuffer; readonly attribute double outputTime; // or "playbackTime" if everybody prefers readonly attribute AudioBuffer outputBuffer; };

The current interface seems very "synthesis oriented" while for analysis tasks you probably what to refer to the input time rather than the output or "playback" (think, for example, of an onset detector of which you want to relate the detected onset times to the audioContext.currentTime at the input of the ScriptProcessorNode rather than the output). Calculating it from the playbackTime (playbackTime - latency) is somehow strange...

P.S.: We just implemented an application that could perfectly use this (using several workarounds in meantime): performing onset detection on several mobile clients, sending the onset times and a segments of samples to a server, perform cross-synthesis, get sample-accurate clock synchronization, analyse clock drifts over multiple onsets, get sample-accurate inter onset times... continuing towards source localization.

[cwilso]

On Wed, Aug 13, 2014 at 1:41 AM, Norbert Schnell notifications@github.com wrote:

For me one of the main questions concerning the ScriptProcessorNode is: Could we implicitly flush denormals in this worker?

Probably. Isn't most code compiled with DAZ (denormals are zero) anyway?

Secondly (and even if I slightly mixed up the playbackTime in my last post), I'd like to propose once more to include an explicit input and output time into the AudioProcessingEvent. ...Calculating it from the playbackTime (playbackTime - latency) is somehow strange...

With the new synchronous design, there is no built in latency in the node. There will be input latency in the hardware level, of course, and output latency in the hardware, but not in the script processor itself. I/O latency would have to be revealed in the getusermedia and context.destination nodes, if it can be.

[karlt]

Chris Wilson writes:

On Wed, Aug 13, 2014 at 1:41 AM, Norbert Schnell notifications@github.com wrote:

For me one of the main questions concerning the ScriptProcessorNode is: Could we implicitly flush denormals in this worker?

Probably. Isn't most code compiled with DAZ (denormals are zero) anyway?

JS arithmetic is specified as IEEE 754, but JS can detect and flush subnormals manually where necessary.

JS SIMD intrinsics have undefined FTZ behaviour. I don't know whether or not it is practical for implementations to provide FTZ for intrinsics but not scalar arithmetic.

[cwilso]

Current proposal is hosted on my fork (http://cwilso.github.io/web-audio-api/). Start at http://cwilso.github.io/web-audio-api/#widl-AudioContext-createAudioWorker-AudioWorkerNode-DOMString-scriptURL-unsigned-long-numberOfInputChannels-unsigned-long-numberOfOutputChannels to review the creation method, and the bulk of the text begins at http://cwilso.github.io/web-audio-api/#the-audio-worker.

Notably: AudioWorker is collapsed into AudioWorkerNode, and AudioParams are supported by Audio Workers.

[tambien]

I've got a use-case of the ScriptProcessorNode which doesn't seem to be addressed here. I've been using a square-wave combined with a script processor as a clock source for audio applications. this gives me a very accurate clock with the ability to have a tempo curve over time which can be scheduled and automated. It seems like the AudioWorkerNode would destroy this approach since it would add a lot of latency communicating between the worker thread and the main thread to scheduling AudioNodes. Take a look at my implementation:

https://github.com/TONEnoTONE/Tone.js/blob/dev/Tone/core/Clock.js

I never use these callbacks for any UI or DOM stuff, just as a callback which is invoked right before an event is scheduled to happen with the exact time that it's supposed to happen for scheduling AudioNodes. I haven't found another reliable approach which allows this level of scheduling, tempo curves, and automation. setInterval/Timeout are insufficient approaches for these needs.

[adelespinasse]

It seems like the AudioWorkerNode would destroy this approach since it would add a lot of latency communicating between the worker thread and the main thread to scheduling AudioNodes.

There's also inter-thread communication latency with the ScriptProcessorNode; it's just built-in, whereas with an AudioWorkerNode it would be explicitly handled by your code.

[cwilso]

Yotam, I think you're misunderstanding how the current design works. Although your clock works, it has significant latency in the pipeline - and in fact, if I understand your scenario, you would be radically better off with the Audio Worker approach.

Currently, the ScriptProcessorNode has to create fairly large buffer latency for its input AND its output - both of which (in Chrome's implementation) it double-buffers, in order to get around the cross-process-latency problem. Since the system needs to buffer up the input before it can ask for the output, your 2048 buffersize is equating to around 100ms of latency on a 44.1k device. In short, your ticks are going to be off by 100ms just due to the delay of getting the oscillator's data to the script processor. (So any tempo changes, etc. are delayed by 100ms.)

On top of that, since your code is getting a 2k batch of data at a time, your tick() callbacks are going to be jittered on 2k boundaries as well - of course, the time parameter to the call is correct, but they're delayed by 100ms, and they're jittered by up to 50ms. Your clock really isn't very accurate in this code.

By contrast, the Audio Worker proposal would have zero additional latency between an oscillator and the worker, and the block size is 128 so your jitter would be more on the order of 3ms. The only "unpredictable" delay would be the delay of the callbacks into main code - but you already have this problem. That's what you have a comment on line 6 that says "can cause audio glitches. use sparingly." With the Audio Worker, the callbacks would be postMessages, and they would not affect audio quality. There could of course be additional jitter and latency in those callbacks, depending on the congestion of the main thread.

In addition, I'd point out that AudioWorker explicitly enables creating AudioParams on the element (for clock frequency, e.g.) and you could skip the whole process of using a chained oscillator and just calculate phase in the audio process. Unless you're using the output directly - in which case, you're better off having this all in the audio process - you don't need the oscillator and your code to detect an uptick (which seems a bit unnecessary), you just need to calculate phase at every sample.

[tambien]

Thanks Chris. I'm glad you think that I'd be better off with AudioWorkers.

I wanted to clarify that for my Clock class, jitter is not a big concern. I chose a large buffer size to balance the callback rate against performance. The crux of the approach is that the callback is necessarily invoked BEFORE the AudioContext time in which the event is supposed to occur. If it's 100ms before or 1ms before are both fine as long as the time passed into the callback is accurate and has not passed yet.

If i'm understanding the new spec correctly, I don't know if i can guarantee that the callback will be invoked before the time the event is supposed to happen because of the variable latency of postMessage. is that correct? so it seems possible that the callback will be invoked with a time that has already passed.

The Clock is not scheduling itself within it's own thread, but synchronizing audio and automation events in the main thread. For a drum machine for example, i could set the oscillator to tick at a sixteenth note and use the time provided by the callback to trigger the start of my drum samples. I could time all of the events ahead of time, but the nice part of the Clock class is that I can use the most up to date application state before triggering a playback event, plus the ability to curve the tempo over time.

[cwilso]

I'm not sure I understand what you mean by "as long as the time passed into the callback is accurate and has not passed yet," but I think maybe you are sure of things that aren't really assured.

In the current ScriptProcessor, the time is "accurate" - that is, it accurately reflects when the buffer is intended to be played. It has not passed yet, UNLESS the main thread is congested enough to overrun the additional latency added by the script processor (and this glitching definitely can happen - the ScriptProcessor is an async callback across threads, after all) - but only because the callback is being called ahead of the main audio time, by bufferSize*2 in Chrome's case (and I believe in Firefox's as well). There is still variable latency (i.e. jitter) in the onaudioprocess calls - particularly when your main thread has a lot of processing going on. Do a graph of timing of your audioprocess calls, and you'll see this. It is possible, BTW, for the underlying system to batch up calls to onaudioprocess, for example if it wants to build up a larger latency to let the processor sleep for longer; that would mess with the consistency of those timing calls even more.

In the Audio Worker design, in order to fire an event in the main thread, you would have PRECISELY the same async callback across threads (except only one way, of course, in your scenario, so actually you could have less latency). You can have the same "assurance" that it will be called ahead of the main thread, simply by posting your message the same amount ahead of time - there's no difference in how likely you are to get the callback before the time passes (and thus how sure you are you would get it before the time passes), and no difference in how accurate the timestamp will be (against real time or the audio context currentTime, you just add bufferSize to it). On our last WG call, we discussed polyfilling the old ScriptProcessor design on top of Audio Workers; I won't get around to this for quite a while, but I'm quite convinced it can be done in a pretty straightforward manner.

Note, by the way, that you can't tell precisely what AudioContext time it is in "real time"; all you can tell (the AudioContext.currentTime) is what time you're scheduling next (the start time of the next buffer to be scheduled). We have an open issue on that conversion. (Issue #340.)

The design of "set the oscillator to tick at a sixteenth note and use the time provided by the callback to trigger the start of my drum samples" seems like a really bad idea, unless you simply put the triggering AND the sample playback into an Audio Worker. In the current ScriptProcessor design, that means you're going from the audio thread (oscillator) to the main thread (script processor) back to the audio thread (scheduling drum sample, but it probably wouldn't schedule in the same frame). Although it's not documented, I would expect we should have explicitly prohibited altering the audio node graph from within onaudioprocess, since it would have unpredictable results in the current frame. In the Audio Worker design, you would do this in one thread - oscillator feeds into worker, which copies samples from drum buffer.

On Fri, Sep 5, 2014 at 7:53 AM, Yotam Mann notifications@github.com wrote:

Thanks Chris. I'm glad you think that I'd be better off with AudioWorkers, but I'm still concerned.

I wanted to clarify that for my Clock class, jitter is not a big concern. I chose a large buffer size to balance the callback rate against performance. The crux of the approach is that the callback is necessarily invoked BEFORE the AudioContext time in which the event is supposed to occur. If it's 100ms before or 1ms before are both fine as long as the time passed into the callback is accurate and has not passed yet.

If i'm understanding the new spec correctly, I don't know if i can guarantee that the callback will be invoked before the time the event is supposed to happen because of the variable latency of postMessage. is that correct? so it seems possible that the callback will be invoked with a time that has already passed.

The Clock is not scheduling itself within it's own thread, but synchronizing audio and automation events in the main thread. For a drum machine for example, i could set the oscillator to tick at a sixteenth note and use the time provided by the callback to trigger the start of my drum samples. I could time all of the events ahead of time, but the nice part of the Clock class is that I can use the most up to date application state before triggering a playback event, plus the ability to curve the tempo over time.

— Reply to this email directly or view it on GitHub https://github.com/WebAudio/web-audio-api/issues/113#issuecomment-54635166 .

[andremichelle]

I have to ask an unloved question. When can we expect to see the implementation of AudioWorker in Chrome? We are really counting on it for our html version of audiotool.com We already managed to have the entire audio rendering running within a usual Worker, but it is not satisfactory yet. Even tiny delays in the main-ui thread causes glitches in the playback. Any hint would be appreciated.

[rtoy]

Chrome will implement it "soon". Best solution for now is to file a bug at crbug.com/new.

[andremichelle]

'Soon' sounds like probably by the end of the year, which is fine!

Out of curiosity: Why should I file a bug? It is not really a bug, right?

[rtoy]

I do not actually know when it will be done, but it will be implemented eventually.

And it's not a bug, but perhaps feature request. But all changes are supposed to have a bug report/feature request associated with it.

[cwilso]

@andremichelle Note that even with Audio Workers, implementing your entire audio rendering stack inside an Audio Worker is missing a lot of the value of Web Audio. It would be more efficient to create native nodes and use Workers just to do the things a custom node can't (e.g. different filtering or effects).

[sebpiq]

It would be more efficient to create native nodes and use Workers just to do the things a custom node can't

@cwilso that's something I've actually been wondering for a while ... I guess there is an overhead in processing when connecting an AudioWorker to native AudioNodes? So if you create many AudioWorkers the overheads probably sum-up. And if so, is there a point after which a solution "all implemented in a single AudioWorker" starts to be more efficient than "many AudioWorkers connected to many native nodes"?

[joeberkovitz]

Hi Sebastien,

The overhead in connecting an AudioWorker to native AudioNodes should be minimal, compared to the cost of processing samples inside an AudioWorker with JavaScript. Furthermore the cost of processing samples in a native node will be lower than an equivalent JS implementation.

So it’s almost always going to be a bad strategy (not just architecturally, but performance-wise) to put lots of different modular operations inside a single AudioWorker, if any of those operations could be implemented by a native node. It’s better to use the node graph for the purpose it was designed for.

. . . . . ...Joe

Joe Berkovitz President

Noteflight LLC Boston, Mass. phone: +1 978 314 6271 www.noteflight.com "Your music, everywhere"

On Oct 19, 2014, at 3:38 AM, Sebastien Piquemal notifications@github.com wrote:

It would be more efficient to create native nodes and use Workers just to do the things a custom node can't

@cwilso that's something I've actually been wondering for a while ... I guess there is an overhead in processing when connecting an AudioWorker to native AudioNodes? So if you create many AudioWorkers the overheads probably sum-up. And if so, is there a point after which a solution "all implemented in a single AudioWorker" starts to be more efficient than "many AudioWorkers connected to many native nodes"?

— Reply to this email directly or view it on GitHub.