AudioWorkletProcessor should define settable properties as well as parameters

[bfgeek]

Designing something under pressure in 5 secs always has consequences. :)

I've talked to some other folks, I don't think we should add a onPropChange callback, instead just passed into a map in process. I.e.

registerAudioProcessor('foo', class {
  process(inputs, outputs, params, properties) {
    const freq = properties.get('freq');
  }
});

This simplifies the model a bunch as don't have to specify when callbacks are delivered, and matches the other worklet specs better.

[joeberkovitz]

@bfgeek I assume we'd still have the list of properties statically declared, yes?

@rtoy @padenot @hoch Any reactions?

[hoch]

This pattern makes sense to me. onPropChange is actually a stopgap method to pass the property change into the process method anyway.

To balance the discussion: in WebAudio, the change rate of properties are relative low compared to the parameters. Do we still want to pass this object for every render quantum?

[joeberkovitz]

I think it makes sense to pass it for every render quantum, otherwise the AWP has to sign up to provide some sort of storage between onPropChange() invocations and their actual use of the properties in process() which seems it would usually be a waste of developers' time.

[hoch]

I think we have reached to the consensus of using map-like objects for parameters/properties. However, the design still lacks of a way to send the data from the processor (AWGS) to the node (main global scope).

I think we can use sendEvent(any message)' from the processor and making the nodeEventTargetwith the handler ofonprocessorevent`. Here's a brief sketch:

partial interface AudioWorkletNode : AudioNode {
    attribute EventHandler onprocessorevent;
}

partial interface AudioWorkletProcessor {
    void sendEvent(any message);
}

One cosmetic issue here:

Asymmetrical interface between AWN and AWP: the communication from AWN to AWP is done by passing map-like objects into the process() method, however, channeling data from AWP to AWN will be asynchronous and event-driven. Are we fine with this inconsistency?

[padenot]

With the map-like model, how are concurrent memory access handled? Are we copying, transferring, sharing memory ?

[padenot]

There is a de-facto difference between the control thread (=main thread) model, that has an event loop, and the rendering thread model, that is iso-sychronous, but has some kind of message loop as well. Aligning the API surface while having different model underneath could be nice, but I don't think it's very important.

Passing all the properties of the AudioWorkletNode into each callback can have performance implication, if it's not specified with performance in mind (see my message above).

A the nice benefit of this solution (the map-like) is that you get the parameters synchronously with the audio processing, which is an handy feature.

Something that I don't quite like compared to what we had before is that you change from a model where the performance cost is per communication and where you get to know what changed, to a model where the performance cost is per callback and where you get the whole picture each time.

Now we can argue that this does not matter, it's just a couple pointers moving around. And they we'll see people hooking hundreds of parameters to a Worklet, most of which are going to be unmodified each callback. What if your audio processing algorithm requires computing something expensive if a particular parameter changes? Are you going to keep a copy of all the parameters inside the scope of the worklet, and compute a diff at for each block? I don't think this is a far-fetched use-case.

In any case, sending an event back makes sense, this is a pattern that is familiar to JavaScript developers.

[hoch]

Yes, I believe your concern is valid. FWIW, I believe the current proposal is 'structured cloning' of map-like objects. Throwing pointers back and forth between threads has not been discussed yet, but we can think about it.

I take your response as a we still do not have decision on this. I also do not mind having a generic sendEvent interface on both sides. Developers have to wrap this method to build their own properties and the designing handlers will also be more laborious, but it's not the end of the world.

[joeberkovitz]

Just catching up on this -- @padenot thanks for highlighting the loss of the ability on the part of the developer to know what has changed.

If change-awareness is desirable for some property, though, perhaps the developer could decline to expose it through the maplike and instead provide a setter/getter as part of their AWN subclass and implement the communication using sendData().

Marking for further discussion.

[joeberkovitz]

@hoch About sendEvent() on AWP: I don't think the asymmetry in communication is good. I believe we need async communication in both directions in order to allow custom-node-like behavior to be implemented. I thought we were going in the direction of exposing both sendData() and onsenddata on both the AWP and AWN -- in other words, just like sendMessage minus the Transferables.

[hoch]

@joeberkovitz Yes. I believe having sendEvent() on both sides is more important than having maplike properties and passing them into process() for the pseudo-synchronization. At least for our use case.

[joeberkovitz]

It seems that bidirectional sendData/ondata not only supports writable properties (albeit with more work by the developer) but also it supports arbitrary readable properties that can reflect AWP state, as seen in a number of use cases evidenced in native nodes.

Also, the sendData approach to writable properties allows the AWP to know via on the ondata handler when a property is mutated, which can be important for knowing when an AWP should take some actions e.g. recomputing cached data dependent on a changed property value (a common scenario).

To summarize my POV of the moment: the most conservative approach for V1 is to rely on sendData/ondata as a general purpose mechanism for implementing custom node methods and properties (both settable and not), and to consider the declarative-property idea as a potential future enhancement.

[hoch]

@joeberkovitz I think this is a duplicate of https://github.com/WebAudio/web-audio-api/issues/990.

[joeberkovitz]

Closing