search

servo / media

Mozilla Public License 2.0
82 stars 54 forks source link

Improve WebAudio messaging model #344

Open Manishearth opened 4 years ago

Manishearth commented 4 years ago

While this isn't specced, I was told by @padenot that the ideal messaging model for WebAudio involves one where webaudio commands are batched up in between event loop ticks, and similarly any communication from webaudio to script should happen during these ticks.

To do this the webaudio impl will likely have to be rearchitected to use a message-passing model where the AudioContext is actually a thin wrapper around a sender and some state, and every audio context tick we update the script loop with things like the new currentTime value (etc).

Manishearth commented 4 years ago

Annoyingly this means that audioparam.value also needs to be updated each tick. The correct way to solve this is to mirror the audio param computations on either side, and run them whenever .value is read from.

Manishearth commented 4 years ago

So far I don't think it's that important for this to work, especially since it's not specced in any way which means that a lot of subtleties like "what happens if you do param.value=5; param.value" are unclear without opening another browser and trying it.

It might be worth switching the mutex-based impl over to message passing, at least.

padenot commented 4 years ago

Annoyingly this means that audioparam.value also needs to be updated each tick. The correct way to solve this is to mirror the audio param computations on either side, and run them whenever .value is read from.

Yep, this is what we do in Gecko as well. You'll find that you don't have to keep the full timeline, only future event and a single past event, to compute this.

padenot commented 4 years ago

So far I don't think it's that important for this to work, especially since it's not specced in any way which means that a lot of subtleties like "what happens if you do param.value=5; param.value" are unclear without opening another browser and trying it.

It's clear what should happen in this case (quoting from AudioParam.value):

Getting this attribute returns the contents of the [[current value]] slot. See § 1.6.3 Computation of Value for the algorithm for the value that is returned.

Setting this attribute has the effect of assigning the requested value to the [[current value]] slot, and calling the setValueAtTime() method with the current AudioContext's currentTime and [[current value]]. Any exceptions that would be thrown by setValueAtTime() will also be thrown by setting this attribute.

What is essential indeed is to not use locks to synchronize with a real-time thread.

Manishearth commented 4 years ago

Ah, I see, stuff is specced here: https://webaudio.github.io/web-audio-api/#control-thread-and-rendering-thread

gterzian commented 4 years ago

As part of this, it might make sense to also look into separating the backend from the rendering thread, since such a change, which eventually will be needed I think, would affect the communication model(likely requiring IPC-based message-passing).

In such a separated setup, in multiprocess mode the backend would be in it's own process, whereas the rendering thread(s) would always run in the script process where the DOM audio apis are used(and where a potential audio worklet would also run). This was discussed at https://github.com/servo/servo/issues/23807