Progress report

[goto920]

worklet branch

Working on bug fixes, the addition of AudioWorklet support, and performance improvement in the worklet branch.

So far,

1) AudioWorklet in AudioContext works fine but does not work in OfflineAudioContext. -- addModule() is OK, but new AudioWorkletNode does not recognize added module.

Current status: offline rendering is only possible with ScriptProcessorNode.

2) Bug: the audio is not played until the end in case of slow-down playback -- Adding silence to all the sources works at the cost of memory consumption -- Lowest playback speed is set to 50%. Then silence is

If the sources are four tracks of 50MiB (internally decoded in PCM), the total memory required becomes 100 x 4 + original 50 x 4 = 600MiB or more

3) Trying to reduce UI update by static rendering but the code became longer -- I am trying to apply sub-component, update by a prop, style

[goto920]

Offline rendering (for fast processing, recording, and export to file) with AudioWorklet is now working on Chrome.

Processed audio samples are stored in the AudioWorkletProcessor and transmitted as a large message (over 50MB) to the AudioWorkletNode. Firefox crashes when the message is transmitted.

AudioWorkletNode is undefined on Safari. That is the reason why a blank screen appears at startup. I prepared App.js and AppNoWorklet.js, which will be chosen automatically by a browser. (Searching for the correct way to choose or in src/index.js

In this app, the end of playback notice is fired first in ScriptProcessorNode or AudioWorkletProcessor.

Now only one source is added silence. It turned out it is not necessary to add silence to all sources after some experiments. (Both in the case of AudioContext and OfflineAudioContext). Also tried with OscillatorNode (never-ending stream), but it did not work as expected.

[goto920]

Good news.

OfflineAudioContext's output buffer is usable in the case of AudioWorklet. (Not in the case of ScriptProcessorNode.) oncomplete(e) { play or export (e.renderedBuffer); } works as described in spec. So, the recording function in AudioWorklet is not used anymore and the worklet is offloaded and Firefox does not crash:-)

Still, e.renderedBuffer is garbage in the case of ScriptProcessorNode. In OfflineAudioContext, ScriptProcessorNode works only as a source or the last node. According to my experiment, the output stream from ScriptProcessorNode is garbage if the node reads input. That causes the problem with e.renderedBuffer. Then I had to implement recording in ScriptProcessorNode.

[dj-fiorex]

Status of the app on major browsers:

Platform	Browser	Status
Android	Chrome	Live and offline playback doesn't work, with or withouth AudioWorklet
Android	Firefox	Live and offline playback work withouth AudioWorklet
IOS	Safari	AudioWorkletNode is not defined
IOS	Chrome	AudioWorkletNode is not defined
Desktop	Chrome	Live and offline playback work
Desktop	Firefox	Live and offline playback work

[goto920]

@dj-fiorex

Thank you for your status check.

I prepared a check script to find if AudioWorkletNode, context.audioWorklet.addModule() are defined. Now Safari and iOS Chrome (Safari inside) should work without AudioWorklet.

https://goto920.github.io/demos/simple-mixer/check-audioworklet.html

According to the results, there are three types of rendering engines.

Firefox Chrome: Ubuntu (Linux), Windows 10, macOS, Android Safari: macOS Safari, iOS Safari, iOS Chrome -- WIndows 10 (20H2), macOS (latest for M1 CPU), iOS (12 for iPhone 6), Android 7 (old phone)

Details are in docs/

[dj-fiorex]

@goto920 I made a commit that enables user microphone record, check it out! While we understand how to solve the delay problem of the user microphone, I put a slider to delay the start of the tracks and synchronize, let's say, the recording

[goto920]

@dj-fiorex

Thank you for the code. The recording function works on my Linux PC but the sound is somewhat of low quality. The sound from the speaker was not recorded. So, I think audio setups are needed including disabling echo cancellation.

I am integrating the recording function to src/App.js in the main branch. The audio setup and a callback function for mediaRecorder is moved to a handler method when the mic icon is pressed.

In playAB(), only mediaRecorder.start()/ stop(), and playback of recorded track will be added. For latency, comparing the recording of the sound from the speakers and source may be used for good estimation.