JS WASM can't process more than 12 AdvancedAudioPlayers

[tralves]

Good day,

I noticed that when processing more than 12 AdvancedAudioPlayers in WASM, some players stop playing and eventually nothing plays. This does not happen on the native platforms I tested (Android, iOS and OSX).

I did extensive testing in these sample apps:

• https://elegant-keller-bbe4de.netlify.app/ (Press Test Sample Perf). Uses SO 2.5 linked to superpowered.bc. github
• https://lucid-newton-d41384.netlify.app/ Sample SP 2.5 app using standard superpowered.wasm.github.

I know that:

• When it happens, processStereo stops creating an output and returns false, and getPositionMs doesn't advance.
• With:
  • 10 simultaneous AAPs work fine,
  • 20 AAPs, sound stops after 9515 processStereo()
  • 30 APPs, sound stops after 3695 processStereo()
  • 40 AAPs, sound stops after 151 processStereo()
• It's not a performance issue. SP takes about 100ms to process over 1000s of audio;
• It's not a timing issue. I tested processing the audio outside the process() callbacks. Even with all the time in the world, it fails to process all the players;
• Not a memory issue. Doesn't matter if the WASM is given 32mb or 256mb;
• It's about the number of playing AAPlayers that are in the playing state at a given time. I can instantiate 40 AAPlayers but if it only plays 5 of them, everything goes fine.

I hope this helps and it's an easy fix. I am out of ideas. Tell me if you want me to try something. Tiago

[gaborszanto]

Is this with our JavaScript API or the (deprecated, experimental) bitcode version?

[tralves]

Tested in both cases. Same result.

On Mon, 28 Feb 2022, 13:16 Gabor Szanto, @.***> wrote:

Is this with our JavaScript API or the (deprecated, experimental) bitcode version?

— Reply to this email directly, view it on GitHub https://github.com/superpoweredSDK/web-audio-javascript-webassembly-SDK-interactive-audio/issues/30#issuecomment-1054421470, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAT27I3MELA3J3PIXWDAMEDU5ONVZANCNFSM5PRN36YQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you authored the thread.Message ID: <superpoweredSDK/web-audio-javascript-webassembly-SDK-interactive-audio/issues/30/1054421470 @github.com>

[gaborszanto]

Our JavaScript API grows memory dynamically and reports the allocations on the console. How big is the allocation when this happens?

[tralves]

After loading all the samples, this is the memory report:

WASM memory 1257903821594: 512 kb stack, 34 mb heap, 35 mb total.

When playing, I don't get any more memory growth reposts. It's also important to notice that if I create the same 20 Players (same memory allocated) but only ever processStereo() on 5 of them, the problem doesn't happen. That's why I guess is that it's not memory related.

[gaborszanto]

I just checked the source of https://lucid-newton-d41384.netlify.app/. I see the same AudioInMemory format is loaded into multiple players. We didn't think about this use-case. Please try a separate sample for each player, let's see if that's the problem.

[tralves]

I had already tested that, but just to make sure, I updated the https://lucid-newton-d41384.netlify.app/ app. With 20 samples, the sound stops. With 10 samples (just commenting out samples from const sampleUrls[], it works;

[gaborszanto]

playSynchronizedToPosition is meant to be used for a different use-case. What happens if you simply call play()?

[tralves]

I had tried that before and tried again. Same outcome. With 20 samples it stops playing, 10 samples works fine.

[gaborszanto]

How much time does it take for processing 20 players in the audio processing callback?

[gaborszanto]

How much time does it take for processing 20 players in the audio processing callback?

[tralves]

I am not at the computer, but my tests said around 140ms. I measured the time in the main thread because the performance API is not available in the audio worklet.

I think the problem in not in the processing time, but in the amount of "active" players. I tested having 20 players but only processing 5 of them per callback and the problem occurred.

On Thu, 10 Mar 2022, 06:50 Gabor Szanto, @.***> wrote:

How much time does it take for processing 20 players in the audio processing callback?

— Reply to this email directly, view it on GitHub https://github.com/superpoweredSDK/web-audio-javascript-webassembly-SDK-interactive-audio/issues/30#issuecomment-1063864043, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAT27I6D7MF7HT3OZMJGN5DU7HAUTANCNFSM5PRN36YQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you authored the thread.Message ID: <superpoweredSDK/web-audio-javascript-webassembly-SDK-interactive-audio/issues/30/1063864043 @github.com>

[gaborszanto]

Measurement on the main thread is impossible, it should be done directly in the audio processing callback. Web audio typically runs with the buffer size of 128, which is only 2.666 ms if the sample rate is 48000 Hz.

If the performance API is not available in the Audio Worklet, then for the measurement purposes please switch to ScriptProcessorNode by changing superpoweredwebaudio.js. Your processing will run in the main scope and you can properly measure how much percentage is spent with audio processing in the audio processing callback.

[tralves]

I didn't report that well. I am on vacations with the family, so I can't access the code.

That time is for processing batches of 200 (or 400, can't remember) process callbacks. At a samplerate of 44100, we need about 172 processing callbacks. So the test told me that 140ms to process more than 1s worth of sound means that the processing is probably fast enough.

I'll get back with more details when I get back.

On Fri, 11 Mar 2022, 04:23 Gabor Szanto, @.***> wrote:

Measurement on the main thread is impossible, it should be done directly in the audio processing callback. Web audio typically runs with the buffer size of 128, which is only 2.666 ms if the sample rate is 48000 Hz.

If the performance API is not available in the Audio Worklet, then for the measurement purposes please switch to ScriptProcessorNode by changing superpoweredwebaudio.js. Your processing will run in the main scope and you can properly measure how much percentage is spent with audio processing in the audio processing callback.

— Reply to this email directly, view it on GitHub https://github.com/superpoweredSDK/web-audio-javascript-webassembly-SDK-interactive-audio/issues/30#issuecomment-1064844284, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAT27I6QKGMB2SK2VXWDZ23U7LYHLANCNFSM5PRN36YQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you authored the thread.Message ID: <superpoweredSDK/web-audio-javascript-webassembly-SDK-interactive-audio/issues/30/1064844284 @github.com>

[gaborszanto]

That kind of measurement is not usable. Please measure how I described above.

[tralves]

Here goes the processing results of 20 samples until the sound stops:

WASM memory 1431559195757: 512 kb stack, 82 mb heap, 84 mb total.
processAudio false: 2.100000001490116
processor.js:70 processAudio false: 0.10000000149011612
processor.js:70 processAudio true: 3.899999998509884
2processor.js:70 processAudio true: 0.20000000298023224
processor.js:70 processAudio true: 0.29999999701976776
processor.js:70 processAudio true: 0.10000000149011612
processor.js:70 processAudio true: 0.19999999552965164
processor.js:70 processAudio true: 0.29999999701976776
3processor.js:70 processAudio true: 0.20000000298023224
processor.js:70 processAudio true: 0.5999999940395355
processor.js:70 processAudio true: 0.20000000298023224
processor.js:70 processAudio true: 0.29999999701976776
processor.js:70 processAudio true: 0.09999999403953552
2processor.js:70 processAudio true: 0.20000000298023224
processor.js:70 processAudio true: 0.3999999985098839
processor.js:70 processAudio true: 0.29999999701976776
processor.js:70 processAudio true: 0.20000000298023224
processor.js:70 processAudio true: 0.29999999701976776
processor.js:70 processAudio true: 0.30000000447034836
processor.js:70 processAudio true: 0.29999999701976776
processor.js:70 processAudio true: 0.5
processor.js:70 processAudio true: 0.20000000298023224
processor.js:70 processAudio true: 0.19999999552965164
2processor.js:70 processAudio true: 0.29999999701976776
processor.js:70 processAudio true: 0.30000000447034836
processor.js:70 processAudio true: 0.3999999985098839
processor.js:70 processAudio true: 0.4000000059604645
processor.js:70 processAudio true: 0.29999999701976776
processor.js:70 processAudio true: 0.20000000298023224
processor.js:70 processAudio true: 0.10000000149011612
processor.js:70 processAudio true: 0.19999999552965164
processor.js:70 processAudio true: 0.3999999985098839
processor.js:70 processAudio true: 0.09999999403953552
processor.js:70 processAudio true: 0.4000000059604645
processor.js:70 processAudio true: 0.10000000149011612
processor.js:70 processAudio true: 0.30000000447034836
processor.js:70 processAudio true: 0.20000000298023224
processor.js:70 processAudio true: 0.6999999955296516
processor.js:70 processAudio true: 0.29999999701976776
processor.js:70 processAudio true: 0
processor.js:70 processAudio true: 0.30000000447034836
processor.js:70 processAudio true: 0.19999999552965164
processor.js:70 processAudio true: 0.20000000298023224
processor.js:70 processAudio true: 0.3999999985098839
2processor.js:70 processAudio true: 0.29999999701976776
processor.js:70 processAudio true: 0.19999999552965164
processor.js:70 processAudio true: 0.20000000298023224
processor.js:70 processAudio true: 0.29999999701976776
processor.js:70 processAudio true: 0.5
processor.js:70 processAudio true: 0.19999999552965164
processor.js:70 processAudio true: 0.5
processor.js:70 processAudio true: 0.10000000149011612
processor.js:70 processAudio true: 0.20000000298023224
processor.js:70 processAudio true: 0.19999999552965164
processor.js:70 processAudio true: 0.5
processor.js:70 processAudio true: 0.30000000447034836
processor.js:70 processAudio true: 0.19999999552965164
processor.js:70 processAudio true: 0.30000000447034836
processor.js:70 processAudio true: 0.19999999552965164
processor.js:70 processAudio true: 0.30000000447034836
processor.js:70 processAudio true: 0.20000000298023224
processor.js:70 processAudio true: 0.10000000149011612
processor.js:70 processAudio false: 0.10000000149011612
processor.js:70 processAudio false: 0
processor.js:70 processAudio false: 0.10000000149011612
processor.js:70 processAudio false: 0
processor.js:70 processAudio false: 0.19999999552965164
processor.js:70 processAudio false: 0
processor.js:70 processAudio false: 0.10000000149011612
processor.js:70 processAudio false: 0.20000000298023224
processor.js:70 processAudio false: 0
processor.js:70 processAudio false: 0.20000000298023224
processor.js:70 processAudio false: 0
processor.js:70 processAudio false: 0.10000000149011612
processor.js:70 processAudio false: 0
processor.js:70 processAudio false: 0.19999999552965164

These times are measured in ms, so the speed is more than enough. Something else is causing the processing to stop when there are these many players.

[gaborszanto]

So this was all measured with ScriptProcessorNode, and which API provided the times exactly?

[tralves]

Ah, missed your question. Yes, it is using ScriptProcessorNode. Each processAudio consists of processStereo() on 20 players. Players started with simple player.play();.

This is the commit for this test: https://github.com/tralves/example_advancedaudioplayer_perf/commit/9aa9cb6f44b098bc1c57d9a4dff315f50bf1443c

Btw, with fewer players, times were practically the same. There is something weird when playing lots of players.

[gaborszanto]

Please add some code that checks the contents of outputBuffer.array[0...buffersize * 2] for not a number values, after each player.processStereo. Maybe one of the players goes rogue and puts invalid audio into the buffer.

[hugorodrigues]

I was able to reproduce this issue while opening #32, very similar behaviour.

[gaborszanto]

Discovered a memory allocation bug. Please update to the latest version and check if it fixes it.