search

ccrma / webchuck-ide

A Web-Based Programming Sandbox for ChucK
https://chuck.stanford.edu/ide
MIT License
12 stars 2 forks source link

Crackle / jitter in audio on iOS when tapping on screen #16

Closed musicmichaelc closed 1 month ago

musicmichaelc commented 6 months ago

The audio is audibly interrupted / distorted when tapping on certain UI elements in the WebChucK-IDE in iOS (tested using 12.9 inch, 2nd generation, iOS 17.2).

For example, if I take the helloSine.ck, tap on the "add shred" button to run the code, then tap on the code editor to modify the frequency, there is a crackle / jitter in the audio. After tapping "Replace shred", when I tap back into the code's text field, the crackling sound occurs again.

I was not able to duplicate this issue on my laptop (Linux UbuntuStudio 22.04.3 LTS (GNU/Linux 6.5.0-17-lowlatency x86_64) using Firefox), therefore I suspect it might be an issue particular to iOS.

However, when I use only keyboard commands (editing code with keybindings only and using keyboard shortcuts to add, replace and remove shreds) on iOS (using a bluetooth connected keyboard), the issue doesn't show up -- it seems as far as I can tell to only occur when the focus is shifted from other UI elements (such as the buttons on the toolbar or the menu items) back to the text field with the code.

terryzfeng commented 5 months ago

Thanks for sharing this behavior! I'm testing on my iPad Pro 12.9 running iOS 17.3.1 and can't reproduce the same issue. Are you using Safari?

Glitches can happen from time to time on mobile/tablet platforms, especially when running more computationally heavy code. Do you experience this on mobile too? Does this happen in the WebChucK IDE 2?

Thanks so much for your feedback!

musicmichaelc commented 4 months ago

I tried it with several browsers, namely Firefox, DuckDuckGo, Safari, Brave, Aloha and Ghostery Dawn (though as far as I know, all browsers on iOS use the same underlying framework anyway). I tested it on the IDE 2, though it did this on the earlier version too. On the mobile phone was also jitter...

Indeed it seems to be a mobile/tablet issue, and perhaps newer devices handle it better (mine is a 2nd generation iPad, so it is a bit older). However, I didn't need to run anything computationally heavy in order to cause the jitter -- just a simple sine wave and tapping on different UI elements (buttons / textfields) can already do that...

Other than that, the Web IDE generally seems to work very well, and as I said before, I can mostly avoid the jitter by using keyboard commands instead of tapping. It just seems odd that screen tapping could cause such major interruptions in the audio.

terryzfeng commented 4 months ago

Thanks for all the investigation! There are probably some performance improvements that can be made to the IDE (I have a hunch on a couple of things). Perhaps disabling some HID monitoring or ChucK time display could help...

Additionally, it is known that Web Audio API (which is how WebChucK does audio) isn't as optimized on Safari (and webkit architecture) browsers. If you could test this stress test page and let me know if you have the same issues, that would be immensely helpful!

https://ccrma.stanford.edu/~tzfeng/projects/stress/

terryzfeng commented 1 month ago

Unfortunately, there's not much I can do directly to improve the WebChucK AudioWorklet & Web Audio performance at this time, but Safari performance limitations (MacOS/iOS) are a known issue.

I've linked another thread for now and hopefully we can address this at some point!

https://github.com/WebAudio/web-audio-api/issues/2391