AudioWorklet support via SEPIA Web Audio?

[fquirin]

Hi everybody,

I just saw this project and thought it was very interesting and fits quite well to a library I've just released :slightly_smiling_face: . For my SEPIA Open Assistant project I've built the SEPIA Web Audio Library that can handle custom audio pipelines with AudioWorklet and Web-Worker support. There is pretty good WASM support as well since the resampler for example can use Speex via a WASM module.

The library has a module that interfaces with Vosk via the SEPIA STT-Server (a WebSocket streaming STT server). Currently I prefer to host Vosk on a Raspberry Pi 4 instead of running it on the client, but I'm pretty sure much of the code could be reused :smiley: .

Let me know if this sounds interesting to you and I can help to get started!

[ccoreilly]

Hi! Thanks for your suggestion, I have a half baked AudioWorklet implementation and would like to move away from scriptProcessor but I have had very little time in the last months. I was looking at your library and it looks good, it'd be nice to have fallbacks. I'll have a deeper look and see if it fits well :)

I was unaware of the SEPIA Framework and I see you've built mostly everything from scratch, nice job! Tangential but you might find Ona, another project I started, interesting. It builds upon Mycroft and the many existing skills though.

[fquirin]

... would like to move away from scriptProcessor

It's still everywhere although it was deprecated years ago :sweat_smile: , one reason I built the new web audio lib. Its not surprising though because it was deprecated before the alternative was working :man_facepalming: . 2020/2021 was a good time for AudioWorklet since Safari finally got support and Chrome 90 fixed a lot of bugs on mobile ^^.

The lib has some examples for worklet and worker modules in the 'modules' folder, shouldn't be too hard to get started ... hopefully :-)

I was unaware of the SEPIA Framework and I see you've built mostly everything from scratch, nice job! Tangential but you might find Ona, another project I started, interesting. It builds upon Mycroft and the many existing skills though.

Ty. My hope is that we can get better compatibility between the open-source projects, especially Rhasspy, Mycroft and SEPIA. I'm planning to work on this more focused after the next update :+1:

[pavlonadolynskyi]

@fquirin @ccoreilly I tried AudioWorklet and it cuts audio in Desktop Safari, had to switch to scriptProcessor. Found exactly same problem https://stackoverflow.com/questions/73365059/audioworklet-playback-cut-in-half-on-iphone-devices-and-safari

[fquirin]

@pavlonadolynskyi yes, I just tested it on Mac OS 12 and iOS 16.0.3 and can confirm that it's broken now. iOS Safari will at least record well for about 4s before resetting and continue. Mac Safari is completely messed up, jumping around, skipping frames etc.. I've also tested the legacy script processor and it has very bad recording quality on Mac for some reason. I've even tested the latest Safari developer preview on Mac, same issues.

Welcome to the world of Safari. Sometimes I feel this browser is even worse than IE11 😒. I'm thinking about dropping support at least once a month, it's just not possible because Apple is blocking all other browsers on mobile 😑.

Btw you can reproduce my experiments with the SEPIA app at https://sepia-framework.github.io/app/index.html . Just skip login, open the settings and look for "media devices" and then "microphone setup". "Source processor: Legacy .." is the deprecated scriptProcessor.

[fquirin]

I've create a report via Apples Feedback Assistant. You should be able to see it after login (! another annoying habit with Apple) here: https://feedbackassistant.apple.com/feedback/11684212

[pavlonadolynskyi]

@fquirin thanks!