feat: WASM support - Githubissues

filiph commented 6 months ago

Description

Creating this issue to discuss and track work on WASM support.

Requirements

[x] WASM compiled
[x] flutter_soloud can successfully be called from other WASM-compiled Dart projects
[x] verify pkg:flutter_soloud works on all major browsers

alnitak commented 6 months ago

I've made some attempts to compile on Linux. In the web directory there is the compile script. This succesfully generates the .wasm and .js in the web/build folder. I stopped here.

IMHO the next step should be to use dart:js_interop to interop Flutter with the created JS.

superciccio commented 6 months ago

Hi,

I tried to import the generated wasm into a simple html file.

This is the error I got,

Uncaught (in promise) TypeError: WebAssembly.instantiate(): Imports argument must be present and must be an object

what I noticed is that, it changes based on the object created for importObject

const importObject = {
  imports: { imported_func: (arg) => console.log(arg) },
};

     WebAssembly.instantiateStreaming(
      fetch("libflutter_soloud_plugin.wasm"),
      importObject,
    ).then(
          (obj) => console.log(obj),
);

My understanding is that we want to skip the js part and use the wasm directly within the app, anyone correct me If I got it totally wrong.

I havent tried to use/import the js part without the wasm.

cant fix, the stupid formatting :(

filiph commented 6 months ago

My understanding is that we want to skip the js part and use the wasm directly within the app, anyone correct me If I got it totally wrong.

I think that's the best approach at this stage, yes. The whole Flutter-on-web story seems to be (slowly but surely) approaching a time when apps and games are just compiled to WASM. So if it's possible for one WASM binary to use another WASM binary as a "DLL", then that's what we want.

alnitak commented 6 months ago

If I understand correctly, it is or it will be possible to use just the compiled WASM and use that with all the platforms? Making it possible to get rid of the FFI part? This would be wow!

superciccio commented 6 months ago

My understanding is that we want to skip the js part and use the wasm directly within the app, anyone correct me If I got it totally wrong.

I think that's the best approach at this stage, yes. The whole Flutter-on-web story seems to be (slowly but surely) approaching a time when apps and games are just compiled to WASM. So if it's possible for one WASM binary to use another WASM binary as a "DLL", then that's what we want.

have you got, projects/repo where to "steal" the build phase. Currently it build, but I'm not sure if the errors I am having are because of me or the wasm created...

also, I'm missing something else, how the flutter/dart will interact with it ?

https://flutterweb-wasm.web.app/ in the console is

how ? :D

@filiph

filiph commented 6 months ago

I'm afraid I don't know. I haven't yet touched WASM, and I'm afraid it'll be some time before I can make time. I'm trying to get someone from Flutter to chip in, but in the meantime I found these (that I intend to read for inspiration):

filiph commented 6 months ago

Word from the Flutter team:

in the current state of world, using dart:js_interop to interop Flutter is a viable route. In the future, there maybe new tooling to allow Dart FFI works directly on web, but it’s probably not happening anytime soon.

alnitak commented 6 months ago

Word from the Flutter team:

in the current state of world, using dart:js_interop to interop Flutter is a viable route. In the future, there maybe new tooling to allow Dart FFI works directly on web, but it’s probably not happening anytime soon.

Does this mean we should go for a federated plugin only for web? If so, I've tried this approach and taken some steps, but I haven't reached a good point. I could write these few steps in detail.

filiph commented 6 months ago

TBH, I don't know what it means. I don't think it necessarily means a need for a federated plugin?

wolfenrain commented 6 months ago

TBH, I don't know what it means. I don't think it necessarily means a need for a federated plugin?

You can have both internal binding implementation (IO and web) live in the same package, as long as the correct implementation is getting imported using this:

import 'package:flutter_soloud/io_implementation.dart' 
    if (dart.library.html) 'package:flutter_soloud/web_implementation.dart';

filiph commented 6 months ago

A few points that I need clarified:

Do we need to wire Flutter SoLoud to WebAudio (e.g. by programming a backend), or is this already done by the miniaudio backend that the package uses?
Is there a precedent of SoLoud (C++) with miniaudio backend running on the web as a WASM binary?
Is it possible to compile the flutter_soloud Flutter package and its Dart FFI SoLoud (C++) dependency into one WASM binary?
Is it possible to compile a Flutter app as a WASM binary (apparently yes?)?
Is it possible to somehow "link" the Flutter app WASM binary with the flutter_soloud Flutter WASM binary? OR, is every package (even an FFI one) just automatically compiled to one WASM blob?

/cc @johnpryan @alnitak

alnitak commented 6 months ago

@filiph

Yes, we don't need to write a new backend. Miniaudio acts as the audio backend for SoLoud C++ on all platforms.
I spent some time compiling an example provided by SoLoud and it works with miniaudio! I uploaded it here (click to start).
Yes, I already compiled what Dart FFI uses for binding C++. In the web dir there is a compile script that compiles sources stored in the src dir, bindings.cpp included which is the cpp source that contains all the FFI functions needed by flutter_soloud plugin.
Yes it is possible (I didn't try). But this option is still on the beta channel. It is supposed to compile the Flutter code to wasm and we just need to use the wasm lib from Flutter app in any way you compiled it.

I am not familiar with this kind of stuff, but my thoughts to proceed are:

compile src with emscripten which generates a .wasm and a .js
use a conditional import like @wolfenrain pointed out. So, when using the plugin on the web, we should import something like bindings_player_web.dart instead of bindings_player_ffi.dart.
in bindings_player_web.dart we should then use JS interop to call the .js functions generated with 1)

Problems:

the plugin will have to provide an already compiled wasm and js. Not really a problem but they should be linked to the plugin.
write bindings_player_web.dart code.
since the plugin uses an isolate to manage some functionalities, I don't know if this is feasible because AFAIK, JS doesn't support threads(?), but for sure there are other ways(?).

Don't take the things I have written as certainties, because as I have already written, they are not things I am familiar with.

superciccio commented 6 months ago

By the way, there is this issue opened, they also talk about audio in the comments

https://github.com/dart-lang/sdk/issues/46690

In the meantime I tried to get the js and wasm to load, where the js maps the functions inside the wasm.

The command I tried where copied from this thread https://github.com/jarikomppa/soloud/issues/356#issuecomment-1492750114

unfortunately that didnt work

Uncaught (in promise) TypeError: WebAssembly.instantiate(): Import #0 module="env": module is not an object or function
SoLoud_welcome.js:4226 Stack overflow detected.  You can try increasing -sSTACK_SIZE (currently set to 65536)
handleException @ SoLoud_welcome.js:4226
0029d602:0xb05 Uncaught (in promise) RuntimeError: memory access out of bounds
    at 0029d602:0xb05
    at SoLoud_welcome.js:691:12
    at callMain (SoLoud_welcome.js:4608:15)
    at doRun (SoLoud_welcome.js:4658:23)
    at run (SoLoud_welcome.js:4673:5)
    at runCaller (SoLoud_welcome.js:4585:19)
    at removeRunDependency (SoLoud_welcome.js:629:7)
    at receiveInstance (SoLoud_welcome.js:821:5)
    at receiveInstantiationResult (SoLoud_welcome.js:839:5)

so point 3 that @alnitak suggested I'm not sure will work, cause the js and wasm are still linked to each other and needed to be imported in order to work ( happy to be wrong )

multi thread in the js world is webworker, see this link for a short explanation (https://medium.com/techtrument/multithreading-javascript-46156179cf9a)

Like, this example (https://blog.logrocket.com/getting-started-webassembly-flutter-web/) calls wasm from Flutter but is very basic the audio is huge in comparison

alnitak commented 6 months ago

@superciccio I got the same error when trying to compile a standalone example. I had to add -sSTACK_SIZE=1048576 -sALLOW_MEMORY_GROWTH (maybe just the latter is needed).

The command I tried where copied from this thread ...

Did you use that because the web/compile script isn't working for you? What are the very few steps you took to start trying? I mean, is it worth creating a new project just for the web to try all this?

superciccio commented 6 months ago

I did wanted to try If I could use wasm + js in a simple web page.

Classic hello world, no flutter/dart involved

But I cannot get my head around.

In theory if we can get this first step, just loading with no errors is a very good progress ( maybe )

On Fri, 29 Mar 2024, 15:04 Marco Bavagnoli, @.***> wrote:

@superciccio https://github.com/superciccio I got the same error when trying to compile a standalone example. I had to add -sSTACK_SIZE=1048576 -sALLOW_MEMORY_GROWTH (maybe just the latter is needed).

The command I tried where copied from this thread ...

Did you use that because the web/compile https://github.com/alnitak/flutter_soloud/tree/main/web script isn't working for you? What are the very few steps you took to start trying? I mean, is it worth creating a new project just for the web to try all this?

— Reply to this email directly, view it on GitHub https://github.com/alnitak/flutter_soloud/issues/46#issuecomment-2027292280, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABBP2JX6NF7UIQCEI6I7GLLY2VRF7AVCNFSM6AAAAABESLHOH2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRXGI4TEMRYGA . You are receiving this because you were mentioned.Message ID: @.***>

alnitak commented 6 months ago

Maybe I could help with this. Save the following code to main.cpp in an empty dir. I have added some comments inside. Hope it help somehow.

main.cpp

``` #include // Bypass CORS: // For Chrome: edit shortcut or with cmd: C:\Chrome.exe --disable-web-security // For Firefox: Open Firefox and type about:config into the URL bar. // Search for: security.fileuri.strict_origin_policy set to false /// Run Chrome as below and open generated main.html /// chromium --disable-web-security --user-data-dir=~/chromeTemp --disable-site-isolation-trials /// Build with: /// emcc -o main.html ./main.cpp -O2 -s EXPORT_ALL=1 -sASSERTIONS -sSTACK_SIZE=1048576 -sALLOW_MEMORY_GROWTH int main(int argc, char *argv[]) { printf("START\n"); int n = 0; while (n < 3) { printf("loop %d\n", n); n++; } printf("END\n"); return 0; } ```

superciccio commented 6 months ago

Thanks.

I think we miss each other I was talking an hello world with soloud 😅

That's the all point. Your example ( main.cpp ) is not complex as ours.

Have you ever go it loaded in your browser ( wasm + is ) only ?

On Fri, 29 Mar 2024, 15:30 Marco Bavagnoli, @.***> wrote:

Maybe I could help with this. Save the following code to main.cpp in an empty dir. I have added some comments inside. Hope it help somehow. main.cpp

include

// Bypass CORS: // For Chrome: edit shortcut or with cmd: C:\Chrome.exe --disable-web-security // For Firefox: Open Firefox and type about:config into the URL bar. // Search for: security.fileuri.strict_origin_policy set to false

/// Run Chrome as below and open generated main.html /// chromium --disable-web-security --user-data-dir=~/chromeTemp --disable-site-isolation-trials

/// Build with: /// emcc -o main.html ./main.cpp -O2 -s FULL_ES2=1 -s EXPORT_ALL=1 -sASSERTIONS -sSTACK_SIZE=1048576 -sALLOW_MEMORY_GROWTH

int main(int argc, char *argv[]) { printf("START\n");
int n = 0;
while (n < 3)
{
    printf("loop %d\n", n);
    n++;
}

printf("END\n");
return 0;
}

— Reply to this email directly, view it on GitHub https://github.com/alnitak/flutter_soloud/issues/46#issuecomment-2027323224, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABBP2JTFDYGZYOI7GV7RSZ3Y2VUGJAVCNFSM6AAAAABESLHOH2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRXGMZDGMRSGQ . You are receiving this because you were mentioned.Message ID: @.***>

alnitak commented 6 months ago

I think we miss each other I was talking an hello world with soloud 😅

Oh, I see ;)

Have you ever go it loaded in your browser ( wasm + is ) only ?

Yes, the example I linked before is compiled mostly in the same way as my previous "Hello World", it uses just SoLoud lib. But if you mean our flutter_soloud C++ sources, I didn't.

superciccio commented 6 months ago

"But if you mean our flutter_soloud C++ sources, I didn't."

Yep I meant that 😪

On Fri, 29 Mar 2024, 15:54 Marco Bavagnoli, @.***> wrote:

I think we miss each other I was talking an hello world with soloud 😅

Oh, I see ;)

Have you ever go it loaded in your browser ( wasm + is ) only ?

Yes, the example I linked before https://marcobavagnoli.com/soloud/monotone.html is compiled mostly in the same way as my previous "Hello World", it uses just SoLoud lib. But if you mean our flutter_soloud C++ sources, I didn't.

— Reply to this email directly, view it on GitHub https://github.com/alnitak/flutter_soloud/issues/46#issuecomment-2027348288, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABBP2JWF2HZREUEMZ7JKZ63Y2VXBRAVCNFSM6AAAAABESLHOH2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRXGM2DQMRYHA . You are receiving this because you were mentioned.Message ID: @.***>

alnitak commented 6 months ago

I created a new plugin with only web platform to see what problems we could hit. Calling simple C functions (bound in the JS) from Dart seems to work, but things have gone horribly with some more complex C functions.

Some more info are written in the README and I have opened an issue where I explain better the problems.

mit-mit commented 5 months ago

Hi @filiph, can you check out the notes in https://github.com/dart-lang/sdk/issues/46690#issuecomment-2014675471, and then perhaps respond over there with feedback?

yeikel16 commented 5 months ago

since the plugin uses an isolate to manage some functionalities, I don't know if this is feasible because AFAIK, JS doesn't support threads(?), but for sure there are other ways(?).

Hi, I not expert in this tema but thread is posible by using Web Workers. Also supports shared workers, which allows you to synchronize across multiple tabs.

This approach by using Web Workers is implemented right now in the drift package. In his docs show how handle about.

NashIlli commented 4 months ago

+1

maks commented 4 months ago

Umm not quite @yeikel16 , the answer to multithreading for audio in the browser is not Web Workers, its Audio Worklets which are specifically:

The AudioWorklet interface of the Web Audio API is used to supply custom audio processing scripts that execute in a separate thread to provide very low latency audio processing.

Audio Worklets support wasm, there are lots of examples to be found.

This is actually the path I was going on to be able to use Dart to write audio processing functions but I don't think Darts WASM support currently allows for compiling standalone functions into WASM.

alnitak commented 4 months ago

Hi @maks, thanks for your feedback.

I have explored Audio Worklets despite my complete ignorance of JS. Managing our own audio callback with a custom Audio Worklet, implies not using SoLoud/miniaudio C lib and therefore a separate audio engine written in JS (if I am not wrong). But I saw that since some months, miniaudio supports Audio Worklets. When compiling to WASM, emscriten generates 2 more files: [libname].aw.js and [libname].ww.js I suppose they are the audio buffer and the audio callback. Still, I think they have to be managed with a custom JS code.

Currently I am working on a separate testing project to add web support to flutter_soloud. I reached some results, but I am stuck mainly on having a common way for all platforms to communicate between a thread and the main Dart Isolate. More info in this issue

maks commented 4 months ago

@alnitak yes sorry its taken me a while to respond here.

I had also previously looked into using miniaudio from Dart too but from the point of view of writing DSP code (ie code thats generating samples inside the usual callback function of miniaudio and other libraries like SDL, etc) but of course ran into the issue of not being able to do so due to lack of sync callbacks. My understanding of audio worklets is that they basically work the same way as other operating system audio APIs and libraries that wrap them such as miniaudio, SDL, etc, in that you provide a callback function implementation which gets called by the library/framework that needs to fill in a buffer of audio samples before it returns.

But getting back to the point at hand, I would suggest that perhaps using Soloud is not a good match for the case of browsers, because the web audio api provides much of the functionality that Soloud does. Instead it may be better to instead try to provide Solouds api built on top of web audio, though of course thats much more work than just doing a ffigen over solouds api.

Alternatively it may more sense to have a Dart package that provides a light abstraction over flutter_soloud and then it can make use of the existing new web package to call into the browser webaudio apis to implement that abstraction on the web platform.

alnitak commented 4 months ago

@maks agree with you: on the web the Audio Worklet is mandatory if you want to implement a DSP.

Maybe I am wrong, but by compiling SoLoud/miniaudio with emscripten to WASM and JS, your web app will use Web Audio. If this is true the SoLoud/miniaudio audio samples callback is used on the web (and maybe can be replaced by your custom DSP code instead of writing a custom audio worklet in JS).

Also if SoLoud/miniaudio uses Web Audio, then the compiled lib will have all the SoLoud functionalities it provides.

In short, the testing project works on the web. But I am having some problems like sending events from C to Dart (ie when a sound ends). I have seen the new NativeCallable but it's not compatible with web since it uses dart:ffi and my wish is/was to use one code-base for all platforms.

Alternatively it may more sense to have a Dart package that provides a light abstraction over flutter_soloud and then it can make use of the existing new web package to call into the browser webaudio apis to implement that abstraction on the web platform.

Agree with this, IMHO should be the right direction now.

alnitak commented 2 months ago

Some good news regarding this issue! It took some time, but I believe I have reached a stable point.

The isolate used to check the sound state and send audio commands to the engine has been removed. This change has already been pushed to the main branch.
The native C/C++ code has been compiled to WASM using emscripten.
To communicate from a native thread (the audio thread) to Dart, a Web Worker is created in C code using inline JS code with EM_ASM. This Web Worker sends events to Dart, currently just the sound ended event. The Worker is stored in the WASM Module at runtime. By doing this, the Worker can be easily used in native code to send messages and in Dart to receive them. For additional notes, please refer to WEB_NOTES.md.

Here is the web example. I have tried it on Firefox, Chrome, and on my phone browser. I think there is still some work to do. If anyone wants to try the code, it is in the web branch. I would appreciate any suggestions, especially regarding what I did using JS interop!

edit Has been noticed some cache problems on the web server where the example is stored which I don't know yet how to resolve. I have copied the example on another server but it is not always online: here

filiph commented 2 months ago

This is fantastic, Marco! Congrats on taking this exploration this far.

I have a question regarding the web example. It seems that if I play a sound on the web, at least part of the loading is synchronous, and it blocks the UI (as in, the UI is unresponsive for a few seconds). Is this something that can be worked around somehow, or is this inevitable when using WASM on the web?

alnitak commented 2 months ago

at least part of the loading is synchronous, and it blocks the UI

You are right Filip. This is due mainly at least to two reasons:

the time to load the file can be improved by compiling the WASM with "-O2" or "-O3" compiler arguments, but I saw errors rising when doing so.
I also tried to use a separate thread on C for loading, but I haven't succeeded yet (remember this in the last PR?).

So, I don't know yet if this can be easily fixed.

tejainece commented 1 month ago

Thanks for the amazing work. I will try it out this week.

alnitak commented 1 month ago

Thanks for the amazing work. I will try it out this week.

@tejainece please, for the web support, wait for 3.24.1 to land due to this issue. It's a matter of few time.

Also, if you do not mind, please try using the pitch branch, thanks a lot.

maks commented 1 month ago

@alnitak thanks for your great work on this! I'll try to get some time to try out your new work on an upcoming project. Would you have a quick summary of whats in the pitch branch? is it mainly just the pitch shift added in native c++ code?

Also 3.24.1 shipped today so that annoying web build bug is now fixed.

alnitak commented 1 month ago

Hi @maks, thanks for the interest.

@alnitak thanks for your great work on this! I'll try to get some time to try out your new work on an upcoming project.

This is great!! Keep me in touch for everything! Just one thing for iOS IPA builds if you will try. The README.md is not yet updated for this problem: native functions will be stripped out by XCode when optimizing the code, so you should follow this.

Would you have a quick summary of whats in the pitch branch? is it mainly just the pitch shift added in native c++ code?

This is the CHANGELOG, hopefully, with all the changes since the latest version v2.0.1 on pub.dev. In addition to the pitch shift, in that branch, there is also the feature to manage audio filters and their parameters more easily. And filters can be set also to single sounds (not on the Web for now).

Also 3.24.1 shipped today so that https://github.com/flutter/flutter/issues/153222 is now fixed.

Yes, I tried and it's working great now!

alnitak commented 1 month ago

Version 2.1.0 has finally been published with Web support.

I've also checked that the examples (and also flutter_soloud_example) work when compiled to WASM, ie with

flutter run -d chrome --web-renderer canvaskit --wasm -t lib/metronome/metronome.dart --release

so flutter_soloud can successfully be called from other WASM-compiled Dart projects.

After much time and effort, I'm delighted to finally close this issue.

alnitak / flutter_soloud

feat: WASM support #46

include