search

dart-lang / sdk

The Dart SDK, including the VM, dart2js, core libraries, and more.
https://dart.dev
BSD 3-Clause "New" or "Revised" License
9.95k stars 1.53k forks source link

[ffi/js-interop/wasm] Unified API for using dart:ffi and JS Interop with WASM #46690

Open dcharkes opened 2 years ago

dcharkes commented 2 years ago

When developers have native code that lends itself to compilation to WASM, this native code could be used in Dart Web through JS interop.

However, we do not have a unified API for interacting with the same native library on Dart Native through dart:ffi, and on Dart Web through JS interop.

We envision such native libraries to be used in the following way:

Compile to WASM only for Dart Web, and keep using C dynamic libraries for Dart Native:

  1. Dart Web: Dart <-JS interop-> JS <-> WASM
  2. Dart Native: Dart <-dart:ffi-> C (package:ffigen can auto-generate bindings)

First, it would convenient if we would have a code generator that would generate the JS interop bindings for a package compiled to WASM.

Second, it would be convenient if we had some kind of unified API that could both target dart:ffi's types and the generated WASM bindings, if it would be at all possible to have a unified API. The language runtimes of C and Assembly are different, so it might not fully be possible. So this requires exploration.

(An alternative approach would be to also combine the C libraries to WASM for Dart Native:

  1. Dart Web: Dart <-JS interop-> JS <-> WASM
  2. Dart Native: Dart <-package:wasm-> WASM (package:wasm internally does: Dart <-dart:ffi-> C <-Wasmer runtime-> WASM.)

The downsides of this approach is having to ship a WASM runtime such as Wasmer with the Dart Native apps, and it might be slower than just using the native library. The potential upside is that the native library does not have to be cross-compiled.)

Community contributions are welcome.

Filing an issue because we don't have one tracking this yet.

unicomp21 commented 2 years ago

Thanks @dcharkes !

The browser case is what I'm especially interested in.

I wonder if the codegen for rust typescript bindings could be useful here? It's a somewhat indirect approach (ie targetted code must be wrapped w/ rust first), but might be a way to approach the goal?

liamappelbe commented 2 years ago

https://github.com/dart-lang/wasm/issues/34 is tracking package:wasm's support for web. No ETA on that, but the more user interest there is, the sooner we'll start working on it.

safasofuoglu commented 2 years ago

I'm fairly interested in getting this right. A few dozen hours of research led me to believe a truly unified ffi might be possible and could bring additional benefits. Here is my imagination of the lifecycle:

Packaging and distribution

Building

Running

Before going deeper, I wanted to share this excerpt to find out whether I'm moving in the right direction.

@dcharkes @mraleph @mit-mit @askeksa-google would love your thoughts.

mraleph commented 2 years ago

@safasofuoglu

The compiled wasm artifact of the native library can be distributed within the pub package alongside dart bindings. This could become the only/recommended way of distributing dart native extensions.

When building for native, the library wasm file is AOT-compiled by the sdk and post-processed to expose the initial C API of the original library.

I don't think that's a good way - WASM gives you certain degree of portability, but it abstracts you from the underlying platform and puts your code into a sandbox - meaning that your code can no longer access specific platform features or APIs. This is rather limiting e.g. cryptographic library might contain hand written platform specific assembly for performance and security reasons, or library might be talking to OS specific APIs. Source code (or platform specific binaries) is a much more flexible distribution format for arbitrary libraries than WASM.

safasofuoglu commented 2 years ago

@mraleph thanks.

your code can no longer access specific platform features or APIs

This can be (partially) resolved by linking with WASI-libc (and WASI polyfills for the web). It won't support OS specific APIs.

might contain hand written platform specific assembly for performance and security reasons,

Hand-written assembly is a no-go, but most SIMD intrinsics can be ported. I was able to compile the complete dart sdk with all dependencies (boringssl, zlib etc) through emscripten, with simd intrinsics where supported. Unable to run anything, but it compiles.

Source code (or platform specific binaries) is a much more flexible distribution format for arbitrary libraries than WASM.

No doubt. However, in case of pub, distributing binaries is a problem (https://github.com/dart-lang/sdk/issues/36712) and distributing source code means additional compiler dependencies and a wrapping build-runner (https://crates.io/crates/cc).

Hence, I was wondering whether a unified wasm+ffi approach might cover a good percentage of use cases to be preferable.

dvc94ch commented 2 years ago

This project already supports generating dart and js for rust projects [0]. I'd suggest going with compiling flutter/dart to wasm and then linking using the wasm linker is probably a better long term solution. Once flutter/dart compile to wasm I can add support to ffi-gen. For this to work nicely there needs to be a way to link flutter/dart binaries using the system linker as in https://github.com/dart-lang/sdk/issues/47718

badlogic commented 1 year ago

We have a use case for a unified API for using dart:ffi on both web and non-web platforms. We have a dart:ffi binding for our Spine Runtimes, binding a C++ library to Dart, see https://github.com/EsotericSoftware/spine-runtimes/tree/4.2-beta/spine-flutter (it's a WIP).

Ideally, we could make this work for the web as well by compiling the C++ code to WASM, while being able to use the existing dart:ffi based bindings.

There's a very interesting third party approach to do exactly that: https://github.com/EPNW/web_ffi/

Sadly, it has a few (minor) limitations, e.g. no struct returns. But the basic principle is exactly what'd we'd expect it to work like. We can't currently use web_ffi, as the exposed API is not compatible with the latest dart::ffi.

maks commented 1 year ago

Now that there is experimental support for Wasm as a target in Flutter, I wanted to ask if there is any updates on being able to use FFI from Dart targeted to Wasm. The use case I have in mind is that I have an open source C library (audio synth) that I currently use via FFI in a Flutter app on Linux, MacOS, Android, iOS and would now love to be able to try to target web as well, but currently I don't think there is any support for doing FFI from Dart (running compiled to wasm) to call a C lib also compiled to Wasm.

liamappelbe commented 1 year ago

The use case I have in mind is that I have an open source C library (audio synth) that I currently use via FFI in a Flutter app on Linux, MacOS, Android, iOS and would now love to be able to try to target web as well

I don't know anything about using FFI from Dart that's been compiled to wasm, but I do know about doing audio on the web. I think the limitation you're going to run into is that the only API your library could be wrapping is the Web Audio API, which is pretty high level (you wire up graphs of audio processing nodes to produce effects). There's no low level audio API like on the other OSs (where you just have to fill an audio buffer every N milliseconds) for your C library to wrap. Wasm doesn't give you any access to any APIs other than what the browser provides, so the audio library you're calling would have to have explicit support for wasm via the web audio API.

maks commented 1 year ago

@liamappelbe its quite off topic here, but just actually I've spent a bit of time myself looking into WebAudio API and there is in fact functionality in the API to do exactly that: low level, per frame low latency audio generation with audio worklets and a great example of doing so is Peter Salomonsens excellent work which actually already uses assemblyscript compiled to wasm to do exactly that.

But as I said, while the above is quite off-topic, its actually possible already right now to have C libs compiled to wasm generate low latency audio via WebAudio. However what I'm asking about is future support for this once Dart's wasm support progresses further and it becomes the preferred deployment target for Flutter webapps, it would be nice to have support for FFI interop.

maks commented 1 year ago

Also @liamappelbe I think you might be pulling my leg a bit about not knowing about FFI with Dart Wasm since you literally (co)wrote the article about Darts initial wasm support 👍🏻 with Michael 🙂 and about Webaudio as I've come across your cool work before with https://onlinesequencer.net.

liamappelbe commented 1 year ago

I think you might be pulling my leg a bit about not knowing about FFI with Dart Wasm since you literally (co)wrote the article about Darts initial wasm support

There's a lot of aspects to Dart/wasm interop, and I only really know about calling into wasm from Dart 🙃 . Calling into FFI from Dart that has been compiled to wasm is sort of the opposite. It's an interesting idea, but would need a lot of implementation work, and AFAIK no one is working on this.

dcharkes commented 1 year ago

Now that there is experimental support for Wasm as a target in Flutter, I wanted to ask if there is any updates on being able to use FFI from Dart targeted to Wasm. The use case I have in mind is that I have an open source C library (audio synth) that I currently use via FFI in a Flutter app on Linux, MacOS, Android, iOS and would now love to be able to try to target web as well, but currently I don't think there is any support for doing FFI from Dart (running compiled to wasm) to call a C lib also compiled to Wasm.

Actually, we already use dart:ffi in dart2wasm. The bindings with the graphics engine are through dart:ffi if I'm not mistaken.

However, at this point, it is not clear how we can generalize that to more than one module to interface with. dart2wasm compiles to wasmGC. The FFI is able to interop with a single wasm-non-gc module. More than one wasm-non-gc module would require the multiple linear memory proposal or some way to have multiple memories collaborate with a single linear memory. I don't know the details, but @askeksa-google who is working on dart2wasm will know them.

dcharkes commented 3 months ago

@eyebrowsoffire and I had a fruitful discussion yesterday.

Our long term vision would be to have dart:ffi bindings work in dart2wasm and dart2js provided that the C code was compiled with emscripten and wasi. In other words, the meaning of dart:ffi in wasm and js backend would be the emscripten ABI.

There are however a bunch of complications with regards to linear memory when there are two "dylibs" that both have a separate linear memory. Which could be addressed with the wasm multi memory proposal, which is still in flux. (At some point we'll have to come up with an API that allows one to specify which linear memory should be used for pointer loads and stores, possibly using the @DefaultAsset() annotation on library;. This the subsequently leads into some questions about how to compile such Dart program.) @eyebrowsoffire can comment more on the technical complications here.

(The alternative to multi-memory would be having to compile multiple native libraries in one compilation with emscripten, or to compile one of them as the "main" module and the others as non-main. Which would require coordination between the dart packages wrapping those native libraries. This would break composition.)

For now, the first step is likely that we will only support dart:ffi in dart2wasm (dart2js would be a different beast), we will only support a single emscripten compiled library (due to only having a single linear memory), and we will only support Dart standalone (Flutter already uses the one single linear memory for Skia).

The current way to use a single emscripten compiled library in Dart standalone can be found here: https://github.com/dart-lang/sdk/wiki/Dart-GSoC-2024-Project-Ideas#idea-ffigenpad

We will likely expose that at some point through the build hook integration with Wasm assets:

maks commented 1 month ago

@dcharkes thanks for the update on this! 👍🏻 One thing I am a bit confused about though is if as you mention Skia in Flutter is already using the one single linear memory, how does Flutters newly stable WASM support work?

dcharkes commented 1 month ago

@dcharkes thanks for the update on this! 👍🏻 One thing I am a bit confused about though is if as you mention Skia in Flutter is already using the one single linear memory, how does Flutters newly stable WASM support work?

Dart compiled to WASM is targeting WASM-GC not ordinary WASM. GC-ed memory is not the same as linear memory. Ordinary WASM is using linear memory (like C memory), and WASM-GC is using GCed memory (like Java/Kotlin/C#/etc.)

maks commented 1 month ago

Thanks @dcharkes ! That makes sense, I didnt realise that WASM-GC meant that there was separate memory apart from that linear memory of the "mvp" WASM. I clearly need to read up a bit more on the WASM-GC to get a better idea of how this works together with the existing WASM spec, thanks again! 👍🏻