search

hydro-project / hydroflow

Hydro's low-level dataflow runtime
https://hydro.run/docs/hydroflow/
Apache License 2.0
469 stars 34 forks source link

WASM running of hydroflow #329

Open MingweiSamuel opened 1 year ago

MingweiSamuel commented 1 year ago

Should be possible to do, but do we want to? Would be fun but maybe not high priority. Mentioned in slack briefly @shadaj

shadaj commented 1 year ago

I've been slowly working through this in down time, current goal is to get a demo on the website

MingweiSamuel commented 1 year ago

The surface syntax parsing and processing should be good, but I anticipate issues with Tokio on WASM

shadaj commented 1 year ago

Oh yes true, I guess there are two issues: get the Hydroflow Language / Dedalus compilers to work on WASM, and get the Hydroflow runtime to work. Found https://github.com/shosti/wraft as a potentially interesting benchmark for the runtime; in principle we should be able to wire up WebRTC as the networking layer with Tokio on WASM.

MingweiSamuel commented 1 year ago

hydroflow_lang #331

shadaj commented 1 year ago

@tylerhou will look into this soon

tylerhou commented 1 year ago

I took a brief look at this today. The latest version of Tokio has limited support for WASM including runtime: https://github.com/tokio-rs/tokio/blob/a7945b469d634cf205094d8a1661720358622cc0/tokio/src/lib.rs#L387. Our current version also supports all those features above but it is not documented (tested by compiling for WASM with those features enabled).

For WASM we would need to not specify tokio { ... features = "full" } in hydroflow/Cargo.toml and instead selectively enable the features based if the target is WASM. hydroflow compiles if we do this and also disable diagnostics for hydroflow_lang. Very rough patch that compiles for WASM: https://gist.github.com/tylerhou/bd3ae3e2bb58c3f0b05fc100a63298b7

There are still two outstanding issues:

  1. Documentation for tokio says that while time compiles for non-WASI WASM targets, it panics at runtime. We use timers in only one file -- hydroflow/tests/surface_async.rs -- and its a test, so maybe this is not an issue.
  2. cargo test -p hydroflow --target wasm32-unknown-unknown does not work because the getrandom library (which is included tvia rand and transitively through textnonce -> rand in dev dependencies) requires us to build with the js feature flag or target wasm32-unknown-emscripten so it knows which random implementation to call. https://docs.rs/getrandom/latest/getrandom/#webassembly-support

wasm.patch

shadaj commented 1 year ago

This is very exciting progress! For the second issue, I think we can add a js feature flag to hydroflow itself that then enables the js flag on getrandom. That way we can support both wasm32-unknown-unknown and wasm32-wasi (I'm not particularly concerned about -emscripten).

MingweiSamuel commented 5 months ago

(Compilation in WASM, i.e. running hydroflow_lang/macro, has been working for a long time)