Modular Rust Arrow libraries in wasm-bindgen

[kylebarron]

Problem statement

The biggest hurdle with WebAssembly in the browser is that multiple Wasm modules can't share the same memory space. This means that having e.g. parquet-wasm and geoarrow-wasm as two separate NPM modules is annoying! You have to use parquet-wasm to load parquet into Arrow in Wasm... but then copy the data to JS, and then copy it into the next wasm module to do more processing with it! This is slow, memory intensive, and not user friendly.

Solution

In https://github.com/domoritz/arrow-wasm, Dominik's goal appeared to be to see if Arrow in rust/wasm would be faster than Arrow in JS. But since working with raw buffers is pretty fast in JS, it's not surprising that Wasm overhead would outweigh any other speedups.

I think the potential of arrow-wasm instead is in being a foundational library for other wasm-bindgen libraries.

So I see various potential libraries:

• parquet-wasm: depends on arrow-wasm directly; used by consumers who only want to parse Parquet and get it into JS. PR is here: https://github.com/kylebarron/parquet-wasm/pull/292
• geoparquet-wasm: depends on arrow-wasm and geoarrow-rs; used by consumers who only want to parse GeoParquet to GeoArrow and get it into JS. PR is here: https://github.com/kylebarron/geoparquet-wasm/pull/6
• geoarrow-wasm-slim: depends on arrow-wasm, doesn't include geoparquet io to keep a small bundle size. Used by consumers who somehow otherwise have their data as geoarrow in the browser.
• geoarrow-wasm-full: includes extras like re-exporting geoparquet-wasm. Used by consumers who want to fetch geoparquet to geoarrow but also do some geospatial operations.

Other libraries for other formats might make sense to add in the future. like geoarrow-flatgeobuf, which uses rust to parse flatgeobuf into geoarrow. Etc.

Drawbacks

• It would be nice if it were possible to add wasm-bindgen methods onto existing structs from another crate, but that doesn't appear to be possible 🥲 . This means that any crates other than arrow-wasm may only use structs defined in arrow-wasm in a functional manner.
• There will be a proliferation of feature flags. E.g. geoarrow-wasm might have feature flags for each compression in parquet-wasm?

cc @H-Plus-Time

[H-Plus-Time]

Yep, this is a good way the way to do things, though I'd add a slight proviso to the drawbacks:

• It isn't possible unless using wrapper structs - at least a fixed 8 byte memory cost per level of wrapping, unsure exactly how much of the function call hopping can be optimized away. Probably 'strongly preferred' would be a better way of terming it.
• Provided there isn't some awful quirk in depending crates, it should at least be relatively boilerplate. The difficulty would probably come from complicated conditional flag combinations (maybe once you're at the level of geopolars or js-polars, the impact of, say, individua compressions at the arrow-wasm/parquet-wasm level are too small to bother with flags other than 'all compressions').

I'd probably also do geoarrow-wasm-slim and geoarrow-wasm-full in the one npm package and manage stuff via exports (~40MB + 2x geoarrow-wasm's size - I reckon all the packages that produce {node,bundler,esm,esm2} bundles can be slimmed down by 25-50% with one straightforward and one less straightforward tweak).

[kylebarron]

• It isn't possible unless using wrapper structs

What I was hoping was that I could re-export all the existing struct's methods, and just add one new method. That doesn't really seem possible without manually wrapping the wrapped-struct's methods one by one?

• Provided there isn't some awful quirk in depending crates, it should at least be relatively boilerplate. The difficulty would probably come from complicated conditional flag combinations (maybe once you're at the level of geopolars or js-polars, the impact of, say, individua compressions at the arrow-wasm/parquet-wasm level are too small to bother with flags other than 'all compressions').

Yeah I agree. It should be straightforward, just annoying.

I reckon all the packages that produce {node,bundler,esm,esm2} bundles can be slimmed down by 25-50% with one straightforward and one less straightforward tweak).

If you have packaging recommendations I'm all ears 🙂 . esm2 was a "temporary" hack to get esm working in deck.gl I believe, or something like that. Because the esm export used syntax only available in some specific environment.