kylebarron / parquet-wasm

Rust-based WebAssembly bindings to read and write Apache Parquet data
https://kylebarron.dev/parquet-wasm/
Apache License 2.0
482 stars 19 forks source link

bump arrow libraries to version 51 #496

Closed jdoig closed 2 months ago

jdoig commented 2 months ago

move parquet-wasm arrow versions forward, inline with arrow-wasm

details on the zstd-sys issue:

https://github.com/gyscos/zstd-rs/issues/269

https://github.com/apache/arrow-rs/issues/5565

kylebarron commented 2 months ago

Thanks! This looks great! We just need to look into the Node test for why that's failing

kylebarron commented 2 months ago

It looks like this is crashing on the upgrade to async-compat 0.2.3, which brings in this additional commit https://github.com/smol-rs/async-compat/commit/581a2c9adaa84549d097ce8d57df6a91f77c9c86. Then in Node that errors with:

stderr | tests/js/index.test.ts > read file stream
panicked at /Users/kyle/.cargo/registry/src/index.crates.io-6f17d22bba15001f/async-compat-0.2.3/src/lib.rs:459:10:
called `Result::unwrap()` on an `Err` value: Error { kind: Unsupported, message: "operation not supported on this platform" }

Stack:

Error:
    at /Users/kyle/github/kylebarron/parquet-wasm/pkg/node/parquet_wasm.js:5990:17
    at logError (/Users/kyle/github/kylebarron/parquet-wasm/pkg/node/parquet_wasm.js:243:18)
    at module.exports.__wbg_new_abda76e883ba8a5f (/Users/kyle/github/kylebarron/parquet-wasm/pkg/node/parquet_wasm.js:5989:65)
    at console_error_panic_hook::Error::new::h6bcff1050bdaae04 (wasm://wasm/086bba32:1:27682647)
    at console_error_panic_hook::hook_impl::ha691f0381a6092e0 (wasm://wasm/086bba32:1:22137276)
    at console_error_panic_hook::hook::hbaf33af14a98f6de (wasm://wasm/086bba32:1:28155985)
    at core::ops::function::Fn::call::h3c70bb7523ccfcb5 (wasm://wasm/086bba32:1:27886019)
    at std::panicking::rust_panic_with_hook::h1e6ac5d404b8e31b (wasm://wasm/086bba32:1:24961881)
    at std::panicking::begin_panic_handler::{{closure}}::h24b0f4622f2766a5 (wasm://wasm/086bba32:1:25815338)
    at std::sys_common::backtrace::__rust_end_short_backtrace::h19f35d272c126e7c (wasm://wasm/086bba32:1:28448328)

cc @H-Plus-Time, I don't know whether this is our bug or upstream's bug. For now, I can just pin to async-compat 0.2.2

kylebarron commented 2 months ago

Thanks!

github-actions[bot] commented 2 months ago

Asset Sizes

AssetUncompressed SizeCompressed Size
async_full/parquet_wasm_bg.wasm5.28MB $\color{red}\textbf{+33KB +1\%}$1.21MB $\color{red}\textbf{+34.6KB +3\%}$
slim/parquet_wasm_bg.wasm3.45MB $\color{red}\textbf{+5.46KB +0\%}$544KB $\color{red}\textbf{+11.6KB +2\%}$
sync/parquet_wasm_bg.wasm4.74MB $\color{red}\textbf{+30.5KB +1\%}$1.03MB $\color{red}\textbf{+34.2KB +3\%}$