search

duckdb / duckdb-wasm

WebAssembly version of DuckDB
https://shell.duckdb.org
MIT License
1.01k stars 110 forks source link

Throw an error with mismatched Apache Arrow version #1761

Open willium opened 2 weeks ago

willium commented 2 weeks ago

What happens?

Right now, if you have a mismatched apache arrow version, DuckDB WASM fails silently or throws very confusing errors. I was trying to create a table, and nothing happened.

To Reproduce

Target Arrow 15 or 16, and try to use DuckDB.

Browser/Environment:

Chrome

Device:

Mac

DuckDB-Wasm Version:

Latest

DuckDB-Wasm Deployment:

App

Full Name:

William Wolf

Affiliation:

Gestalt

domoritz commented 2 weeks ago

This keeps coming up. I wonder what the best way to resolve this is. Maybe only include arrow as a peer dependency or a more robust check for isArrowTable?

domoritz commented 2 weeks ago

Related https://github.com/vega/vega-loader-arrow/issues/350

rob-blackbourn commented 2 weeks ago

Is the arrow esm vs csj problem I reported here #1708 ?

domoritz commented 2 weeks ago

That's related. The problem can also happen when you have e.g. arrow 14 but duckdb wasm uses arrow 15.

carlopi commented 2 weeks ago

Problem seems quite hard, it's the problem of being successful, but instanceof feel like the wrong solution here. Would it not be possible to duck-type this and assume that if something looks like an arrow table has to be an arrow table?

Or possibly using stuff like obj.constructor.someStaticFieldThatEncodesTheArrowVersion ?

carlopi commented 2 weeks ago

One note is that DuckDB-Wasm has possibly a simpler job that regular arrow, since we can implement detection our side, once a semi-reliable detection function is approved.

stefnotch commented 4 days ago

One option might be to re-export apache arrow? At least that's what some Rust libraries do for their dependencies.

domoritz commented 4 days ago

That doesn't help if another library has arrow as a dependency. This really needs to be solved at the npm/yarn/etc level but we can try to warn people when they may have multiple versions.