search

duckdb / duckdb-wasm

WebAssembly version of DuckDB
https://shell.duckdb.org
MIT License
1.32k stars 132 forks source link

`registerFileBuffer` larger than 2GiB #1705

Open ravwojdyla opened 7 months ago

ravwojdyla commented 7 months ago

What happens?

My understanding is that duckdb-wasm should be able to utilize up to 4GiB of memory (https://v8.dev/blog/4gb-wasm-memory), I'm currently not able to load an existing duckdb file that is slightly more than 2GiB. I suspect it's due to the use (and expectation) of Uint8Array at https://github.com/duckdb/duckdb-wasm/blob/26245e67554ed521bad0372e41bc792f8171d864/packages/duckdb-wasm/src/bindings/bindings_interface.ts#L38

Should I be able to load a duckdb file that's larger than 2GiB (but smaller than 4GiB)?

To Reproduce

Create a duckdb dump that's over 2GiB, try to load it using registerFileBuffer, see example here:

https://github.com/holdenmatt/duckdb-wasm-kit/blob/d598b570593280a7daabb0db2930ad8a3cd1ee38/src/init/initializeDuckDb.ts#L54-L59

Browser/Environment:

Chrome 123

Device:

Mac

DuckDB-Wasm Version:

1.28.1-dev99.0

DuckDB-Wasm Deployment:

Internal

Full Name:

Rafal Wojdyla

Affiliation:

Related Sciences

phyyou commented 4 months ago

https://github.com/WebAssembly/memory64/blob/main/proposals/memory64/Overview.md

I found memory64 at wasm is marked done.

Is it possible load more than 4GB of data with Memory64 support?