Open ravwojdyla opened 2 months ago
My understanding is that duckdb-wasm should be able to utilize up to 4GiB of memory (https://v8.dev/blog/4gb-wasm-memory), I'm currently not able to load an existing duckdb file that is slightly more than 2GiB. I suspect it's due to the use (and expectation) of Uint8Array at https://github.com/duckdb/duckdb-wasm/blob/26245e67554ed521bad0372e41bc792f8171d864/packages/duckdb-wasm/src/bindings/bindings_interface.ts#L38
Uint8Array
Should I be able to load a duckdb file that's larger than 2GiB (but smaller than 4GiB)?
Create a duckdb dump that's over 2GiB, try to load it using registerFileBuffer, see example here:
registerFileBuffer
https://github.com/holdenmatt/duckdb-wasm-kit/blob/d598b570593280a7daabb0db2930ad8a3cd1ee38/src/init/initializeDuckDb.ts#L54-L59
Chrome 123
Mac
1.28.1-dev99.0
Internal
Rafal Wojdyla
Related Sciences
What happens?
My understanding is that duckdb-wasm should be able to utilize up to 4GiB of memory (https://v8.dev/blog/4gb-wasm-memory), I'm currently not able to load an existing duckdb file that is slightly more than 2GiB. I suspect it's due to the use (and expectation) of
Uint8Arrayat https://github.com/duckdb/duckdb-wasm/blob/26245e67554ed521bad0372e41bc792f8171d864/packages/duckdb-wasm/src/bindings/bindings_interface.ts#L38Should I be able to load a duckdb file that's larger than 2GiB (but smaller than 4GiB)?
To Reproduce
Create a duckdb dump that's over 2GiB, try to load it using
registerFileBuffer, see example here:https://github.com/holdenmatt/duckdb-wasm-kit/blob/d598b570593280a7daabb0db2930ad8a3cd1ee38/src/init/initializeDuckDb.ts#L54-L59
Browser/Environment:
Chrome 123
Device:
Mac
DuckDB-Wasm Version:
1.28.1-dev99.0
DuckDB-Wasm Deployment:
Internal
Full Name:
Rafal Wojdyla
Affiliation:
Related Sciences