search

holdenmatt / duckdb-wasm-kit

Hooks and utilities to make it easier to use duckdb-wasm in React apps.
https://www.npmjs.com/package/duckdb-wasm-kit
MIT License
101 stars 4 forks source link

2GiB limit on the DuckDB file? #13

Closed ravwojdyla closed 4 months ago

ravwojdyla commented 4 months ago

https://github.com/holdenmatt/duckdb-wasm-kit/blob/d598b570593280a7daabb0db2930ad8a3cd1ee38/src/init/initializeDuckDb.ts#L59

Does the use of Uint8Array imply there's a 2GiB limit on the file size?

ravwojdyla commented 4 months ago

Noting that duckdb-wasm itself is expecting Uint8Array:

https://github.com/duckdb/duckdb-wasm/blob/26245e67554ed521bad0372e41bc792f8171d864/packages/duckdb-wasm/src/bindings/bindings_interface.ts#L38

But I'm not sure if that's actually "up to date" limit: https://github.com/duckdb/duckdb-wasm/discussions/1241#discussioncomment-6433453

ravwojdyla commented 4 months ago

Linking https://github.com/duckdb/duckdb-wasm/issues/1705

ramonvermeulen commented 4 months ago

https://shell.duckdb.org/docs/classes/index.AsyncDuckDB.html#registerFileBuffer

I think the max size might indeed be 2gb since it is typed as Uint8Array, not sure what the reason for this is tho.

ravwojdyla commented 4 months ago

Closing this in favor of https://github.com/duckdb/duckdb-wasm/issues/1705