search

electric-sql / pglite

Lightweight WASM Postgres with real-time, reactive bindings.
https://pglite.dev
Apache License 2.0
9.15k stars 194 forks source link

About memory usage #406

Open Bloomingg opened 2 days ago

Bloomingg commented 2 days ago

I am currently facing a scenario where I perform extensive database operations in the browser and store a large amount of data. Previously, I used @jlongster/sql.js for this purpose. However, I’ve noticed that memory usage increases with the database size. Despite using IndexedDB for storage, it seems that a copy of the data is still kept in memory. I am urgently seeking an alternative library to address this memory issue. Could you tell me if pglite might have the same problem? I would appreciate your insights.

pmp-p commented 2 days ago

What is really the problem ?

Database access is all about creating fast access indexes to tabular data. The more indexes the more memory used => the faster they get. Indexes sizes and location will depend highly on your tables constructs and db engine configuration ( where we aimed defaults to low mem usage )

thruflo commented 2 days ago

I guess one aspect here is whether PGlite loads the whole database (indexes and all) into memory. Which I believe it does right now? As opposed to somehow keeping data on disk until needed to serve a query.

Bloomingg commented 2 days ago

I guess one aspect here is whether PGlite loads the whole database (indexes and all) into memory. Which I believe it does right now? As opposed to somehow keeping data on disk until needed to serve a query.

Yes, actually I want to know if PGlite loads the entire database into memory, as this may slow down the entire application when the data volume is large.

Bloomingg commented 2 days ago

What is really the problem ?

Database access is all about creating fast access indexes to tabular data. The more indexes the more memory used => the faster they get. Indexes sizes and location will depend highly on your tables constructs and db engine configuration ( where we aimed defaults to low mem usage )

I apologize if I didn't articulate my question clearly. In short, I want to know whether pglite will occupy a large amount of memory when I operate data in a database with a large amount of data (such as 200MB+).