search

typicode / lowdb

Simple and fast JSON database
MIT License
21.49k stars 923 forks source link

It is possible to remove lowdb limits with a per adapter. #484

Closed zelferry closed 3 years ago

zelferry commented 3 years ago

"If you have large JavaScript objects (~10-100MB) you may hit some performance issues. This is because whenever you call db.write, the whole db.data is serialized and written to storage. Depending on your use case, this can be fine or not. It can be mitigated by doing batch operations and calling db.write only when you need it."

but if you have a way to make a custom adapter, then that adapter can have functions, several libraries and systems to load large JavaScript objects for lowdb to work on a large scale?

(and because I have a bot that uses lowdb for some things and I wanted to know how to "break" the lowdb limit)

typicode commented 3 years ago

Maybe. However, I'm not aware of such library.

There would still be challenges. db.data holds the full database in memory (so if it exceeds your RAM it will crash, node would need to be started with additional params to increase memory limit).

You could split db.json in multiple files for unrelated data, that could help with serialization cost.

SQLite has higher limits and scales better. It would be a good choice for a local database, if you're hitting limits with lowdb.