search

techfort / LokiJS

javascript embeddable / in-memory database
http:/techfort.github.io/LokiJS
MIT License
6.73k stars 482 forks source link

Using LokiJS for big data #877

Closed mstimvol closed 2 years ago

mstimvol commented 3 years ago

I'm currently thinking about using LokiJS in my new project. Unfortunately I don't know if Loki can fit my requirements because the database has at least 100GB. I was already checking out other issues/stackoverflow posts and found:

As stated in issue 551, the suggested max size is 600 MB because of the time required for saving the JSON file. But now I found IncrementalIndexedDBAdapter (see https://github.com/techfort/LokiJS/blob/master/tutorials/Persistence%20Adapters.md) and it says "split into small chunks with individual collection documents". So I think this solves the issue discribed in issue 551, but I'm not sure if this also enables me to handle a 100GB without having the entire database in memory... any suggestions?

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

cepm-nate commented 2 years ago

LokiJS is an in-memory database storage and querying library. The two referenced posts refer to saving data from memory onto disk. That means it had to be in the memory at one point in time.

Unless you can load all 100GB into memory, I don't think LokiJS would be a good fit.

I would love to see a version of LokiJS that could query & update data directly in indexedDB! Especially if it could also handle automatic paging and partitioning. That would remove the troublesome "flush to disk" and "load from disk" that happens currently. Any development headed in that direction?

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.