search

techfort / LokiJS

javascript embeddable / in-memory database
http:/techfort.github.io/LokiJS
MIT License
6.75k stars 481 forks source link

lokijs-indexed-adapter breaks badly on Safari 13 #806

Closed radex closed 5 years ago

radex commented 5 years ago

It seems that Safari 13 has a bug (which I sadly didn't catch while it was in beta) in the way it measures how much space is used by the IndexedDB database. It thinks that much more is in use than there actually is. In Nozbe 4, just by using the app, we could get Safari to ask for 10GB of storage space, even though in reality, <100MB was used (still a lot, but two orders of magnitude less).

I think this has to do with measuring how much space the sqlite write-ahead-log takes up, and so every time Loki database gets saved (so essentially after every database operation), 100MB is added to the measure.

(I'm pretty sure this issue affects all LokiJS databases that are large enough - it surely affects all WatermelonDB databases: https://github.com/Nozbe/watermelondb/ - which use LokiJS under the hood, and I can't think of a reason why it wouldn't affect Loki)

I reported the issue: https://bugs.webkit.org/show_bug.cgi?id=202137 , but given the apparent rarity of the issue: https://twitter.com/bradeeoh/status/1176178498408370176 and knowing Apple's schedules, it will take a while until a fix is shipped.

And so I'm thinking of alternative solutions to this problem - I'm of course willing to do the work of putting it together, but I could use some advice

What I'm thinking about -- is it really necessary to save the entire database on every change? that gets very slow on our scale. Partitioning adapter helps a bit, because it saves only the dirty collection, but in our case, just one collection takes up the majority of space.

Is it possible (without reworking the entire library) to save only the dirty records? A loki collection could keep track of which records were added, updated, or deleted, and only those changes would be committed to IDB — plus probably one big record for all the collection metadata

radex commented 5 years ago

@techfort @obeliskos Hey, what do you think about this issue? TL;DR is the last paragraph

radex commented 5 years ago

808 is the fix + makes Loki on IndexedDB faster

radex commented 5 years ago

closing - use IncrementalIndexedDBAdapter to solve this issue