evanstade
commented
3 months ago So I don't think that we're likely to want to surface an API for IDB that deals with an implementation detail like the fact that LevelDB requires periodic compaction, and also websites should not need to call purge() to paper over a poor user agent implementation. But I am going to hazard a guess that you've encountered this issue on Chromium in particular. If so, can you file a bug report on crbug.com and fill out the template with repro steps etc, or just give as much detail as possible about the behavior you're seeing? Because when I write a test app that just writes a bunch of stuff to IDB repeatedly, the size doesn't necessarily behave "rationally" but does not grow without bound either.
I don't even know if all vendors are using LevelDB (it'd be funny, as WebSQL died to not be dependent on SQLite and if that's the case here everyone is dependent on LevelDB logic?!)
They aren't, and FWIW WebSQL was deprecated largely because arbitrary websites can't be trusted to inject arbitrary SQL. User agents do use SQLite for other things so there's already that dependency in terms of internal browser usage. In Chromium, IndexedDB may one day use SQLite as well.
nickchomey
WebReflection
coming from https://github.com/whatwg/storage/issues/174
What problem are you trying to solve?
There is Desktop and there is Mobile, then there is IndexedDB which works in both but it keeps adding over and over and over KB to each store and there's no way to cleanup space without forcing a version change.
A version change can only move up, it's persistent on the IndexedDB layer but it's unknown at the user level.
This makes it a catch-22 like situation where a site needs IndexedDB to store a version to orchestrate a purge (deleteObjectStore before createObjectStore and re-add all items) and behind the scene, apparently, there's LevelDB compact() that would work for this task but nowhere is exposed in the wild.
The forever growing nature of stores in IndexedDB is an issue and it leads to errors due quota, where you can just save two integers as both key and value and see it failing at some point once you have set a quota limit on devtools so that it's both unpractical and unpredictable when it fails at compacting itself, but it's amazing the only storage Web developers are supposed to use has such convoluted API to give them control even over a cursor but there's no way whatsoever to actually free the previously used space or get rid of hidden, kept alive via LevelDB, data that nobody can reach or need out there.
This feature request is to provide a way to tell LevelDB to compact its data and trash stores that can't be reached anymore either due version change or due update of the correlated data per key.
Thank you for making sense of this surely powerful API, yet one that lacks the most important bit to scale: allow more complex sites to contain the amount of storage needed by each user.
What solutions exist today?
None. There are workarounds that don't really work and API limitations around the versioning ... there's no way to guarantee that if my initial store via
store.put(2, 1)took little Kb to exist, furtherstore.put(2, 1)won't just keep bloating amount of space needed by that store that still contains, and can return, only two bytes of data.How would you solve it?
I would expose somehow LevelDB.compact() as that's the only solution that apparently work in NodeJS (not on the Web).
I don't even know if all vendors are using LevelDB (it'd be funny, as WebSQL died to not be dependent on SQLite and if that's the case here everyone is dependent on LevelDB logic?!) but a
store.purge()operation within a transaction that fulfill once it's successful and actually trashes all unnecessary reclaimed data to keep alive in users disks something not readable anymore forever, would solve the issue.Will it be slow? Fine, let applications and developers decide when to use it, so it's backward compatible but actually it will make everyone happier once available across browsers.
Anything else?
We all advocate on the Web for less RAM, less CPU usage, less bandwidth, less stress for everyone's device, and this standard came out with a forever growing, impossible to tackle, glitch around space used on physical Hardware that surely doesn't play well with the million of Websites that would like to use it and the million of users that would see their disk filled with unreachable data because all sites use IndexedDB and no site ever reclaim previous wasted space in the name of performance ... so thanks for hopefully think about any possible solution or API that won't need a version change and an upgrade transaction to drop and recreate a storage, or one that simply reclaim, or compact, the used data.
I work in the WASM enabling field and blobs out of foreing PLs are also the norm ... add one heavy blob package without checking it was there before identical and see your disk quota growing per each visit ... this is not just annoying, it's breaking everyone expectations about how much can IndexedDB scale.
Thank you.