Closed polywock closed 2 months ago
I'd rather keep the existing API and just pass a regex. Alternatively globs would be flexible and match existing webext match
ing.
browser.storage.local.get(/^flag:/)
browser.storage.local.get({match: 'flag:*'})
browser.storage.local.get(/^flag:/)
Looks great, but the other one isn't backward compatible. Whichever format it ends up being, it should probably be consistent with proposal #475.
but the other one isn't backward compatible.
Ah you're right, I forgot objects are used to provide defaults. {glob: true}
could be the last parameter then, while the first one remains a string/object as usual.
The simplest solution would be to add getAllKeys(lowerBound?:string, upperBound?:string)
method.
However, chrome.storage API was never designed to be performant and flexible, so maybe a better solution would be to add chrome.indexedDB
which would expose the full power of IndexedDB in all parts of the extension. Firefox already uses IndexedDB internally for chrome.storage anyway, so it's indeed a viable solution. It could be also a subset of IndexedDB named as chrome.db
, but I don't see why not expose the full API as the hardest thing to do is probably Promisification.
regex
JS RegExp won't work in Chrome as it only supports RE2 syntax inside the browser processes for the sake of guaranteed performance due to the absence of backtracking. It may be worse in other browsers, e.g. Safari supports a very limited/nonstandard subset of RE for a similar reason in declarativeNetRequest.
match
pattern
would work for most cases I guess, but it's too limited, so ideally it should be extended with ranges like [a-f]
.
Since regex isn't an option. Out of all the suggestions so far, I think a match pattern suggested by @tophf (in #475) and @fregante is the most well rounded. I have updated my proposal to reflect this.
There is little appetite for the proposed API, but the browser vendors are willing to consider adding a new method to retrieve all known storage keys (return value: a list of strings). This would enable you to implement the filtering yourself, and then pass the array of keys to get/remove (or a new object with set
if the goal is to bulk-replace values).
What are your thoughts on this counter-proposal?
A dumb getAllKeys() is the bare minimum, but it'll still be wasteful when there are many keys, so it's best if we can specify lower/upper bounds for the keys just like IndexedDB's getAllKeys or something similar.
@Rob--W I'm supportive, but ideally both should be implemented. Some advantages for a filter based approach.
@polywock, to clarify - if only getKeys()
was implemented, would you be a potential consumer for that? Or would that be too limited and you would workaround this in other ways? We discussed some of the the points you made in the meeting but ultimately we preferred the simpler getKeys()
API.
I'd definitely use it in my module to only fetch items that belong to it ("starts with cache:")
I think it's a fair compromise, if the filter for .get() isn't accepted. However I wonder what browsers gain from this dual API since it feels like it halves the performance (two lookups)
I also have another module that populates/autosaves the options page into the storage, given a schema. It currently uses a single key and the options are stored as a single object.
With this API, I would be able to store one item per field, without losing the core features (like the ability to migrate keys that are no longer in the schema)
what browsers gain
Simplicity. They want to keep this API simple. A getAllKeys method was always a necessity though, so it's a good compromise for them, whereas we'll still get the benefit of not reading huge values.
@oliverdunk It's not my preferred method, but getKeys() will be very useful.
Because browser.storage
api doesn't support multiple namespaces (databases), as a result, developers need to put all kinds of data for different purposes in one space. A common solution is to set a prefix for keys to distinguish different types of data.
The original intent of #475 was to support multiple space. Considering that browsers may not support this change, a compromise was made. If browsers supported multiple spaces, this proposal #505 would not exist.
So the proposal #505 itself is a product of compromise. Now, getAllKeys()
is a further compromise. Developers need to write another wrapper function, e.g. getDataByPrefix(prefix)
. It helps, but not directly. And this is mainly useful when values are large objects. If the data is small, developers can just read all the data directly.
The original intent of https://github.com/w3c/webextensions/issues/475 was to support multiple space.
Since I have long since edited my 475 proposal. For those who don't know, in 475 I had two proposals. The primary proposal was to support buckets/spaces for the storage API, and allow retrieving, deleting from a specific bucket, or listening to changes on a specific bucket. From the feedback, I switched focus to the secondary proposal (StorageArea.onChanged supporting a filter). I wouldn't say this was a compromise since I included the secondary proposal alongside the main proposal. I prefer the secondary proposal as of today, especially if paired with #505 (or even the counter-proposal, getKeys).
All things considered, I'm still satisfied a counter-proposal was accepted. It won't be as efficient as the original, but it will still allow for getting filtered keys more efficiently. It would be interesting to do benchmarks to figure out at what point it's better to use getKeys() + filter + get(filtered keys)
compared to the current get() + filter
.
In addition, if using getKeys() to filter becomes common, the browsers might later implement a dedicated function to optimize it.
out at what point it's better to use
getKeys() + filter + get(filtered keys)
compared to the currentget() + filter
My guess is that it has limited use for storage.sync
since you're reading 100KB at most from disk. Also a micro-optimization for storage.session
since reading from RAM is extremely fast.
That leaves storage.local
, particularly with unlimitedStorage
.
In any case, the most common usage would probably look something like this:
const cache = await chrome.storage.local.get(await chrome.storage.local.getKeys('cache:*'))
await chrome.storage.local.remove(await chrome.storage.local.getKeys('cache:*'))
@fregante getKeys()
doesn't have parameters, it returns all key in the storage, then filter key by yourself. Another, storage.session
is not as fast as you think in current Chrome implementation. Considering OS disk cache, storage.local
is almost as fast as storage.session
.
PS: I didn't say getKeys()
is not useful. It is useful if you have a lot of data.
I had a chance to speak to the engineering team about this. On the Chrome side, we're supportive of a getKeys()
method to get all keys from a given storage area. We would also be supportive of a deleteKeys()
that takes multiple keys to delete if there is interest from other browsers.
We would also be supportive of a deleteKeys() that takes multiple keys
This is already implemented: chrome.storage.XXXX.~delete~remove() accepts an array of keys.
This is already implemented: chrome.storage.XXXX.delete() accepts an array of keys.
Do you mean remove()
? It looks like you're right though - in which case we definitely shouldn't duplicate that.
Closing in favor of #601.
Why it's Needed (Efficiency)
I regularly need to use storage keys that involve dynamic elements like
flag:[tabId]
. The only way to get all keys starting with the prefixflag:
is usingStorageArea.get()
to get all the storage items and then to filter.This isn't very efficient. The storage areas support 10MB of data and even more with the
unlimitedStorage
permission. That's potentially a huge amount of storage data that needs to be serialized and sent over. There's also a special concern when using the Sync storage area.To delete all keys with prefix
flags:
is even worse as it requires a three step process. Get all storage items, filter for keys you want to delete, and useStorageArea.remove
to delete them.Why it's Needed (Organization)
This proposal allows extension developers to group, retrieve, and delete a collection of related keys.
```ts browser.storage.local.get(['content/a', 'content/b', 'content/c', 'content/d', 'content/e', 'content/f', ...]) // becomes browser.storage.local.get(['content/*'], true) ```Proposal
A way to retrieve and delete storage items using a filter. The filter can be a list of globs or an object. Preferably the same filter should be used for this proposal and #475.
Usage if new methods are introduced.
Usage if existing methods are used.
The second parameter true indicates we're using filters.