Storage APIs

[modeveci]

For 2.28 and 2.38, WebKitWebSiteDataManager has been configure like this:

auto* websiteDataManager = webkit_website_data_manager_new( "local-storage-directory", wpeStoragePath, "disk-cache-directory", wpeDiskCachePath, "local-storage-quota", localStorageDatabaseQuotaInBytes, "indexeddb-directory", indexedDBPath, "per-origin-storage-quota", indexedDBSizeBytes, nullptr);

LocalStorage size can be independently customised than other storage areas (Cache and IndexDB storage; per-origin-storage-quota).

from 2.42 on, there are API changes, instead of per-origin-storage-quota, there are two new APIs with different meanings.

• origin-storage-ratio:A domain ratio from total ratio setting to be used for data storage
• total-storage-ratio: Total ratio of base volume size which all domains can use for data storage.

If we want to upgrade plugin for 2.42 and update webSiteDataManager we will have this pre configuration.

            auto* websiteDataManager = webkit_website_data_manager_new(
                 "local-storage-directory", wpeStoragePath,
                 "disk-cache-directory", wpeDiskCachePath,
                 "local-storage-quota", localStorageDatabaseQuotaInBytes,
                 "indexeddb-directory", indexedDBPath,
                 "origin-storage-ratio", originStorageRatio,
                 "total-storage-ratio", totalStorageRatio,
                  nullptr);

• How do we set IndexDB (or Cache storage) quota as this line is not present anymore: "per-origin-storage-quota", indexedDBSizeBytes,?
• As origin-sorage-ratio means all domain data including localstorage, what will happen to already deployed platforms/apps localstorage files.
• With the new apis; how do we control localstorage quota (in KB) when it is already part of each domain ratio?
• For directory setting I have seen this deprecation comment, can we still use independent directories for localstorage indexdb or other memories? if so, how total-ratio calculates the size based on "base-directory"

Maybe I have misunderstanding of API changes, but can we look at that with consideration of already deployed Apps?

Thank you in advance! Ozgur

[magomez]

I've been looking a bit into how origin-storage-ratio and total-storage-ratio works.

First, it seems that WebKit is moving to an approach to use 2 base directories: a base-data-directory to contain localStorage and indexedDB files (and any other user data), and a base-cache-directory to contain all the caches. The different data/cache elements will be subdirectories inside those base dirs.

For reference, this is the default dir structure (/root comes from the user home directory):

• default base-data-directory: /root/.local/share/

• default base-cache-directory: /root/.cache/

• base-data-directory/storage/ -> base dir for storage data. Used to calculate the available filesystem space.

• base-data-directory/localstorage -> default dir for localStorage. Configurable with local-storage-directory.

• base-data-directory/databases/indexeddb/ -> default dir for indexeddb databases. Configurable through indexeddb-directory.

• base-cache-directory/CacheStorage -> default dir for caches. Configurable through disk-cache-directory.

Then the origin-storage-ratio property basically indicates which percentage of the disk will be the max used by each origin to store its data (all the data except localstorage) and cache. And the total-storage-ratio indicates the percentage of the disk that will be the max used for all domains.

The quota for localStorage seems to be handled differently. Upstream this quota is hardcoded to 5MB per origin. Downstream we can modify it through the local-storage-quota property. And the space used for it doesn't seem to count for the origin-storage-ratio and total-storage-ratio calculations. This means that origin-storage-ratio is equivalent to per-origin-storage-quota, just defined with a value that's calculated from the disk space.

So for example if origin-storage-ratio is 0.01 and the dir base-data-directory/storage/ is on a 5GB disk, then it's 50MB of storage per domain as max. And if total-storage-ratio is 0.1 on that same disk, it means 500MB as max storage for all domains.

Then, regarding different directories, as I commented, WebKit is moving to an approach where separate directories should not be used, but just the two mentioned above. And then WebKit will create the appropriate required dirs inside those base dirs. This is being done for security reasons. And this ties very well with the approach of having the previous origin-storage-ratio and total-storage-ratio properties, as we just need to handle the size of a dir and the included subdirs.

But it seems that separate directories can still be used. And the contents of those custom directories are actually taken into account when calculating the space used and whether it's below the limits set by origin-storage-ratio and total-storage-ratio.

So, back to your questions one by one.

How do we set IndexDB (or Cache storage) quota as this line is not present anymore: "per-origin-storage-quota", indexedDBSizeBytes,?

Using origin-storage-ratio as commented before, to define the quota for indexedDB and Cache (together). The functionality is the same as per-origin-storage-quota, but providing a percentage instead of an absolute number.

Let's say you have that you have a disk size of X MB. And you want to have for example 20MB for indexedDB and CacheStorage. You can set origin-storage-ratio to 20/X, which defines the total space available for storage.

As origin-storage-ratio means all domain data including localstorage, what will happen to already deployed platforms/apps localstorage files.

It seems that origin-storage-ratio doesn't take localStorage into account, so the behavior using local-storage-quota should be the same.

With the new apis; how do we control localstorage quota (in KB) when it is already part of each domain ratio?

As I mentioned, you can still use local-storage-quota, because localStorage is not taken into account by origin-storage-ratio.

For directory setting I have seen this deprecation comment, can we still use independent directories for localstorage indexdb or other memories? if so, how total-ratio calculates the size based on "base-directory"

Yes, separate directories for localStorage, cache and indexeddb can still be used, despite they will become unsupported at some point (they will still be there for 2.46, but there will be a big API change for 2.52 and we will probably drop them). After that the idea is to use only base-data-directory and base-cache-directory.

Ideally you should use base-data-directory to point to a persistent location where the used data will be stored. I don't think having custom paths for localStorage and indexedDB is needed. And then set base-cache-directory to some dir on a volatile storage.

And when calculating the space used per origin, those separate directories will be taken into account as they are now.

I hope this clarifies all your doubts.

[modeveci]

Thank you very much @magomez for detailed explanation. I will discuss with @MFransen69 and @emutavchi.

[modeveci]

@magomez can you elaborate this?

But it seems that separate directories can still be used. And the contents of those custom directories are actually taken into account when calculating the space used and whether it's below the limits set by origin-storage-ratio and total-storage-ratio.

The limits set by origin-storage-ratio and total-storage-ratio are coming from the size of base-data-directory/storage/ as you explained. The usage of custom directories are taking into account for "used space" but not determining limits, right?

As we do not set base-data-directory now, it points by default to /root/.local/share/storage and the size of this path as "volume size". If we use indexedDBPath let's say: /opt/persistent/indexdb. the size of this custom path will not play a role to determine available memory rationed by origin-storage-ratio. Am I correct?

[magomez]

@magomez can you elaborate this?

But it seems that separate directories can still be used. And the contents of those custom directories are actually taken into account when calculating the space used and whether it's below the limits set by origin-storage-ratio and total-storage-ratio.

The limits set by origin-storage-ratio and total-storage-ratio are coming from the size of base-data-directory/storage/ as you explained. The usage of custom directories are taking into account for "used space" but not determining limits, right?

Exactly.

As we do not set base-data-directory now, it points by default to /root/.local/share/storage and the size of this path as "volume size". If we use indexedDBPath let's say: /opt/persistent/indexdb. the size of this custom path will not play a role to determine available memory rationed by origin-storage-ratio. Am I correct?

Yes. you're right. Even when the size used by /opt/persistent/indexdb is taken into account to calculate the used space, the limit comes from the filesystem where base-data-directory/storage/ lives. But you should be able to set the base-data-directory to /opt/persistent, so the size will be calculated from the filesystem of /opt/persistent/storage.