search

paritytech / substrate

Substrate: The platform for blockchain innovators
Apache License 2.0
8.39k stars 2.65k forks source link

[FRAME] Paginated Storage Primitives #14747

Open ggwpez opened 1 year ago

ggwpez commented 1 year ago

The paged-list pallet provides a quite narrow solution to the problem of paginated data structures.
This MR implements StorageList and StoragePagedNMap. Only the latter contains code, the former is an adapter type of it. Alike it should be possible to use it as StoragePagedDoubleMap and StoragePagedMap. All variants are counted, so the Counted prefix is omitted.

Metadata

It currently uses fudged metadata by setting the metadata of a StoragePagedList<Value> to the metadata of a StorageMap<u32, Vec<Value>>. This is correct on the storage level, since it maps the page-index (u32) to a page (Vec<Value>), but when reading the page, it is unclear to the reader which offset to use within the page.
FRAME metadata V16 would be needed to fix this properly so that the reader is aware of the metadata field in storage.

Changes

paritytech-cicd-pr commented 1 year ago

The CI pipeline was cancelled due to failure one of the required jobs. Job name: cargo-check-benches Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/3359288

jasl commented 1 year ago

All variants are counted

I have yet to learn the source code, but I am curious about the counter of DoubleMap or NMap. There's a common scenario, for example, DoubleMap<CollectionId, ItemId>, I would like to list items in a collection, and it's useful to have the counter of how many items are in a collection.

Does this is the scope of the PR aim? Does the counter support multi-level?

Does the pagination ensure the results are ordered for map structures?

ggwpez commented 1 year ago

@jasl after reading your comment irealize that the name of the storage primitives is badly chosen. Its not really a "Map" in the common sense; its rather a List structure that can be accessed through a map key.
So for example a StoargePagedNMap has the functions append_one(key, value), iter(key) and len(key) (and some other).
All operations always need the full key. This means that these prefix operations like on normal NMaps are not possible... It would therefore not support sub-level counting operations.
The full key is AFAIK always needed, since the metadata needs to be updated.

jasl commented 1 year ago

Thank you for clarifying!

So this feature is for optimizing iterating a full list of storage? like when doing migration?