near / nearcore

Reference client for NEAR Protocol
https://near.org
GNU General Public License v3.0
2.32k stars 623 forks source link

feat(resharding) - Introducing ShardLayoutV2 and nighshade layout v4 #12066

Closed wacban closed 1 month ago

wacban commented 1 month ago

Adding a new shard layout structure. It replaces the boundary accounts with a mapping from shard id to the account range of that shard. This is necessary in order to implement the account id to shard id mapping. Previously it relied on shard ids being contiguous and ordered by the account ranges, neither of which will be the case in this shard layout.

codecov[bot] commented 1 month ago

Codecov Report

Attention: Patch coverage is 93.75000% with 12 lines in your changes missing coverage. Please review.

Project coverage is 71.60%. Comparing base (e5aa208) to head (797ce1f). Report is 6 commits behind head on master.

Files with missing lines Patch % Lines
core/primitives/src/shard_layout.rs 93.68% 11 Missing and 1 partial :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #12066 +/- ## ========================================== + Coverage 71.59% 71.60% +0.01% ========================================== Files 823 824 +1 Lines 165137 165348 +211 Branches 165137 165348 +211 ========================================== + Hits 118223 118400 +177 - Misses 41789 41819 +30 - Partials 5125 5129 +4 ``` | [Flag](https://app.codecov.io/gh/near/nearcore/pull/12066/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | Coverage Δ | | |---|---|---| | [backward-compatibility](https://app.codecov.io/gh/near/nearcore/pull/12066/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `0.17% <0.00%> (-0.01%)` | :arrow_down: | | [db-migration](https://app.codecov.io/gh/near/nearcore/pull/12066/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `0.17% <0.00%> (-0.01%)` | :arrow_down: | | [genesis-check](https://app.codecov.io/gh/near/nearcore/pull/12066/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `1.26% <0.00%> (-0.01%)` | :arrow_down: | | [integration-tests](https://app.codecov.io/gh/near/nearcore/pull/12066/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `38.71% <9.37%> (-0.05%)` | :arrow_down: | | [linux](https://app.codecov.io/gh/near/nearcore/pull/12066/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `71.38% <93.75%> (+0.01%)` | :arrow_up: | | [linux-nightly](https://app.codecov.io/gh/near/nearcore/pull/12066/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `71.18% <93.75%> (+0.02%)` | :arrow_up: | | [macos](https://app.codecov.io/gh/near/nearcore/pull/12066/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `54.09% <93.22%> (+0.03%)` | :arrow_up: | | [pytests](https://app.codecov.io/gh/near/nearcore/pull/12066/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `1.52% <0.00%> (-0.01%)` | :arrow_down: | | [sanity-checks](https://app.codecov.io/gh/near/nearcore/pull/12066/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `1.32% <0.00%> (-0.01%)` | :arrow_down: | | [unittests](https://app.codecov.io/gh/near/nearcore/pull/12066/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `65.33% <93.22%> (+0.03%)` | :arrow_up: | | [upgradability](https://app.codecov.io/gh/near/nearcore/pull/12066/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `0.21% <0.00%> (-0.01%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

staffik commented 1 month ago

We might want to do sth with ShardUId::next_shard_prefix(). Also, there is remove_range_by_shard_uid():

fn remove_range_by_shard_uid(store_update: &mut StoreUpdate, shard_uid: ShardUId, col: DBCol) {
    let key_from = shard_uid.to_bytes();
    let key_to = ShardUId::next_shard_prefix(&key_from);
    store_update.delete_range(col, &key_from, &key_to);
}
wacban commented 1 month ago

Reviewers, can you have another look please? The most notable change is that the mapping is now from account range to shard id (used to be the other way around). It will allows us to optimize the account id to shard mapping in the future. Also addressed the comments and did some cleanups.