[reshardingV3] State ShardUIdMapping - initial implementation

staffik commented 1 month ago

Tracking issue: https://github.com/near/nearcore/issues/12050

Summary

Currently the changes should be almost no-op, as we do not explicitly save anything to DBCol::ShardUIdMapping. The only difference is that we make an additional read from DBCol::ShardUIdMapping column every time we access State column. The main logic is in Store::get_impl_state(). These changes implement mapping for reads, writes will be handled in the next PR.

Changes:

Added DBCol::ShardUIdMapping that is initially empty and will be populated on future resharding events.
Slight refactor: only allow Store to create StoreUpdate.
Store::get_impl_state() - special get() implementation for the State column.

Next steps (see tracking issue https://github.com/near/nearcore/issues/12050):

Use mapping for writes to db.
Handle copy_state_from_store in cold_storage.rs.
Integration.
State clean up (e.g. gc parent state when it is no longer referenced by any child).
Tests.

staffik commented 1 month ago

There is

scan_db_column(
    col: &str,
    lower_bound: Option<&[u8]>,
    upper_bound: Option<&[u8]>,
    store: Store)

and it might not be possible to scan child shard only. It is only a debug tool and we probably would need to live with that.

codecov[bot] commented 1 month ago

Codecov Report

Attention: Patch coverage is 89.65517% with 6 lines in your changes missing coverage. Please review.

Project coverage is 71.70%. Comparing base (99ecfa4) to head (fee1c6a). Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
core/store/src/lib.rs	85.71%	0 Missing and 4 partials :warning:
core/store/src/adapter/trie_store.rs	83.33%	0 Missing and 2 partials :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #12084 +/- ## ======================================= Coverage 71.69% 71.70% ======================================= Files 825 825 Lines 165834 165850 +16 Branches 165834 165850 +16 ======================================= + Hits 118902 118921 +19 + Misses 41751 41745 -6 - Partials 5181 5184 +3 ``` | [Flag](https://app.codecov.io/gh/near/nearcore/pull/12084/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | Coverage Δ | | |---|---|---| | [backward-compatibility](https://app.codecov.io/gh/near/nearcore/pull/12084/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `0.17% <0.00%> (-0.01%)` | :arrow_down: | | [db-migration](https://app.codecov.io/gh/near/nearcore/pull/12084/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `0.17% <0.00%> (-0.01%)` | :arrow_down: | | [genesis-check](https://app.codecov.io/gh/near/nearcore/pull/12084/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `1.26% <0.00%> (-0.01%)` | :arrow_down: | | [integration-tests](https://app.codecov.io/gh/near/nearcore/pull/12084/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `38.87% <29.31%> (+0.03%)` | :arrow_up: | | [linux](https://app.codecov.io/gh/near/nearcore/pull/12084/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `71.39% <89.65%> (+0.01%)` | :arrow_up: | | [linux-nightly](https://app.codecov.io/gh/near/nearcore/pull/12084/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `71.28% <89.65%> (+0.01%)` | :arrow_up: | | [macos](https://app.codecov.io/gh/near/nearcore/pull/12084/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `54.19% <89.65%> (-0.06%)` | :arrow_down: | | [pytests](https://app.codecov.io/gh/near/nearcore/pull/12084/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `1.57% <0.00%> (-0.01%)` | :arrow_down: | | [sanity-checks](https://app.codecov.io/gh/near/nearcore/pull/12084/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `1.38% <0.00%> (-0.01%)` | :arrow_down: | | [unittests](https://app.codecov.io/gh/near/nearcore/pull/12084/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `65.37% <89.65%> (-0.01%)` | :arrow_down: | | [upgradability](https://app.codecov.io/gh/near/nearcore/pull/12084/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `0.21% <0.00%> (-0.01%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

staffik commented 1 month ago

@wacban @shreyan-gupta I will go offline in a moment. Feel free to merge / take it over, I can continue moving it forward in 10 days.

wacban commented 3 weeks ago

Ah I just saw your zulip post that the write path will be done separately, ignore my comment about testing it. Just have a look at my comments and fix the CI and we should be good to go.

near / nearcore