near / nearcore

Reference client for NEAR Protocol
https://near.org
GNU General Public License v3.0
2.3k stars 600 forks source link

feat: Shadow tracking #11689

Open staffik opened 6 days ago

staffik commented 6 days ago

Part of: https://github.com/near/near-one-project-tracking/issues/65 An option for non-validator node to track shards of given validator.

During stateful -> stateless protocol upgrade a node will track all shards and will require a lot of RAM. After the migration we can move the validator key to a new, smaller node, that does not track all shards. To make it with minimal downtime, the new node needs to have appropriate shards in place and memtries loaded in memory, then we hot swap the validator key without stopping the new node. But before that happen the new node is not a validator and we need a way to tell it which validator's shards it should track.

codecov[bot] commented 5 days ago

Codecov Report

Attention: Patch coverage is 37.25490% with 32 lines in your changes missing coverage. Please review.

Project coverage is 71.79%. Comparing base (59e2b88) to head (489ed1d). Report is 1 commits behind head on master.

Files Patch % Lines
chain/chain/src/test_utils/kv_runtime.rs 0.00% 17 Missing :warning:
chain/epoch-manager/src/adapter.rs 0.00% 9 Missing :warning:
chain/epoch-manager/src/shard_tracker.rs 0.00% 3 Missing and 1 partial :warning:
chain/epoch-manager/src/lib.rs 88.23% 0 Missing and 2 partials :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #11689 +/- ## ========================================== - Coverage 71.81% 71.79% -0.02% ========================================== Files 790 790 Lines 161945 161985 +40 Branches 161945 161985 +40 ========================================== - Hits 116294 116292 -2 - Misses 40618 40651 +33 - Partials 5033 5042 +9 ``` | [Flag](https://app.codecov.io/gh/near/nearcore/pull/11689/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | Coverage Δ | | |---|---|---| | [backward-compatibility](https://app.codecov.io/gh/near/nearcore/pull/11689/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `0.23% <0.00%> (-0.01%)` | :arrow_down: | | [db-migration](https://app.codecov.io/gh/near/nearcore/pull/11689/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `0.23% <0.00%> (-0.01%)` | :arrow_down: | | [genesis-check](https://app.codecov.io/gh/near/nearcore/pull/11689/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `1.35% <2.00%> (+<0.01%)` | :arrow_up: | | [integration-tests](https://app.codecov.io/gh/near/nearcore/pull/11689/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `37.91% <35.29%> (+0.01%)` | :arrow_up: | | [linux](https://app.codecov.io/gh/near/nearcore/pull/11689/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `69.12% <37.25%> (-0.02%)` | :arrow_down: | | [linux-nightly](https://app.codecov.io/gh/near/nearcore/pull/11689/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `71.28% <37.25%> (-0.02%)` | :arrow_down: | | [macos](https://app.codecov.io/gh/near/nearcore/pull/11689/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `52.61% <37.25%> (+0.03%)` | :arrow_up: | | [pytests](https://app.codecov.io/gh/near/nearcore/pull/11689/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `1.58% <2.00%> (-0.01%)` | :arrow_down: | | [sanity-checks](https://app.codecov.io/gh/near/nearcore/pull/11689/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `1.38% <2.00%> (-0.01%)` | :arrow_down: | | [unittests](https://app.codecov.io/gh/near/nearcore/pull/11689/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `66.32% <37.25%> (-0.03%)` | :arrow_down: | | [upgradability](https://app.codecov.io/gh/near/nearcore/pull/11689/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `0.27% <0.00%> (-0.01%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

staffik commented 4 days ago

Can you test that it works well when the shard assignment is changing and the node should state sync?

Tested in forknet20, I will add a dedicated pytest.