test(state-sync): shard swap in single shard tracking

[VanBarbascu]

Pytest to check decentralised state sync of nodes tracking one shard. Keep shard shuffling off until the implementation is done.

This test sets a dumper node only for the state sync headers. Sets 4 validator nodes, each tracking 1 shard. Sets an RPC node to handle the random traffic that changes the state. Only allow validator nodes to share parts by enabling the state snapshot.

Check if the network can got for 6 epochs while shard shuffling is on. Validator nodes are expected to download parts from each other.

Check http://127.0.0.1:3040/debug/pages/epoch_info for the validator assignment rotation.

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 71.52%. Comparing base (523837b) to head (8d65d28). Report is 1 commits behind head on master.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #12108 +/- ## ========================================== + Coverage 71.51% 71.52% +0.01% ========================================== Files 820 820 Lines 165091 165091 Branches 165091 165091 ========================================== + Hits 118059 118078 +19 + Misses 41892 41870 -22 - Partials 5140 5143 +3 ``` | [Flag](https://app.codecov.io/gh/near/nearcore/pull/12108/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | Coverage Δ | | |---|---|---| | [backward-compatibility](https://app.codecov.io/gh/near/nearcore/pull/12108/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `0.17% <ø> (ø)` | | | [db-migration](https://app.codecov.io/gh/near/nearcore/pull/12108/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `0.17% <ø> (ø)` | | | [genesis-check](https://app.codecov.io/gh/near/nearcore/pull/12108/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `1.26% <ø> (ø)` | | | [integration-tests](https://app.codecov.io/gh/near/nearcore/pull/12108/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `38.71% <ø> (-0.03%)` | :arrow_down: | | [linux](https://app.codecov.io/gh/near/nearcore/pull/12108/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `71.32% <ø> (+0.01%)` | :arrow_up: | | [linux-nightly](https://app.codecov.io/gh/near/nearcore/pull/12108/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `71.11% <ø> (+0.01%)` | :arrow_up: | | [macos](https://app.codecov.io/gh/near/nearcore/pull/12108/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `53.36% <ø> (-0.01%)` | :arrow_down: | | [pytests](https://app.codecov.io/gh/near/nearcore/pull/12108/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `1.52% <ø> (ø)` | | | [sanity-checks](https://app.codecov.io/gh/near/nearcore/pull/12108/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `1.32% <ø> (ø)` | | | [unittests](https://app.codecov.io/gh/near/nearcore/pull/12108/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `65.23% <ø> (+<0.01%)` | :arrow_up: | | [upgradability](https://app.codecov.io/gh/near/nearcore/pull/12108/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `0.21% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[VanBarbascu]

If there's an easy way to make the dumper upload only the headers and not the parts, that would be nice in the short term. In the long run once we fully decentralize we could just get rid of the dumper.

This is what I was aiming for when I set the dumper to skip snapshot:

# State snapshot is disabled for dumper. We only want to dump the headers.
node_config_dump["store.state_snapshot_enabled"] = False

Could there be a conflict between store.state_snapshot_enabled and store.state_snapshot configs? When we set store.state_snapshot_enabled to True, store.state_snapshot is set to EveryEpoch. When it is False, store.state_snapshot stays the same.

The default value for store.state_snapshot changed in the meantime from OnlyForResharding to EveryEpoch so that is why you are experiencing this behaviour.

[VanBarbascu]

Done! Now it should not dump the parts.