near / nearcore

Reference client for NEAR Protocol
https://near.org
GNU General Public License v3.0
2.32k stars 623 forks source link

feat: Add chunk endorsements bitmap to BlockHeader #12024

Closed tayfunelmas closed 1 month ago

tayfunelmas commented 2 months ago

Previously we added ChunkEndorsementBitmap to BlockInfo to pass the information about which endorsements are received to the EpochManager. We used this in a tool to simulate the validator rewards and kickoffs if we used the received endorsements stats (instead of chunk production stats).

In this PR, we complete the picture by adding the bitmap to the BlockHeader. This bitmap is passed to the BlockInfo when saving it for the EpochManager. The bitmap is produced directly from the endorsement signatures in the BlockBody, with the same number of shards and chunk validator assignments per shard.

We also add validation logic for block body and header. For the block body, we can compare the endorsement signatures to the bitmap values. For the block header, we can compare the length of the bitmap to the number of validator assignments.

NOTE: I am currently reusing ProtocolFeature::ChunkEndorsementsInBlockHeader for the new block header version. And due to some tests assuming that only the last block header version is used, I moved this feature beyond Nightly. Once I can verify the easy mode works, I will send a PR to introduce a separate feature to start using BlockHeaderV5 and in Nightly.

codecov[bot] commented 2 months ago

Codecov Report

Attention: Patch coverage is 43.53312% with 179 lines in your changes missing coverage. Please review.

Project coverage is 71.43%. Comparing base (0e6179a) to head (7e54fdb). Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
core/primitives/src/block_header.rs 55.67% 81 Missing and 1 partial :warning:
...hain/src/stateless_validation/chunk_endorsement.rs 13.55% 49 Missing and 2 partials :warning:
test-utils/testlib/src/process_blocks.rs 0.00% 16 Missing :warning:
core/primitives/src/block.rs 47.82% 12 Missing :warning:
core/primitives/src/test_utils.rs 0.00% 10 Missing :warning:
.../stateless_validation/chunk_endorsements_bitmap.rs 80.00% 3 Missing :warning:
chain/chain/src/chain.rs 60.00% 1 Missing and 1 partial :warning:
chain/chain-primitives/src/error.rs 0.00% 1 Missing :warning:
core/primitives/src/views.rs 0.00% 1 Missing :warning:
tools/state-viewer/src/replay_headers.rs 0.00% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #12024 +/- ## ========================================== - Coverage 71.51% 71.43% -0.08% ========================================== Files 814 814 Lines 164513 164774 +261 Branches 164513 164774 +261 ========================================== + Hits 117648 117704 +56 - Misses 41699 41903 +204 - Partials 5166 5167 +1 ``` | [Flag](https://app.codecov.io/gh/near/nearcore/pull/12024/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | Coverage Δ | | |---|---|---| | [backward-compatibility](https://app.codecov.io/gh/near/nearcore/pull/12024/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `0.17% <0.00%> (-0.01%)` | :arrow_down: | | [db-migration](https://app.codecov.io/gh/near/nearcore/pull/12024/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `0.17% <0.00%> (-0.01%)` | :arrow_down: | | [genesis-check](https://app.codecov.io/gh/near/nearcore/pull/12024/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `1.27% <0.00%> (-0.01%)` | :arrow_down: | | [integration-tests](https://app.codecov.io/gh/near/nearcore/pull/12024/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `38.55% <39.74%> (-0.03%)` | :arrow_down: | | [linux](https://app.codecov.io/gh/near/nearcore/pull/12024/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `71.22% <43.53%> (-0.08%)` | :arrow_down: | | [linux-nightly](https://app.codecov.io/gh/near/nearcore/pull/12024/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `71.02% <43.53%> (-0.09%)` | :arrow_down: | | [macos](https://app.codecov.io/gh/near/nearcore/pull/12024/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `54.11% <23.34%> (+0.01%)` | :arrow_up: | | [pytests](https://app.codecov.io/gh/near/nearcore/pull/12024/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `1.54% <0.00%> (-0.01%)` | :arrow_down: | | [sanity-checks](https://app.codecov.io/gh/near/nearcore/pull/12024/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `1.34% <0.00%> (-0.01%)` | :arrow_down: | | [unittests](https://app.codecov.io/gh/near/nearcore/pull/12024/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `65.28% <23.34%> (-0.12%)` | :arrow_down: | | [upgradability](https://app.codecov.io/gh/near/nearcore/pull/12024/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `0.21% <0.00%> (-0.01%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.