near / nearcore

Reference client for NEAR Protocol
https://near.org
GNU General Public License v3.0
2.32k stars 623 forks source link

feat: increase main_storage_proof_size_soft_limit #11997

Closed pugachAG closed 2 months ago

pugachAG commented 2 months ago

This PR changes storage proof soft size limit from 3mb to 4mb. Soft limits defines the value after which we stop executing receipts and move them to the delayed queue. This change addresses the issues on mainnet where we have up to 12% of chunks hitting the size limit. It is worse on shard 5 where it reaches 25%. With this PR we decrease the number of such chunks by ~70%. This change potentially increases the max possible size of state witness, but in practice compressed storage proof doesn't increase much with the current mainnet traffic (experiment is 4mb limit and update_shard is 3mb):

Screenshot 2024-08-29 at 13 48 00

Also see zulip thread for more context.

codecov[bot] commented 2 months ago

Codecov Report

Attention: Patch coverage is 0% with 1 line in your changes missing coverage. Please review.

Project coverage is 71.52%. Comparing base (0e5e7c7) to head (442e7d4).

Files with missing lines Patch % Lines
core/primitives-core/src/version.rs 0.00% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #11997 +/- ## ========================================== - Coverage 71.53% 71.52% -0.02% ========================================== Files 814 814 Lines 164424 164425 +1 Branches 164424 164425 +1 ========================================== - Hits 117623 117597 -26 - Misses 41640 41662 +22 - Partials 5161 5166 +5 ``` | [Flag](https://app.codecov.io/gh/near/nearcore/pull/11997/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | Coverage Δ | | |---|---|---| | [backward-compatibility](https://app.codecov.io/gh/near/nearcore/pull/11997/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `0.17% <0.00%> (ø)` | | | [db-migration](https://app.codecov.io/gh/near/nearcore/pull/11997/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `0.17% <0.00%> (ø)` | | | [genesis-check](https://app.codecov.io/gh/near/nearcore/pull/11997/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `1.28% <0.00%> (-0.01%)` | :arrow_down: | | [integration-tests](https://app.codecov.io/gh/near/nearcore/pull/11997/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `38.58% <0.00%> (-0.01%)` | :arrow_down: | | [linux](https://app.codecov.io/gh/near/nearcore/pull/11997/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `71.21% <0.00%> (-0.02%)` | :arrow_down: | | [linux-nightly](https://app.codecov.io/gh/near/nearcore/pull/11997/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `71.11% <0.00%> (-0.01%)` | :arrow_down: | | [macos](https://app.codecov.io/gh/near/nearcore/pull/11997/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `54.07% <0.00%> (-0.01%)` | :arrow_down: | | [pytests](https://app.codecov.io/gh/near/nearcore/pull/11997/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `1.54% <0.00%> (-0.01%)` | :arrow_down: | | [sanity-checks](https://app.codecov.io/gh/near/nearcore/pull/11997/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `1.35% <0.00%> (-0.01%)` | :arrow_down: | | [unittests](https://app.codecov.io/gh/near/nearcore/pull/11997/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `65.41% <0.00%> (-0.02%)` | :arrow_down: | | [upgradability](https://app.codecov.io/gh/near/nearcore/pull/11997/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `0.21% <0.00%> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

pugachAG commented 2 months ago

@jancionear thanks for pointing out the test, it actually helped me to discover that I've forgotten to update stable version: 9b94e6793034b5b5221560f3cbb6a919f25a2b60

jancionear commented 2 months ago

@jancionear thanks for pointing out the test, it actually helped me to discover that I've forgotten to update stable version: https://github.com/near/nearcore/commit/9b94e6793034b5b5221560f3cbb6a919f25a2b60

Ah right, missed that too. You'll also probably need to update the hashes generated by cargo insta (?)

walnut-the-cat commented 2 months ago

Aug 30th status report