search

near / nearcore

Reference client for NEAR Protocol
https://near.org
GNU General Public License v3.0
2.31k stars 619 forks source link

Increase MAX_COMPRESSED_STATE_WITNESS_SIZE to 48 MiB #11683

Closed jancionear closed 3 months ago

jancionear commented 3 months ago

The MAX_COMPRESSED_STATE_WITNESS_SIZE has been set to 32 MiB in https://github.com/near/nearcore/pull/11511. Back then it was a reasonable choice, but since then we increased some of the runtime parameters, which increased the worst case witness size by ~17MiB. (https://github.com/near/nearcore/pull/11582, https://github.com/near/nearcore/pull/11629). I'd feel safer with a larger limit on the compressed witness size.

codecov[bot] commented 3 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 71.75%. Comparing base (cf00e1b) to head (3181517). Report is 4 commits behind head on master.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #11683 +/- ## ========================================== - Coverage 71.75% 71.75% -0.01% ========================================== Files 790 790 Lines 161830 161838 +8 Branches 161830 161838 +8 ========================================== + Hits 116114 116119 +5 - Misses 40678 40681 +3 Partials 5038 5038 ``` | [Flag](https://app.codecov.io/gh/near/nearcore/pull/11683/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | Coverage Δ | | |---|---|---| | [backward-compatibility](https://app.codecov.io/gh/near/nearcore/pull/11683/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `0.23% <ø> (-0.01%)` | :arrow_down: | | [db-migration](https://app.codecov.io/gh/near/nearcore/pull/11683/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `0.23% <ø> (-0.01%)` | :arrow_down: | | [genesis-check](https://app.codecov.io/gh/near/nearcore/pull/11683/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `1.35% <ø> (-0.01%)` | :arrow_down: | | [integration-tests](https://app.codecov.io/gh/near/nearcore/pull/11683/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `37.80% <ø> (+0.01%)` | :arrow_up: | | [linux](https://app.codecov.io/gh/near/nearcore/pull/11683/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `69.16% <ø> (+0.01%)` | :arrow_up: | | [linux-nightly](https://app.codecov.io/gh/near/nearcore/pull/11683/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `71.23% <ø> (-0.01%)` | :arrow_down: | | [macos](https://app.codecov.io/gh/near/nearcore/pull/11683/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `52.63% <ø> (+0.03%)` | :arrow_up: | | [pytests](https://app.codecov.io/gh/near/nearcore/pull/11683/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `1.58% <ø> (-0.01%)` | :arrow_down: | | [sanity-checks](https://app.codecov.io/gh/near/nearcore/pull/11683/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `1.38% <ø> (-0.01%)` | :arrow_down: | | [unittests](https://app.codecov.io/gh/near/nearcore/pull/11683/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `66.34% <ø> (-0.02%)` | :arrow_down: | | [upgradability](https://app.codecov.io/gh/near/nearcore/pull/11683/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `0.28% <ø> (-0.01%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

jancionear commented 3 months ago

LGTM, but I'd also increase MAX_UNCOMPRESSED_STATE_WITNESS_SIZE 2x to be 128mib, assuming compression is on avg decreases the size 2 times. another nit: please update WITNESS_PARTS_CACHE_SIZE comment

You're right, they should be different. Let's set MAX_COMPRESSED_STATE_WITNESS_SIZE to 48 MiB and MAX_UNCOMPRESSED_STATE_WITNESS_SIZE to 64MiB. I don't expect uncompressed witnesses to be larger than 64 MiB.

Updated comment.