codecov[bot]
commented
1 day ago Codecov Report
Attention: Patch coverage is 84.39394% with 103 lines in your changes missing coverage. Please review.
Project coverage is 69.97%. Comparing base (
4e87bae) to head (77de567). Report is 1 commits behind head on master.
| Files with missing lines | Patch % | Lines |
|---|---|---|
| nearcore/src/state_sync.rs | 84.67% | 50 Missing and 45 partials :warning: |
| chain/chain/src/store/mod.rs | 68.00% | 6 Missing and 2 partials :warning: |
Additional details and impacted files
```diff @@ Coverage Diff @@ ## master #12492 +/- ## ========================================== + Coverage 69.84% 69.97% +0.12% ========================================== Files 838 838 Lines 169424 169520 +96 Branches 169424 169520 +96 ========================================== + Hits 118337 118616 +279 + Misses 45839 45756 -83 + Partials 5248 5148 -100 ``` | [Flag](https://app.codecov.io/gh/near/nearcore/pull/12492/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | Coverage Δ | | |---|---|---| | [backward-compatibility](https://app.codecov.io/gh/near/nearcore/pull/12492/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `0.16% <0.00%> (?)` | | | [db-migration](https://app.codecov.io/gh/near/nearcore/pull/12492/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `0.16% <0.00%> (-0.01%)` | :arrow_down: | | [genesis-check](https://app.codecov.io/gh/near/nearcore/pull/12492/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `1.28% <0.00%> (-0.02%)` | :arrow_down: | | [linux](https://app.codecov.io/gh/near/nearcore/pull/12492/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `69.28% <82.87%> (+0.11%)` | :arrow_up: | | [linux-nightly](https://app.codecov.io/gh/near/nearcore/pull/12492/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `69.54% <84.39%> (+0.11%)` | :arrow_up: | | [macos](https://app.codecov.io/gh/near/nearcore/pull/12492/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `50.98% <0.00%> (-0.04%)` | :arrow_down: | | [pytests](https://app.codecov.io/gh/near/nearcore/pull/12492/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `1.58% <0.00%> (-0.02%)` | :arrow_down: | | [sanity-checks](https://app.codecov.io/gh/near/nearcore/pull/12492/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `1.39% <0.00%> (-0.02%)` | :arrow_down: | | [unittests](https://app.codecov.io/gh/near/nearcore/pull/12492/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `69.79% <84.39%> (+0.12%)` | :arrow_up: | | [upgradability](https://app.codecov.io/gh/near/nearcore/pull/12492/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `0.20% <0.00%> (-0.01%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near#carryforward-flags-in-the-pull-request-comment) to find out more.:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
🚨 Try these New Features:
- Flaky Tests Detection - Detect and resolve failed and flaky tests
- JS Bundle Analysis - Avoid shipping oversized bundles
The current state dumper code is sort of difficult to follow, and doesn't make good use of the available cores to obtain and upload parts. It starts one thread per shard that dumps one part on each iteration of a big loop (that includes a good amount of unnecessary/redundant lookups and calculations). So here we rewrite the logic so that instead of starting one thread per shard and looping over part IDs like that, we just figure out what parts need to be dumped when we see a new epoch, and then spawn futures to obtain and upload the parts. Now the part upload speed will be limited by the number of allowed "obtain part" tasks (4), and the speed of generating those parts.
This has the advantage of not needing to change anything to work with dynamic resharding, and the part upload is much faster. On a forknet run with recent mainnet state, the old dumper takes around an hour and a half to dump all the parts, and this version takes around half an hour (could maybe be improved by tweaking/making configurable the number of allowed tasks obtaining parts at a time)
This could be refactored further because there's still some leftover structures from the previous implementation that don't fit super cleanly, but this can be done in a future PR.