search

aptos-labs / aptos-core

Aptos is a layer 1 blockchain built to support the widespread use of blockchain through better technology and user experience.
https://aptosfoundation.org
Other
5.85k stars 3.54k forks source link

[INC-600] fix state db truncation when db-sharding is not enabled. #13196

Closed msmouse closed 2 weeks ago

msmouse commented 2 weeks ago

Description

Fix that state tree / kv truncator do duplicated work when db sharding is not enabled.

Type of Change

Which Components or Systems Does This Change Impact?

How Has This Been Tested?

forge test fullnode_reboot_stress_test were broken since last fix (where state kv db truncation is always attempted), this fixes that.

Key Areas to Review

Checklist

trunk-io[bot] commented 2 weeks ago
⏱️ 8h 42m total CI duration on this PR | Job | Cumulative Duration | Recent Runs | |---|---|---| | [forge-framework-upgrade-test / forge](https://github.com/aptos-labs/aptos-core/actions/runs/8952773318/job/24590694440) | 1h 10m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8952773318/job/24590694440)  | | [rust-smoke-tests](https://github.com/aptos-labs/aptos-core/actions/runs/8952773280/job/24590530906) | 1h 5m | [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/8947634413/job/24579810170) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8952773280/job/24590530906)  | | [rust-targeted-unit-tests](https://github.com/aptos-labs/aptos-core/actions/runs/8952773280/job/24590530937) | 58m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8945046904/job/24573243322) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8945276225/job/24573950384) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8946621395/job/24577524653) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8947634413/job/24579810019) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8952773280/job/24590530937)  | | [execution-performance / single-node-performance](https://github.com/aptos-labs/aptos-core/actions/runs/8952773285/job/24592436287) | 51m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8947634409/job/24579810899) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/8952773285/job/24590530597) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8952773285/job/24592436287)  | | [windows-build](https://github.com/aptos-labs/aptos-core/actions/runs/8952773278/job/24590528641) | 37m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8952773278/job/24590528641)  | | [rust-images / rust-all](https://github.com/aptos-labs/aptos-core/actions/runs/8952773318/job/24590530953) | 35m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8947634455/job/24579810926) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8952773318/job/24590530953)  | | [rust-lints](https://github.com/aptos-labs/aptos-core/actions/runs/8952773280/job/24590530821) | 32m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8945046904/job/24573243442) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/8945276225/job/24573950149) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8946621395/job/24577524558) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8947634413/job/24579809790) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8952773280/job/24590530821)  | | [forge-e2e-test / forge](https://github.com/aptos-labs/aptos-core/actions/runs/8952773318/job/24590694375) | 30m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8947634455/job/24580030990) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8952773318/job/24590694375)  | | [forge-compat-test / forge](https://github.com/aptos-labs/aptos-core/actions/runs/8952773318/job/24590694308) | 25m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8947634455/job/24580030922) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8952773318/job/24590694308)  | | [run-tests-main-branch](https://github.com/aptos-labs/aptos-core/actions/runs/8952773295/job/24590532608) | 21m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8945046444/job/24573245632) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8945276212/job/24573952310) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8946621349/job/24577526974) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8947634424/job/24579814144) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8952773295/job/24590532608)  | | [rust-move-tests](https://github.com/aptos-labs/aptos-core/actions/runs/8952773284/job/24590528817) | 19m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8945046903/job/24573234759) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8945276220/job/24573942337) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8946621394/job/24577518978) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8947634411/job/24579806637) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8952773284/job/24590528817)  | | [cli-e2e-tests / run-cli-tests](https://github.com/aptos-labs/aptos-core/actions/runs/8952773318/job/24590694237) | 18m | [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/8947634455/job/24580030774) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/8952773318/job/24590694237)  | | [framework-upgrade-determinator](https://github.com/aptos-labs/aptos-core/actions/runs/8952773318/job/24590530078) | 14m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8947634455/job/24579809096) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8952773318/job/24590530078)  | | [rust-build-cached-packages](https://github.com/aptos-labs/aptos-core/actions/runs/8952773280/job/24590530843) | 11m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8947634413/job/24579810100) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8952773280/job/24590530843)  | | [general-lints](https://github.com/aptos-labs/aptos-core/actions/runs/8952773280/job/24590530799) | 9m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8945046904/job/24573243133) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8945276225/job/24573950286) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8946621395/job/24577524461) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8947634413/job/24579809875) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8952773280/job/24590530799)  | | [check](https://github.com/aptos-labs/aptos-core/actions/runs/8952773283/job/24590528928) | 8m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8947634404/job/24579806489) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8952773283/job/24590528928)  | | [check-dynamic-deps](https://github.com/aptos-labs/aptos-core/actions/runs/8952773282/job/24590528586) | 7m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8945046917/job/24573234054) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8945276223/job/24573941835) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8946621399/job/24577518772) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8947634407/job/24579806135) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8952773282/job/24590528586)  | | [indexer-grpc-e2e-tests / test-indexer-grpc-docker-compose](https://github.com/aptos-labs/aptos-core/actions/runs/8952773318/job/24590693987) | 4m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8947634455/job/24580030581) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8952773318/job/24590693987)  | | [semgrep/ci](https://github.com/aptos-labs/aptos-core/actions/runs/8952773279/job/24590528589) | 2m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8945046899/job/24573233971) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8945276226/job/24573941966) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8946621401/job/24577518713) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8947634412/job/24579806262) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8952773279/job/24590528589)  | | [node-api-compatibility-tests / node-api-compatibility-tests](https://github.com/aptos-labs/aptos-core/actions/runs/8952773318/job/24590694077) | 2m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8947634455/job/24580030856) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8952773318/job/24590694077)  | | [file_change_determinator](https://github.com/aptos-labs/aptos-core/actions/runs/8952773295/job/24590530500) | 53s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8945046444/job/24573237841) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8945276212/job/24573945654) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8946621349/job/24577522030) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8947634424/job/24579809219) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8952773295/job/24590530500)  | | [file_change_determinator](https://github.com/aptos-labs/aptos-core/actions/runs/8952773280/job/24590528819) | 52s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8945046904/job/24573235002) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8945276225/job/24573942402) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8946621395/job/24577518985) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8947634413/job/24579806597) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8952773280/job/24590528819)  | | [file_change_determinator](https://github.com/aptos-labs/aptos-core/actions/runs/8952773318/job/24590529977) | 22s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8947634455/job/24579809207) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8952773318/job/24590529977)  | | [execution-performance / file_change_determinator](https://github.com/aptos-labs/aptos-core/actions/runs/8952773285/job/24590528774) | 22s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8947634409/job/24579806447) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8952773285/job/24590528774)  | | [permission-check](https://github.com/aptos-labs/aptos-core/actions/runs/8952773291/job/24590528734) | 19s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8945046433/job/24573233127) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8945276215/job/24573941993) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8946621351/job/24577518492) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8947634418/job/24579806210) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8952773291/job/24590528734)  | | [permission-check](https://github.com/aptos-labs/aptos-core/actions/runs/8952773295/job/24590528814) | 18s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8945046444/job/24573233319) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8945276212/job/24573942286) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8946621349/job/24577518767) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8947634424/job/24579806730) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8952773295/job/24590528814)  | | [permission-check](https://github.com/aptos-labs/aptos-core/actions/runs/8952773292/job/24590528602) | 18s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8945046443/job/24573233338) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8945276213/job/24573942023) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8946621352/job/24577518495) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8947634414/job/24579806142) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8952773292/job/24590528602)  | | [permission-check](https://github.com/aptos-labs/aptos-core/actions/runs/8952773288/job/24590528868) | 16s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8945046435/job/24573233484) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8945276216/job/24573942413) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8946621358/job/24577518676) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8947634406/job/24579806573) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8952773288/job/24590528868)  | | [permission-check](https://github.com/aptos-labs/aptos-core/actions/runs/8952773318/job/24590528979) | 4s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8947634455/job/24579806828) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8952773318/job/24590528979)  | | [determine-docker-build-metadata](https://github.com/aptos-labs/aptos-core/actions/runs/8952773318/job/24590530024) | 4s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8947634455/job/24579809160) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8952773318/job/24590530024)  |

🚨 5 jobs on the last run were significantly faster/slower than expected

Job Duration vs 7d avg Delta
framework-upgrade-determinator 11m 4m +177%
rust-lints 10m 6m +77%
rust-build-cached-packages 8m 5m +69%
cli-e2e-tests / run-cli-tests 9m 7m +38%
rust-images / rust-all 17m 13m +29%

settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io

github-actions[bot] commented 2 weeks ago

Forge is running suite realistic_env_max_load on e5ea021a0669ae5642cc6de992766e326e4933e7

github-actions[bot] commented 2 weeks ago

Forge is running suite compat on 01b24e7e3548382dd25440b39a0438a993387f12 ==> e5ea021a0669ae5642cc6de992766e326e4933e7

github-actions[bot] commented 2 weeks ago

:white_check_mark: Forge suite compat success on 01b24e7e3548382dd25440b39a0438a993387f12 ==> e5ea021a0669ae5642cc6de992766e326e4933e7

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> e5ea021a0669ae5642cc6de992766e326e4933e7 (PR)
1. Check liveness of validators at old version: 01b24e7e3548382dd25440b39a0438a993387f12
compatibility::simple-validator-upgrade::liveness-check : committed: 5394 txn/s, latency: 6079 ms, (p50: 5100 ms, p90: 9300 ms, p99: 13400 ms), latency samples: 188820
2. Upgrading first Validator to new version: e5ea021a0669ae5642cc6de992766e326e4933e7
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1447 txn/s, latency: 16258 ms, (p50: 19400 ms, p90: 23700 ms, p99: 24200 ms), latency samples: 88300
3. Upgrading rest of first batch to new version: e5ea021a0669ae5642cc6de992766e326e4933e7
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1086 txn/s, latency: 25397 ms, (p50: 25000 ms, p90: 33700 ms, p99: 34500 ms), latency samples: 56500
4. upgrading second batch to new version: e5ea021a0669ae5642cc6de992766e326e4933e7
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3581 txn/s, latency: 8823 ms, (p50: 9600 ms, p90: 12700 ms, p99: 13000 ms), latency samples: 143260
5. check swarm health
Compatibility test for 01b24e7e3548382dd25440b39a0438a993387f12 ==> e5ea021a0669ae5642cc6de992766e326e4933e7 passed
Test Ok
github-actions[bot] commented 2 weeks ago

:white_check_mark: Forge suite realistic_env_max_load success on e5ea021a0669ae5642cc6de992766e326e4933e7

two traffics test: inner traffic : committed: 7904 txn/s, latency: 4963 ms, (p50: 4800 ms, p90: 5700 ms, p99: 10200 ms), latency samples: 3414720
two traffics test : committed: 100 txn/s, latency: 1850 ms, (p50: 1800 ms, p90: 2000 ms, p99: 2300 ms), latency samples: 1700
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.202, avg: 0.201", "QsPosToProposal: max: 0.253, avg: 0.234", "ConsensusProposalToOrdered: max: 0.456, avg: 0.419", "ConsensusOrderedToCommit: max: 0.388, avg: 0.370", "ConsensusProposalToCommit: max: 0.805, avg: 0.789"]
Max round gap was 1 [limit 4] at version 1564515. Max no progress secs was 4.776789 [limit 15] at version 1564515.
Test Ok
github-actions[bot] commented 2 weeks ago

Forge is running suite realistic_env_max_load on e5ea021a0669ae5642cc6de992766e326e4933e7

github-actions[bot] commented 2 weeks ago

Forge is running suite compat on 01b24e7e3548382dd25440b39a0438a993387f12 ==> e5ea021a0669ae5642cc6de992766e326e4933e7

github-actions[bot] commented 2 weeks ago

Forge is running suite framework_upgrade on 01b24e7e3548382dd25440b39a0438a993387f12 ==> e5ea021a0669ae5642cc6de992766e326e4933e7

github-actions[bot] commented 2 weeks ago

:white_check_mark: Forge suite compat success on 01b24e7e3548382dd25440b39a0438a993387f12 ==> e5ea021a0669ae5642cc6de992766e326e4933e7

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> e5ea021a0669ae5642cc6de992766e326e4933e7 (PR)
1. Check liveness of validators at old version: 01b24e7e3548382dd25440b39a0438a993387f12
compatibility::simple-validator-upgrade::liveness-check : committed: 6950 txn/s, latency: 4879 ms, (p50: 4800 ms, p90: 8100 ms, p99: 9300 ms), latency samples: 236300
2. Upgrading first Validator to new version: e5ea021a0669ae5642cc6de992766e326e4933e7
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1843 txn/s, latency: 15648 ms, (p50: 19000 ms, p90: 22100 ms, p99: 22600 ms), latency samples: 92160
3. Upgrading rest of first batch to new version: e5ea021a0669ae5642cc6de992766e326e4933e7
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1703 txn/s, latency: 16259 ms, (p50: 19100 ms, p90: 21700 ms, p99: 22400 ms), latency samples: 88600
4. upgrading second batch to new version: e5ea021a0669ae5642cc6de992766e326e4933e7
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3590 txn/s, latency: 8727 ms, (p50: 9600 ms, p90: 12600 ms, p99: 12900 ms), latency samples: 143620
5. check swarm health
Compatibility test for 01b24e7e3548382dd25440b39a0438a993387f12 ==> e5ea021a0669ae5642cc6de992766e326e4933e7 passed
Test Ok
github-actions[bot] commented 2 weeks ago

:white_check_mark: Forge suite realistic_env_max_load success on e5ea021a0669ae5642cc6de992766e326e4933e7

two traffics test: inner traffic : committed: 8577 txn/s, latency: 4572 ms, (p50: 4400 ms, p90: 5400 ms, p99: 9600 ms), latency samples: 3705580
two traffics test : committed: 100 txn/s, latency: 1891 ms, (p50: 1800 ms, p90: 2000 ms, p99: 6600 ms), latency samples: 1740
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.211, avg: 0.203", "QsPosToProposal: max: 0.233, avg: 0.209", "ConsensusProposalToOrdered: max: 0.417, avg: 0.388", "ConsensusOrderedToCommit: max: 0.387, avg: 0.374", "ConsensusProposalToCommit: max: 0.775, avg: 0.762"]
Max round gap was 1 [limit 4] at version 1764012. Max no progress secs was 4.967057 [limit 15] at version 1764012.
Test Ok
github-actions[bot] commented 2 weeks ago

:white_check_mark: Forge suite framework_upgrade success on 01b24e7e3548382dd25440b39a0438a993387f12 ==> e5ea021a0669ae5642cc6de992766e326e4933e7

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> e5ea021a0669ae5642cc6de992766e326e4933e7 (PR)
Upgrade the nodes to version: e5ea021a0669ae5642cc6de992766e326e4933e7
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1380 txn/s, submitted: 1384 txn/s, failed submission: 3 txn/s, expired: 3 txn/s, latency: 2270 ms, (p50: 1800 ms, p90: 3900 ms, p99: 5000 ms), latency samples: 117380
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1225 txn/s, submitted: 1226 txn/s, failed submission: 1 txn/s, expired: 1 txn/s, latency: 2972 ms, (p50: 2400 ms, p90: 5700 ms, p99: 11700 ms), latency samples: 93160
5. check swarm health
Compatibility test for 01b24e7e3548382dd25440b39a0438a993387f12 ==> e5ea021a0669ae5642cc6de992766e326e4933e7 passed
Upgrade the remaining nodes to version: e5ea021a0669ae5642cc6de992766e326e4933e7
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1363 txn/s, submitted: 1365 txn/s, failed submission: 2 txn/s, expired: 2 txn/s, latency: 2623 ms, (p50: 2100 ms, p90: 4500 ms, p99: 6900 ms), latency samples: 109080
Test Ok