Remove the "remove_error_details" change

[areshand]

Description

withdrew this change due to negative ecosystem impact after the discussion

How Has This Been Tested?

Existing tests

Key Areas to Review

Type of Change

• [ ] New feature
• [ ] Bug fix
• [ ] Breaking change
• [ ] Performance improvement
• [ ] Refactoring
• [ ] Dependency update
• [ ] Documentation update
• [ ] Tests

Which Components or Systems Does This Change Impact?

• [ ] Validator Node
• [ ] Full Node (API, Indexer, etc.)
• [ ] Move/Aptos Virtual Machine
• [ ] Aptos Framework
• [ ] Aptos CLI/SDK
• [ ] Developer Infrastructure
• [ ] Move Compiler
• [ ] Other (specify)

Checklist

• [ ] I have read and followed the CONTRIBUTING doc
• [ ] I have performed a self-review of my own code
• [ ] I have commented my code, particularly in hard-to-understand areas
• [ ] I identified and added all stakeholders and component owners affected by this change as reviewers
• [ ] I tested both happy and unhappy path of the functionality
• [ ] I have made corresponding changes to the documentation

[trunk-io[bot]]

⏱️ 5h 53m total CI duration on this PR

| Slowest 15 Jobs | Cumulative Duration | Recent Runs | |---|---|---| | [execution-performance / single-node-performance](https://github.com/aptos-labs/aptos-core/actions/runs/11981991391/job/33409355752) | 2h 5m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11962816257/job/33352179410) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11964256604/job/33356390312) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11981422756/job/33407753906) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/11982154905/job/33409822051) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11981991391/job/33409355752) (+1 more) | | [test-target-determinator](https://github.com/aptos-labs/aptos-core/actions/runs/11982154771/job/33409738512) | 28m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11962816297/job/33352019183) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11964256562/job/33356265705) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11981422664/job/33407614561) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11981991356/job/33409229700) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11982154771/job/33409738512) (+1 more) | | [execution-performance / test-target-determinator](https://github.com/aptos-labs/aptos-core/actions/runs/11982154905/job/33409697472) | 26m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11962816257/job/33352010661) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11964256604/job/33356261017) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11981422756/job/33407611354) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11981991391/job/33409226754) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11982154905/job/33409697472) (+1 more) | | [check](https://github.com/aptos-labs/aptos-core/actions/runs/11982154861/job/33409697378) | 22m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11962816245/job/33352011008) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11964256584/job/33356261083) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11981422739/job/33407611176) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11981991374/job/33409226627) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11982154861/job/33409697378) (+1 more) | | [rust-cargo-deny](https://github.com/aptos-labs/aptos-core/actions/runs/11982154866/job/33409738921) | 17m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11962816256/job/33352025090) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11964256587/job/33356268642) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11981422749/job/33407618761) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11981991381/job/33409232554) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11982154866/job/33409738921) (+5 more) | | [rust-move-tests](https://github.com/aptos-labs/aptos-core/actions/runs/11981422755/job/33408320061) | 14m | [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/11981422755/job/33407611360) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/11981422755/job/33408320061)  | | [fetch-last-released-docker-image-tag](https://github.com/aptos-labs/aptos-core/actions/runs/11982154771/job/33409738319) | 10m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11962816297/job/33352019732) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11964256562/job/33356266085) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11981422664/job/33407615026) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11981991356/job/33409230026) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11982154771/job/33409738319) (+1 more) | | [check-dynamic-deps](https://github.com/aptos-labs/aptos-core/actions/runs/11982154922/job/33409697235) | 10m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11962816266/job/33352010148) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11964256585/job/33356260754) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11981422743/job/33407610901) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11981991382/job/33409226437) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11982154922/job/33409697235) (+5 more) | | [rust-move-tests](https://github.com/aptos-labs/aptos-core/actions/runs/11938671605/job/33277449965) | 8m | [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/11938671605/job/33277449965)  | | [rust-move-tests](https://github.com/aptos-labs/aptos-core/actions/runs/11960034582/job/33343164000) | 8m | [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/11960034582/job/33343164000)  | | [rust-move-tests](https://github.com/aptos-labs/aptos-core/actions/runs/11959662020/job/33341941043) | 7m | [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/11959662020/job/33341941043)  | | [rust-move-tests](https://github.com/aptos-labs/aptos-core/actions/runs/11959330808/job/33340884906) | 7m | [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/11959330808/job/33340884906)  | | [rust-move-tests](https://github.com/aptos-labs/aptos-core/actions/runs/11964256577/job/33356261284) | 7m | [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/11964256577/job/33356261284)  | | [rust-move-tests](https://github.com/aptos-labs/aptos-core/actions/runs/11962816258/job/33352010722) | 7m | [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/11962816258/job/33352010722)  | | [rust-move-tests](https://github.com/aptos-labs/aptos-core/actions/runs/11981991384/job/33409226779) | 7m | [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/11981991384/job/33409226779)  |

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

[github-actions[bot]]

Forge is running suite realistic_env_max_load on 0f9c1d0223e8993e688128a5d653ac75716f2b9f

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite framework_upgrade on a12510d9b95620907796f680dab32f9fc90e9bd9 ==> 0f9c1d0223e8993e688128a5d653ac75716f2b9f

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite compat on a12510d9b95620907796f680dab32f9fc90e9bd9 ==> 0f9c1d0223e8993e688128a5d653ac75716f2b9f

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite realistic_env_max_load success on 0f9c1d0223e8993e688128a5d653ac75716f2b9f

two traffics test: inner traffic : committed: 14503.27 txn/s, latency: 2742.28 ms, (p50: 2700 ms, p70: 2700, p90: 2900 ms, p99: 3500 ms), latency samples: 5514440
two traffics test : committed: 100.03 txn/s, latency: 1799.96 ms, (p50: 1400 ms, p70: 1400, p90: 1600 ms, p99: 11500 ms), latency samples: 1720
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 1.978, avg: 1.534", "ConsensusProposalToOrdered: max: 0.329, avg: 0.293", "ConsensusOrderedToCommit: max: 0.373, avg: 0.360", "ConsensusProposalToCommit: max: 0.662, avg: 0.652"]
Max non-epoch-change gap was: 1 rounds at version 54645 (avg 0.00) [limit 4], 1.91s no progress at version 54645 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.60s no progress at version 2898040 (avg 8.60s) [limit 15].
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite framework_upgrade success on a12510d9b95620907796f680dab32f9fc90e9bd9 ==> 0f9c1d0223e8993e688128a5d653ac75716f2b9f

Compatibility test results for a12510d9b95620907796f680dab32f9fc90e9bd9 ==> 0f9c1d0223e8993e688128a5d653ac75716f2b9f (PR)
Upgrade the nodes to version: 0f9c1d0223e8993e688128a5d653ac75716f2b9f
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1304.15 txn/s, submitted: 1306.18 txn/s, failed submission: 2.03 txn/s, expired: 2.03 txn/s, latency: 2442.01 ms, (p50: 2100 ms, p70: 2400, p90: 4200 ms, p99: 5700 ms), latency samples: 115700
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1301.31 txn/s, submitted: 1305.76 txn/s, failed submission: 4.45 txn/s, expired: 4.45 txn/s, latency: 2323.79 ms, (p50: 1800 ms, p70: 2400, p90: 4400 ms, p99: 6600 ms), latency samples: 117000
5. check swarm health
Compatibility test for a12510d9b95620907796f680dab32f9fc90e9bd9 ==> 0f9c1d0223e8993e688128a5d653ac75716f2b9f passed
Upgrade the remaining nodes to version: 0f9c1d0223e8993e688128a5d653ac75716f2b9f
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1194.87 txn/s, submitted: 1196.85 txn/s, failed submission: 1.97 txn/s, expired: 1.97 txn/s, latency: 2500.47 ms, (p50: 2100 ms, p70: 2700, p90: 4500 ms, p99: 5900 ms), latency samples: 109160
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite compat success on a12510d9b95620907796f680dab32f9fc90e9bd9 ==> 0f9c1d0223e8993e688128a5d653ac75716f2b9f

Compatibility test results for a12510d9b95620907796f680dab32f9fc90e9bd9 ==> 0f9c1d0223e8993e688128a5d653ac75716f2b9f (PR)
1. Check liveness of validators at old version: a12510d9b95620907796f680dab32f9fc90e9bd9
compatibility::simple-validator-upgrade::liveness-check : committed: 15074.58 txn/s, latency: 2293.07 ms, (p50: 2000 ms, p70: 2100, p90: 2200 ms, p99: 7300 ms), latency samples: 544980
2. Upgrading first Validator to new version: 0f9c1d0223e8993e688128a5d653ac75716f2b9f
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 7734.64 txn/s, latency: 3739.06 ms, (p50: 4100 ms, p70: 4400, p90: 4500 ms, p99: 4600 ms), latency samples: 144700
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 6840.66 txn/s, latency: 4502.37 ms, (p50: 4500 ms, p70: 4600, p90: 4900 ms, p99: 6700 ms), latency samples: 258580
3. Upgrading rest of first batch to new version: 0f9c1d0223e8993e688128a5d653ac75716f2b9f
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 7881.15 txn/s, latency: 3503.16 ms, (p50: 3600 ms, p70: 3900, p90: 4800 ms, p99: 5200 ms), latency samples: 145980
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 7854.11 txn/s, latency: 4069.64 ms, (p50: 4200 ms, p70: 4300, p90: 6200 ms, p99: 6500 ms), latency samples: 262140
4. upgrading second batch to new version: 0f9c1d0223e8993e688128a5d653ac75716f2b9f
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 9890.62 txn/s, latency: 2732.46 ms, (p50: 2500 ms, p70: 3400, p90: 4300 ms, p99: 4700 ms), latency samples: 173420
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 10567.32 txn/s, latency: 2933.88 ms, (p50: 2500 ms, p70: 2900, p90: 5400 ms, p99: 6600 ms), latency samples: 345020
5. check swarm health
Compatibility test for a12510d9b95620907796f680dab32f9fc90e9bd9 ==> 0f9c1d0223e8993e688128a5d653ac75716f2b9f passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on 5380edcb41710e2765d2e44b06f5048fb7b85eb5

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite compat on a734d235af377347196f25d90291df51a23915b5 ==> 5380edcb41710e2765d2e44b06f5048fb7b85eb5

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite framework_upgrade on a734d235af377347196f25d90291df51a23915b5 ==> 5380edcb41710e2765d2e44b06f5048fb7b85eb5

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite realistic_env_max_load success on 5380edcb41710e2765d2e44b06f5048fb7b85eb5

two traffics test: inner traffic : committed: 14277.88 txn/s, latency: 2786.37 ms, (p50: 2700 ms, p70: 2700, p90: 3000 ms, p99: 6000 ms), latency samples: 5428920
two traffics test : committed: 100.02 txn/s, latency: 1459.51 ms, (p50: 1400 ms, p70: 1400, p90: 1500 ms, p99: 5900 ms), latency samples: 1800
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 1.971, avg: 1.566", "ConsensusProposalToOrdered: max: 0.332, avg: 0.295", "ConsensusOrderedToCommit: max: 0.384, avg: 0.368", "ConsensusProposalToCommit: max: 0.678, avg: 0.664"]
Max non-epoch-change gap was: 1 rounds at version 5411247 (avg 0.00) [limit 4], 1.84s no progress at version 5428294 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.32s no progress at version 2828575 (avg 8.32s) [limit 15].
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite framework_upgrade success on a734d235af377347196f25d90291df51a23915b5 ==> 5380edcb41710e2765d2e44b06f5048fb7b85eb5

Compatibility test results for a734d235af377347196f25d90291df51a23915b5 ==> 5380edcb41710e2765d2e44b06f5048fb7b85eb5 (PR)
Upgrade the nodes to version: 5380edcb41710e2765d2e44b06f5048fb7b85eb5
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1316.90 txn/s, submitted: 1318.50 txn/s, failed submission: 1.60 txn/s, expired: 1.60 txn/s, latency: 2402.71 ms, (p50: 2100 ms, p70: 2400, p90: 3900 ms, p99: 6000 ms), latency samples: 115380
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1328.81 txn/s, submitted: 1332.78 txn/s, failed submission: 3.97 txn/s, expired: 3.97 txn/s, latency: 2243.32 ms, (p50: 2100 ms, p70: 2400, p90: 3600 ms, p99: 5400 ms), latency samples: 120500
5. check swarm health
Compatibility test for a734d235af377347196f25d90291df51a23915b5 ==> 5380edcb41710e2765d2e44b06f5048fb7b85eb5 passed
Upgrade the remaining nodes to version: 5380edcb41710e2765d2e44b06f5048fb7b85eb5
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1313.26 txn/s, submitted: 1316.23 txn/s, failed submission: 2.97 txn/s, expired: 2.97 txn/s, latency: 2420.51 ms, (p50: 2400 ms, p70: 2700, p90: 3600 ms, p99: 5400 ms), latency samples: 114980
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite compat success on a734d235af377347196f25d90291df51a23915b5 ==> 5380edcb41710e2765d2e44b06f5048fb7b85eb5

Compatibility test results for a734d235af377347196f25d90291df51a23915b5 ==> 5380edcb41710e2765d2e44b06f5048fb7b85eb5 (PR)
1. Check liveness of validators at old version: a734d235af377347196f25d90291df51a23915b5
compatibility::simple-validator-upgrade::liveness-check : committed: 14355.81 txn/s, latency: 2284.25 ms, (p50: 1800 ms, p70: 1900, p90: 2500 ms, p99: 15900 ms), latency samples: 556740
2. Upgrading first Validator to new version: 5380edcb41710e2765d2e44b06f5048fb7b85eb5
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 7775.30 txn/s, latency: 3561.09 ms, (p50: 3800 ms, p70: 4300, p90: 4800 ms, p99: 5000 ms), latency samples: 139760
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 7924.55 txn/s, latency: 4075.75 ms, (p50: 4300 ms, p70: 4400, p90: 5800 ms, p99: 6000 ms), latency samples: 260240
3. Upgrading rest of first batch to new version: 5380edcb41710e2765d2e44b06f5048fb7b85eb5
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 8015.65 txn/s, latency: 3618.23 ms, (p50: 4000 ms, p70: 4200, p90: 4300 ms, p99: 4400 ms), latency samples: 147520
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 7212.89 txn/s, latency: 4057.50 ms, (p50: 4300 ms, p70: 4300, p90: 4400 ms, p99: 5500 ms), latency samples: 271700
4. upgrading second batch to new version: 5380edcb41710e2765d2e44b06f5048fb7b85eb5
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 12015.71 txn/s, latency: 2279.99 ms, (p50: 2200 ms, p70: 2500, p90: 3300 ms, p99: 3800 ms), latency samples: 208700
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 11227.92 txn/s, latency: 2726.79 ms, (p50: 2400 ms, p70: 2700, p90: 5000 ms, p99: 7200 ms), latency samples: 366000
5. check swarm health
Compatibility test for a734d235af377347196f25d90291df51a23915b5 ==> 5380edcb41710e2765d2e44b06f5048fb7b85eb5 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on e398992faa7ef8664263ce6a3273ab2b325c5583

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite compat on a734d235af377347196f25d90291df51a23915b5 ==> e398992faa7ef8664263ce6a3273ab2b325c5583

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite framework_upgrade on a734d235af377347196f25d90291df51a23915b5 ==> e398992faa7ef8664263ce6a3273ab2b325c5583

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite realistic_env_max_load success on e398992faa7ef8664263ce6a3273ab2b325c5583

two traffics test: inner traffic : committed: 14340.43 txn/s, latency: 2773.52 ms, (p50: 2700 ms, p70: 2700, p90: 3000 ms, p99: 3300 ms), latency samples: 5452540
two traffics test : committed: 100.06 txn/s, latency: 1564.02 ms, (p50: 1400 ms, p70: 1500, p90: 1600 ms, p99: 7600 ms), latency samples: 1780
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 2.021, avg: 1.582", "ConsensusProposalToOrdered: max: 0.328, avg: 0.294", "ConsensusOrderedToCommit: max: 0.373, avg: 0.361", "ConsensusProposalToCommit: max: 0.666, avg: 0.655"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.83s no progress at version 2702420 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.54s no progress at version 2702418 (avg 8.54s) [limit 15].
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite compat success on a734d235af377347196f25d90291df51a23915b5 ==> e398992faa7ef8664263ce6a3273ab2b325c5583

Compatibility test results for a734d235af377347196f25d90291df51a23915b5 ==> e398992faa7ef8664263ce6a3273ab2b325c5583 (PR)
1. Check liveness of validators at old version: a734d235af377347196f25d90291df51a23915b5
compatibility::simple-validator-upgrade::liveness-check : committed: 14022.39 txn/s, latency: 2390.36 ms, (p50: 1800 ms, p70: 2000, p90: 3300 ms, p99: 14200 ms), latency samples: 456320
2. Upgrading first Validator to new version: e398992faa7ef8664263ce6a3273ab2b325c5583
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 6762.43 txn/s, latency: 4237.41 ms, (p50: 4700 ms, p70: 5300, p90: 5900 ms, p99: 6100 ms), latency samples: 123280
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 5955.17 txn/s, latency: 5383.37 ms, (p50: 5500 ms, p70: 5900, p90: 7200 ms, p99: 7700 ms), latency samples: 227720
3. Upgrading rest of first batch to new version: e398992faa7ef8664263ce6a3273ab2b325c5583
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 7400.42 txn/s, latency: 3869.57 ms, (p50: 4300 ms, p70: 4600, p90: 4600 ms, p99: 4700 ms), latency samples: 142260
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 7452.88 txn/s, latency: 4306.66 ms, (p50: 4500 ms, p70: 4800, p90: 5200 ms, p99: 5600 ms), latency samples: 251500
4. upgrading second batch to new version: e398992faa7ef8664263ce6a3273ab2b325c5583
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 12182.74 txn/s, latency: 2297.20 ms, (p50: 2500 ms, p70: 2600, p90: 2800 ms, p99: 2900 ms), latency samples: 210900
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 10504.68 txn/s, latency: 3030.19 ms, (p50: 2700 ms, p70: 2800, p90: 5900 ms, p99: 7600 ms), latency samples: 377980
5. check swarm health
Compatibility test for a734d235af377347196f25d90291df51a23915b5 ==> e398992faa7ef8664263ce6a3273ab2b325c5583 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite framework_upgrade success on a734d235af377347196f25d90291df51a23915b5 ==> e398992faa7ef8664263ce6a3273ab2b325c5583

Compatibility test results for a734d235af377347196f25d90291df51a23915b5 ==> e398992faa7ef8664263ce6a3273ab2b325c5583 (PR)
Upgrade the nodes to version: e398992faa7ef8664263ce6a3273ab2b325c5583
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1271.55 txn/s, submitted: 1274.25 txn/s, failed submission: 2.70 txn/s, expired: 2.70 txn/s, latency: 2479.64 ms, (p50: 2100 ms, p70: 2400, p90: 4300 ms, p99: 6900 ms), latency samples: 113060
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1265.38 txn/s, submitted: 1268.05 txn/s, failed submission: 2.67 txn/s, expired: 2.67 txn/s, latency: 2405.20 ms, (p50: 2100 ms, p70: 2500, p90: 3900 ms, p99: 5200 ms), latency samples: 113860
5. check swarm health
Compatibility test for a734d235af377347196f25d90291df51a23915b5 ==> e398992faa7ef8664263ce6a3273ab2b325c5583 passed
Upgrade the remaining nodes to version: e398992faa7ef8664263ce6a3273ab2b325c5583
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1352.65 txn/s, submitted: 1355.31 txn/s, failed submission: 2.66 txn/s, expired: 2.66 txn/s, latency: 2340.97 ms, (p50: 2100 ms, p70: 2400, p90: 4500 ms, p99: 6000 ms), latency samples: 122120
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite compat on 327445f70a3b9474f13a792e8ada9b85809dec86 ==> 0e6393e3e30e2e00640607e187d49e6da42a3094

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on 0e6393e3e30e2e00640607e187d49e6da42a3094

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite framework_upgrade on 327445f70a3b9474f13a792e8ada9b85809dec86 ==> 0e6393e3e30e2e00640607e187d49e6da42a3094

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite realistic_env_max_load success on 0e6393e3e30e2e00640607e187d49e6da42a3094

two traffics test: inner traffic : committed: 14490.27 txn/s, latency: 2746.05 ms, (p50: 2700 ms, p70: 2700, p90: 2900 ms, p99: 3000 ms), latency samples: 5509540
two traffics test : committed: 99.99 txn/s, latency: 1445.91 ms, (p50: 1400 ms, p70: 1400, p90: 1600 ms, p99: 5200 ms), latency samples: 1800
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 1.985, avg: 1.509", "ConsensusProposalToOrdered: max: 0.318, avg: 0.292", "ConsensusOrderedToCommit: max: 0.375, avg: 0.361", "ConsensusProposalToCommit: max: 0.666, avg: 0.653"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.87s no progress at version 2971052 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.63s no progress at version 2971050 (avg 8.63s) [limit 15].
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite framework_upgrade success on 327445f70a3b9474f13a792e8ada9b85809dec86 ==> 0e6393e3e30e2e00640607e187d49e6da42a3094

Compatibility test results for 327445f70a3b9474f13a792e8ada9b85809dec86 ==> 0e6393e3e30e2e00640607e187d49e6da42a3094 (PR)
Upgrade the nodes to version: 0e6393e3e30e2e00640607e187d49e6da42a3094
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1409.74 txn/s, submitted: 1413.31 txn/s, failed submission: 3.57 txn/s, expired: 3.57 txn/s, latency: 2323.79 ms, (p50: 2100 ms, p70: 2400, p90: 4200 ms, p99: 5500 ms), latency samples: 126460
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1376.38 txn/s, submitted: 1378.38 txn/s, failed submission: 1.86 txn/s, expired: 2.00 txn/s, latency: 2287.34 ms, (p50: 2100 ms, p70: 2400, p90: 3300 ms, p99: 5200 ms), latency samples: 118508
5. check swarm health
Compatibility test for 327445f70a3b9474f13a792e8ada9b85809dec86 ==> 0e6393e3e30e2e00640607e187d49e6da42a3094 passed
Upgrade the remaining nodes to version: 0e6393e3e30e2e00640607e187d49e6da42a3094
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 976.51 txn/s, submitted: 979.81 txn/s, failed submission: 3.31 txn/s, expired: 3.31 txn/s, latency: 3244.47 ms, (p50: 2400 ms, p70: 3900, p90: 6300 ms, p99: 8000 ms), latency samples: 88620
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite compat success on 327445f70a3b9474f13a792e8ada9b85809dec86 ==> 0e6393e3e30e2e00640607e187d49e6da42a3094

Compatibility test results for 327445f70a3b9474f13a792e8ada9b85809dec86 ==> 0e6393e3e30e2e00640607e187d49e6da42a3094 (PR)
1. Check liveness of validators at old version: 327445f70a3b9474f13a792e8ada9b85809dec86
compatibility::simple-validator-upgrade::liveness-check : committed: 14891.97 txn/s, latency: 1955.05 ms, (p50: 1900 ms, p70: 2100, p90: 2300 ms, p99: 2700 ms), latency samples: 559940
2. Upgrading first Validator to new version: 0e6393e3e30e2e00640607e187d49e6da42a3094
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 7560.04 txn/s, latency: 3862.23 ms, (p50: 4300 ms, p70: 4400, p90: 4600 ms, p99: 4600 ms), latency samples: 143900
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 6757.76 txn/s, latency: 4512.28 ms, (p50: 4600 ms, p70: 4700, p90: 4800 ms, p99: 6700 ms), latency samples: 252020
3. Upgrading rest of first batch to new version: 0e6393e3e30e2e00640607e187d49e6da42a3094
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 6865.36 txn/s, latency: 4151.78 ms, (p50: 4800 ms, p70: 5000, p90: 5100 ms, p99: 5200 ms), latency samples: 125060
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 6433.20 txn/s, latency: 5043.31 ms, (p50: 5100 ms, p70: 5200, p90: 6800 ms, p99: 7200 ms), latency samples: 218060
4. upgrading second batch to new version: 0e6393e3e30e2e00640607e187d49e6da42a3094
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 10347.93 txn/s, latency: 2651.33 ms, (p50: 2600 ms, p70: 2900, p90: 4000 ms, p99: 4300 ms), latency samples: 187400
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 10698.16 txn/s, latency: 2935.40 ms, (p50: 2600 ms, p70: 2900, p90: 5500 ms, p99: 6700 ms), latency samples: 347480
5. check swarm health
Compatibility test for 327445f70a3b9474f13a792e8ada9b85809dec86 ==> 0e6393e3e30e2e00640607e187d49e6da42a3094 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on fd5f4baa28c8e7599dce956102848067b716f51d

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite framework_upgrade on 327445f70a3b9474f13a792e8ada9b85809dec86 ==> fd5f4baa28c8e7599dce956102848067b716f51d

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite compat on 327445f70a3b9474f13a792e8ada9b85809dec86 ==> fd5f4baa28c8e7599dce956102848067b716f51d

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on e5d87528ec68b548b36ae2807cef67f0bb2ca71a

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite framework_upgrade on 327445f70a3b9474f13a792e8ada9b85809dec86 ==> e5d87528ec68b548b36ae2807cef67f0bb2ca71a

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite compat on 327445f70a3b9474f13a792e8ada9b85809dec86 ==> e5d87528ec68b548b36ae2807cef67f0bb2ca71a

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:x: Forge suite framework_upgrade failure on 327445f70a3b9474f13a792e8ada9b85809dec86 ==> e5d87528ec68b548b36ae2807cef67f0bb2ca71a

Compatibility test results for 327445f70a3b9474f13a792e8ada9b85809dec86 ==> e5d87528ec68b548b36ae2807cef67f0bb2ca71a (PR)
Upgrade the nodes to version: e5d87528ec68b548b36ae2807cef67f0bb2ca71a
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 292.38 txn/s, submitted: 322.87 txn/s, failed submission: 0.68 txn/s, expired: 30.49 txn/s, latency: 2088.85 ms, (p50: 2100 ms, p70: 2400, p90: 3000 ms, p99: 4000 ms), latency samples: 94885
Test Failed: Waiting for nodes to catch up to target version and epoch (None, Some(9)) timed out after 20 seconds, current status: Ok([("validator-0", 96123, 8), ("validator-1", 96123, 8), ("validator-2", 96123, 8), ("validator-3", 96123, 8)])

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.89/src/error.rs:85:36
   1: aptos_forge::interface::swarm::wait_for_all_nodes_to_catchup_to_target_version_or_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:463:24
   2: aptos_forge::interface::swarm::wait_for_all_nodes_to_catchup_to_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:396:6
   3: aptos_forge::interface::swarm::SwarmExt::wait_for_all_nodes_to_change_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:289:92
   4: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   5: aptos_forge::interface::swarm::SwarmExt::fork_check::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:200:14
   6: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   7: <aptos_testcases::framework_upgrade::FrameworkUpgrade as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/framework_upgrade.rs:93:63
   8: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   9: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:63
  10: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:107:5
  11: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:73:5
  12: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:31
  13: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/blocking.rs:66:9
  14: tokio::runtime::handle::Handle::block_on_inner::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:324:22
  15: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16
  16: tokio::runtime::handle::Handle::block_on_inner
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:323:9
  17: tokio::runtime::handle::Handle::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:302:18
  18: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:332:50
  19: forge::run_forge_with_changelog
             at ./testsuite/forge-cli/src/main.rs:426:24
  20: forge::main
             at ./testsuite/forge-cli/src/main.rs:329:21
  21: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
  22: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
  23: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
  24: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
  25: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  26: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  27: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  28: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  29: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  30: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  31: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  32: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  33: main
  34: __libc_start_main
  35: _start
Trailing Log Lines:
  30: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  31: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  32: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  33: main
  34: __libc_start_main
  35: _start

=== BEGIN JUNIT ===
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="forge" tests="1" failures="1" errors="0" uuid="ecafb230-671d-4a75-a625-274978034b6e">
    <testsuite name="local" tests="1" disabled="0" errors="0" failures="1">
        <testcase name="framework_upgrade::framework-upgrade">
            <failure message="Waiting for nodes to catch up to target version and epoch (None, Some(9)) timed out after 20 seconds, current status: Ok([(&quot;validator-0&quot;, 96123, 8), (&quot;validator-1&quot;, 96123, 8), (&quot;validator-2&quot;, 96123, 8), (&quot;validator-3&quot;, 96123, 8)])

Stack backtrace:
   0: anyhow::error::&lt;impl anyhow::Error&gt;::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.89/src/error.rs:85:36
   1: aptos_forge::interface::swarm::wait_for_all_nodes_to_catchup_to_target_version_or_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:463:24
   2: aptos_forge::interface::swarm::wait_for_all_nodes_to_catchup_to_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:396:6
   3: aptos_forge::interface::swarm::SwarmExt::wait_for_all_nodes_to_change_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:289:92
   4: &lt;core::pin::Pin&lt;P&gt; as core::future::future::Future&gt;::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   5: aptos_forge::interface::swarm::SwarmExt::fork_check::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:200:14
   6: &lt;core::pin::Pin&lt;P&gt; as core::future::future::Future&gt;::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   7: &lt;aptos_testcases::framework_upgrade::FrameworkUpgrade as aptos_forge::interface::network::NetworkTest&gt;::run::{{closure}}
             at ./testsuite/testcases/src/framework_upgrade.rs:93:63
   8: &lt;core::pin::Pin&lt;P&gt; as core::future::future::Future&gt;::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   9: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:63
  10: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:107:5
  11: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:73:5
  12: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:31
  13: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/blocking.rs:66:9
  14: tokio::runtime::handle::Handle::block_on_inner::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:324:22
  15: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16
  16: tokio::runtime::handle::Handle::block_on_inner
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:323:9
  17: tokio::runtime::handle::Handle::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:302:18
  18: aptos_forge::runner::Forge&lt;F&gt;::run
             at ./testsuite/forge/src/runner.rs:332:50
  19: forge::run_forge_with_changelog
             at ./testsuite/forge-cli/src/main.rs:426:24
  20: forge::main
             at ./testsuite/forge-cli/src/main.rs:329:21
  21: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
  22: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
  23: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
  24: core::ops::function::impls::&lt;impl core::ops::function::FnOnce&lt;A&gt; for &amp;F&gt;::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
  25: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  26: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  27: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  28: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  29: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  30: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  31: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  32: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  33: main
  34: __libc_start_main
  35: _start"/>
        </testcase>
    </testsuite>
</testsuites>
=== END JUNIT ===

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:381"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-15331-1732322031-327445f70a3b9474f13","timestamp":"2024-11-23T00:44:13.223002Z","message":"Deleting namespace forge-framework-upgrade-pr-15331: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:398"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-15331-1732322031-327445f70a3b9474f13","timestamp":"2024-11-23T00:44:13.223024Z","message":"aptos-node resources for Forge removed in namespace: forge-framework-upgrade-pr-15331"}

failures:
    framework_upgrade::framework-upgrade

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.89/src/error.rs:85:36
   1: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:358:13
   2: forge::run_forge_with_changelog
             at ./testsuite/forge-cli/src/main.rs:426:24
   3: forge::main
             at ./testsuite/forge-cli/src/main.rs:329:21
   4: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
   5: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
   6: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
   7: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
   8: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
   9: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  10: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  11: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  12: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  13: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  14: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  15: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  16: main
  17: __libc_start_main
  18: _start
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-validator-0                1/1     Running     0          7m20s
aptos-node-1-validator-0                1/1     Running     0          6m37s
aptos-node-2-validator-0                1/1     Running     0          9m32s
aptos-node-3-validator-0                1/1     Running     0          9m32s
genesis-aptos-genesis-eforge175-kf5l6   0/1     Completed   0          9m33s

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite realistic_env_max_load success on e5d87528ec68b548b36ae2807cef67f0bb2ca71a

two traffics test: inner traffic : committed: 14511.95 txn/s, latency: 2741.24 ms, (p50: 2700 ms, p70: 2700, p90: 2900 ms, p99: 3000 ms), latency samples: 5517860
two traffics test : committed: 100.00 txn/s, latency: 1597.99 ms, (p50: 1300 ms, p70: 1400, p90: 1500 ms, p99: 10000 ms), latency samples: 1760
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 2.006, avg: 1.547", "ConsensusProposalToOrdered: max: 0.318, avg: 0.292", "ConsensusOrderedToCommit: max: 0.378, avg: 0.364", "ConsensusProposalToCommit: max: 0.667, avg: 0.656"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.49s no progress at version 31279 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.62s no progress at version 2517295 (avg 8.62s) [limit 15].
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite compat success on 327445f70a3b9474f13a792e8ada9b85809dec86 ==> e5d87528ec68b548b36ae2807cef67f0bb2ca71a

Compatibility test results for 327445f70a3b9474f13a792e8ada9b85809dec86 ==> e5d87528ec68b548b36ae2807cef67f0bb2ca71a (PR)
1. Check liveness of validators at old version: 327445f70a3b9474f13a792e8ada9b85809dec86
compatibility::simple-validator-upgrade::liveness-check : committed: 13049.73 txn/s, latency: 2617.06 ms, (p50: 1700 ms, p70: 1900, p90: 2800 ms, p99: 25400 ms), latency samples: 492780
2. Upgrading first Validator to new version: e5d87528ec68b548b36ae2807cef67f0bb2ca71a
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 6808.33 txn/s, latency: 4055.42 ms, (p50: 4400 ms, p70: 4500, p90: 5300 ms, p99: 5500 ms), latency samples: 144060
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 7421.26 txn/s, latency: 4349.21 ms, (p50: 4500 ms, p70: 4600, p90: 6500 ms, p99: 6700 ms), latency samples: 250120
3. Upgrading rest of first batch to new version: e5d87528ec68b548b36ae2807cef67f0bb2ca71a
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 7029.41 txn/s, latency: 3981.54 ms, (p50: 4200 ms, p70: 4600, p90: 5400 ms, p99: 5700 ms), latency samples: 130540
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 7277.11 txn/s, latency: 4443.45 ms, (p50: 4700 ms, p70: 4800, p90: 6100 ms, p99: 6500 ms), latency samples: 243260
4. upgrading second batch to new version: e5d87528ec68b548b36ae2807cef67f0bb2ca71a
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 11720.80 txn/s, latency: 2321.29 ms, (p50: 2400 ms, p70: 2700, p90: 3200 ms, p99: 3400 ms), latency samples: 200440
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 11179.61 txn/s, latency: 2809.93 ms, (p50: 2600 ms, p70: 2800, p90: 5000 ms, p99: 6000 ms), latency samples: 358940
5. check swarm health
Compatibility test for 327445f70a3b9474f13a792e8ada9b85809dec86 ==> e5d87528ec68b548b36ae2807cef67f0bb2ca71a passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking