[Not for review]

[vusirikala]

Description

Type of Change

• [ ] New feature
• [ ] Bug fix
• [ ] Breaking change
• [ ] Performance improvement
• [ ] Refactoring
• [ ] Dependency update
• [ ] Documentation update
• [ ] Tests

Which Components or Systems Does This Change Impact?

• [ ] Validator Node
• [ ] Full Node (API, Indexer, etc.)
• [ ] Move/Aptos Virtual Machine
• [ ] Aptos Framework
• [ ] Aptos CLI/SDK
• [ ] Developer Infrastructure
• [ ] Other (specify)

How Has This Been Tested?

Key Areas to Review

Checklist

• [ ] I have read and followed the CONTRIBUTING doc
• [ ] I have performed a self-review of my own code
• [ ] I have commented my code, particularly in hard-to-understand areas
• [ ] I identified and added all stakeholders and component owners affected by this change as reviewers
• [ ] I tested both happy and unhappy path of the functionality
• [ ] I have made corresponding changes to the documentation

[trunk-io[bot]]

⏱️ 40h 9m total CI duration on this PR

| Job | Cumulative Duration | Recent Runs | |---|---|---| | [rust-smoke-tests](https://github.com/aptos-labs/aptos-core/actions/runs/9166674650/job/25202537667) | 7h 14m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135291584/job/25122400668) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9135631314/job/25123275561) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9136246001/job/25124783838) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9150240565/job/25154792817) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9166674650/job/25202537667) (+9 more) | | [execution-performance / single-node-performance](https://github.com/aptos-labs/aptos-core/actions/runs/9166674678/job/25202619499) | 5h 45m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135291597/job/25122460867) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135631322/job/25123339314) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9136246021/job/25124835010) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9150240582/job/25154849604) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9166674678/job/25202619499) (+9 more) | | [rust-targeted-unit-tests](https://github.com/aptos-labs/aptos-core/actions/runs/9166674650/job/25202538101) | 4h 45m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135291584/job/25122400907) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135631314/job/25123275723) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9136246001/job/25124784057) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9150240565/job/25154792729) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9166674650/job/25202538101) (+9 more) | | [forge-e2e-test / forge](https://github.com/aptos-labs/aptos-core/actions/runs/9166674611/job/25202967620) | 3h 53m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135291532/job/25122634309) [⬜](https://github.com/aptos-labs/aptos-core/actions/runs/9135631232/job/25123622843) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9136245878/job/25124990763) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9150240564/job/25154989278) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9166674611/job/25202967620) (+9 more) | | [forge-compat-test / forge](https://github.com/aptos-labs/aptos-core/actions/runs/9166674611/job/25202967495) | 2h 58m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135291532/job/25122633902) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135631232/job/25123622697) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9136245878/job/25124990994) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9150240564/job/25154989200) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9166674611/job/25202967495) (+8 more) | | [rust-images / rust-all](https://github.com/aptos-labs/aptos-core/actions/runs/9166674611/job/25202539685) | 2h 51m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135291532/job/25122403199) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135631232/job/25123278032) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9136245878/job/25124789528) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9150240564/job/25154794144) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9166674611/job/25202539685) (+9 more) | | [rust-move-tests](https://github.com/aptos-labs/aptos-core/actions/runs/9166674648/job/25202529720) | 2h 41m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135291587/job/25122396115) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135631309/job/25123270450) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9136246005/job/25124776487) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9150240577/job/25154789065) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9166674648/job/25202529720) (+10 more) | | [cli-e2e-tests / run-cli-tests](https://github.com/aptos-labs/aptos-core/actions/runs/9166674611/job/25202967345) | 1h 51m | [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9135291532/job/25122634045) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135631232/job/25123622294) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9136245878/job/25124990615) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9150240564/job/25154988894) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9166674611/job/25202967345) (+9 more) | | [rust-lints](https://github.com/aptos-labs/aptos-core/actions/runs/9166674650/job/25202537916) | 1h 18m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135291584/job/25122400540) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135631314/job/25123275148) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9136246001/job/25124783745) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9150240565/job/25154792652) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9166674650/job/25202537916) (+9 more) | | [rust-build-cached-packages](https://github.com/aptos-labs/aptos-core/actions/runs/9166674650/job/25202538244) | 1h 9m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135291584/job/25122401026) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135631314/job/25123275292) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9136246001/job/25124784127) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9150240565/job/25154792912) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9166674650/job/25202538244) (+9 more) | | [run-tests-main-branch](https://github.com/aptos-labs/aptos-core/actions/runs/9166674571/job/25202539839) | 59m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135291525/job/25122403069) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135631226/job/25123278470) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9136245879/job/25124784359) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9150240541/job/25154795476) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9166674571/job/25202539839) (+9 more) | | [check](https://github.com/aptos-labs/aptos-core/actions/runs/9166674647/job/25202529632) | 50m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135291586/job/25122396120) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135631311/job/25123270466) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9136246003/job/25124776419) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9150240570/job/25154788984) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9166674647/job/25202529632) (+9 more) | | [execution-performance / test-target-determinator](https://github.com/aptos-labs/aptos-core/actions/runs/9166674678/job/25202529733) | 47m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135291597/job/25122396206) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135631322/job/25123270481) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9136246021/job/25124776517) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9150240582/job/25154789048) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9166674678/job/25202529733) (+9 more) | | [test-target-determinator](https://github.com/aptos-labs/aptos-core/actions/runs/9166674611/job/25202535507) | 46m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135291532/job/25122398988) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135631232/job/25123273444) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9136245878/job/25124786827) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9150240564/job/25154791597) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9166674611/job/25202535507) (+9 more) | | [adhoc-forge-test / forge](https://github.com/aptos-labs/aptos-core/actions/runs/9103420539/job/25025238153) | 41m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9103420539/job/25025238153)  | | [indexer-grpc-e2e-tests / test-indexer-grpc-docker-compose](https://github.com/aptos-labs/aptos-core/actions/runs/9166674611/job/25202966799) | 29m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135291532/job/25122634439) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9135631232/job/25123622956) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9136245878/job/25124990825) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9150240564/job/25154989002) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9166674611/job/25202966799) (+9 more) | | [general-lints](https://github.com/aptos-labs/aptos-core/actions/runs/9166674650/job/25202537452) | 25m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135291584/job/25122400431) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135631314/job/25123274961) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9136246001/job/25124783915) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9150240565/job/25154792557) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9166674650/job/25202537452) (+9 more) | | [check-dynamic-deps](https://github.com/aptos-labs/aptos-core/actions/runs/9166674655/job/25202529307) | 19m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135291585/job/25122395909) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135631313/job/25123270324) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9136246004/job/25124776215) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9150240566/job/25154788753) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9166674655/job/25202529307) (+10 more) | | [node-api-compatibility-tests / node-api-compatibility-tests](https://github.com/aptos-labs/aptos-core/actions/runs/9166674611/job/25202967173) | 11m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135291532/job/25122634725) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135631232/job/25123622438) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9136245878/job/25124990693) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9150240564/job/25154989120) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9166674611/job/25202967173) (+8 more) | | [semgrep/ci](https://github.com/aptos-labs/aptos-core/actions/runs/9166674656/job/25202529288) | 5m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135291588/job/25122395956) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135631308/job/25123270196) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9136246009/job/25124776218) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9150240567/job/25154788749) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9166674656/job/25202529288) (+9 more) | | [file_change_determinator](https://github.com/aptos-labs/aptos-core/actions/runs/9166674650/job/25202529805) | 2m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135291584/job/25122396084) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135631314/job/25123270599) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9136246001/job/25124776482) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9150240565/job/25154788987) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9166674650/job/25202529805) (+9 more) | | [file_change_determinator](https://github.com/aptos-labs/aptos-core/actions/runs/9166674571/job/25202533042) | 2m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135291525/job/25122397981) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135631226/job/25123272581) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9136245879/job/25124779030) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9150240541/job/25154790496) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9166674571/job/25202533042) (+9 more) | | [file_change_determinator](https://github.com/aptos-labs/aptos-core/actions/runs/9166674611/job/25202535331) | 2m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135291532/job/25122398889) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135631232/job/25123273207) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9136245878/job/25124786902) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9150240564/job/25154791503) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9166674611/job/25202535331) (+8 more) | | [permission-check](https://github.com/aptos-labs/aptos-core/actions/runs/9166674571/job/25202529126) | 51s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135291525/job/25122395804) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135631226/job/25123269907) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9136245879/job/25124775973) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9150240541/job/25154788823) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9166674571/job/25202529126) (+8 more) | | [permission-check](https://github.com/aptos-labs/aptos-core/actions/runs/9166674611/job/25202529816) | 47s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135291532/job/25122395898) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135631232/job/25123270053) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9136245878/job/25124784817) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9150240564/job/25154789174) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9166674611/job/25202529816) (+8 more) | | [permission-check](https://github.com/aptos-labs/aptos-core/actions/runs/9166674568/job/25202528789) | 41s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135291524/job/25122395718) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135631223/job/25123269759) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9136245873/job/25124775867) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9150240540/job/25154788653) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9166674568/job/25202528789) (+8 more) | | [permission-check](https://github.com/aptos-labs/aptos-core/actions/runs/9166674573/job/25202528871) | 37s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135291526/job/25122395715) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135631224/job/25123269757) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9136245872/job/25124775843) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9150240542/job/25154788689) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9166674573/job/25202528871) (+8 more) | | [permission-check](https://github.com/aptos-labs/aptos-core/actions/runs/9166674569/job/25202528998) | 35s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135291523/job/25122395802) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135631225/job/25123269930) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9136245874/job/25124775952) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9150240543/job/25154788879) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9166674569/job/25202528998) (+8 more) | | [determine-docker-build-metadata](https://github.com/aptos-labs/aptos-core/actions/runs/9166674611/job/25202535661) | 31s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135291532/job/25122398780) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9135631232/job/25123273334) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9136245878/job/25124786736) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9150240564/job/25154791415) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9166674611/job/25202535661) (+8 more) | | [determine-forge-run-metadata](https://github.com/aptos-labs/aptos-core/actions/runs/9103420539/job/25025233348) | 4s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9103420539/job/25025233348)  |

🚨 4 jobs on the last run were significantly faster/slower than expected

Job	Duration	vs 7d avg	Delta
rust-build-cached-packages	8m	5m
rust-move-tests	13m	9m
rust-targeted-unit-tests	25m	18m
cli-e2e-tests / run-cli-tests	9m	7m

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

[github-actions[bot]]

Forge is running suite compat on 01b24e7e3548382dd25440b39a0438a993387f12 ==> e02561f0fc915334115c8bf9c04a96c412355402

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on e02561f0fc915334115c8bf9c04a96c412355402

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite compat success on 01b24e7e3548382dd25440b39a0438a993387f12 ==> e02561f0fc915334115c8bf9c04a96c412355402

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> e02561f0fc915334115c8bf9c04a96c412355402 (PR)
1. Check liveness of validators at old version: 01b24e7e3548382dd25440b39a0438a993387f12
compatibility::simple-validator-upgrade::liveness-check : committed: 5333.642424486305 txn/s, latency: 5975.541732427969 ms, (p50: 4900 ms, p90: 10600 ms, p99: 11500 ms), latency samples: 217960
2. Upgrading first Validator to new version: e02561f0fc915334115c8bf9c04a96c412355402
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1628.1005142720644 txn/s, latency: 16169.139363336992 ms, (p50: 19000 ms, p90: 22600 ms, p99: 24400 ms), latency samples: 91100
3. Upgrading rest of first batch to new version: e02561f0fc915334115c8bf9c04a96c412355402
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1543.2671428089602 txn/s, latency: 17715.58919615655 ms, (p50: 19300 ms, p90: 25000 ms, p99: 26000 ms), latency samples: 85340
4. upgrading second batch to new version: e02561f0fc915334115c8bf9c04a96c412355402
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3600.4364598573998 txn/s, latency: 8561.703332870435 ms, (p50: 9900 ms, p90: 11500 ms, p99: 12100 ms), latency samples: 144020
5. check swarm health
Compatibility test for 01b24e7e3548382dd25440b39a0438a993387f12 ==> e02561f0fc915334115c8bf9c04a96c412355402 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:x: Forge suite realistic_env_max_load failure on e02561f0fc915334115c8bf9c04a96c412355402

two traffics test: inner traffic : committed: 7823.940368030366 txn/s, latency: 5007.519550369277 ms, (p50: 4800 ms, p90: 5700 ms, p99: 10200 ms), latency samples: 3385000
two traffics test : committed: 100.08346003197656 txn/s, latency: 1857.6858695652174 ms, (p50: 1800 ms, p90: 2100 ms, p99: 3700 ms), latency samples: 1840
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.205, avg: 0.201", "QsPosToProposal: max: 0.301, avg: 0.263", "ConsensusProposalToOrdered: max: 0.376, avg: 0.348", "ConsensusOrderedToCommit: max: 0.401, avg: 0.390", "ConsensusProposalToCommit: max: 0.750, avg: 0.738"]
Test Failed: check for success

Caused by:
    error!() count in validator logs was 2358, and must be 0

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:83:36
   1: aptos_forge::success_criteria::SuccessCriteriaChecker::check_no_errors::{{closure}}
             at ./testsuite/forge/src/success_criteria.rs:552:13
   2: aptos_forge::success_criteria::SuccessCriteriaChecker::check_for_success::{{closure}}
             at ./testsuite/forge/src/success_criteria.rs:309:42
   3: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:63
   4: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:107:5
   5: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:73:5
   6: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:31
   7: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/blocking.rs:66:9
   8: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:87:13
   9: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/runtime.rs:65:16
  10: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:86:9
  11: tokio::runtime::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/runtime.rs:350:50
  12: aptos_forge::interface::network::NetworkContext::check_for_success
             at ./testsuite/forge/src/interface/network.rs:71:9
  13: <dyn aptos_testcases::NetworkLoadTest as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/lib.rs:229:13
  14: <aptos_testcases::CompositeNetworkTest as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/lib.rs:499:9
  15: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:598:42
  16: aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:666:11
  17: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:598:30
  18: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
  19: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
  20: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
  21: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
  22: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
  23: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
  24: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  25: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  26: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  27: std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
  28: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  29: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  30: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  31: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  32: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  33: __libc_start_main
  34: _start
Trailing Log Lines:
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  30: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  31: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  32: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  33: __libc_start_main
  34: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:292"},"thread_name":"main","hostname":"forge-e2e-pr-13285-1715759206-e02561f0fc915334115c8bf9c04a96c41","timestamp":"2024-05-15T07:59:01.044516Z","message":"Deleting namespace forge-e2e-pr-13285: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:400"},"thread_name":"main","hostname":"forge-e2e-pr-13285-1715759206-e02561f0fc915334115c8bf9c04a96c41","timestamp":"2024-05-15T07:59:01.044552Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-13285"}

failures:
    CompositeNetworkTest

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:83:36
   1: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:618:13
   2: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
   3: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
   4: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
   5: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
   6: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
   7: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
   8: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
   9: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  10: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  11: std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
  12: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  13: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  14: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  15: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  16: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  17: __libc_start_main
  18: _start
Debugging output:
NAME                                   READY   STATUS      RESTARTS   AGE
aptos-node-0-fullnode-eforge38-0       1/1     Running     0          11m
aptos-node-0-validator-0               1/1     Running     0          11m
aptos-node-1-fullnode-eforge38-0       1/1     Running     0          11m
aptos-node-1-validator-0               1/1     Running     0          11m
aptos-node-2-fullnode-eforge38-0       1/1     Running     0          11m
aptos-node-2-validator-0               1/1     Running     0          11m
aptos-node-3-fullnode-eforge38-0       1/1     Running     0          11m
aptos-node-3-validator-0               1/1     Running     0          11m
aptos-node-4-fullnode-eforge38-0       1/1     Running     0          11m
aptos-node-4-validator-0               1/1     Running     0          11m
aptos-node-5-validator-0               1/1     Running     0          11m
aptos-node-6-validator-0               1/1     Running     0          11m
genesis-aptos-genesis-eforge38-qs5hm   0/1     Completed   0          11m

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on b7505b028138dbef2d450570b5f4263a17724e6a

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite compat on 01b24e7e3548382dd25440b39a0438a993387f12 ==> b7505b028138dbef2d450570b5f4263a17724e6a

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite compat success on 01b24e7e3548382dd25440b39a0438a993387f12 ==> b7505b028138dbef2d450570b5f4263a17724e6a

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> b7505b028138dbef2d450570b5f4263a17724e6a (PR)
1. Check liveness of validators at old version: 01b24e7e3548382dd25440b39a0438a993387f12
compatibility::simple-validator-upgrade::liveness-check : committed: 6874.667381205939 txn/s, latency: 4807.688592574216 ms, (p50: 4800 ms, p90: 7800 ms, p99: 8400 ms), latency samples: 241860
2. Upgrading first Validator to new version: b7505b028138dbef2d450570b5f4263a17724e6a
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1302.1752317310206 txn/s, latency: 19671.619668874173 ms, (p50: 18500 ms, p90: 30800 ms, p99: 37000 ms), latency samples: 90600
3. Upgrading rest of first batch to new version: b7505b028138dbef2d450570b5f4263a17724e6a
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1830.7852524389402 txn/s, latency: 15703.745357220887 ms, (p50: 19000 ms, p90: 22100 ms, p99: 22600 ms), latency samples: 91540
4. upgrading second batch to new version: b7505b028138dbef2d450570b5f4263a17724e6a
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3054.4289072976253 txn/s, latency: 9975.298597560975 ms, (p50: 9900 ms, p90: 12400 ms, p99: 13000 ms), latency samples: 131200
5. check swarm health
Compatibility test for 01b24e7e3548382dd25440b39a0438a993387f12 ==> b7505b028138dbef2d450570b5f4263a17724e6a passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite realistic_env_max_load success on b7505b028138dbef2d450570b5f4263a17724e6a

two traffics test: inner traffic : committed: 8017.97996456559 txn/s, latency: 4887.18741051015 ms, (p50: 4500 ms, p90: 5700 ms, p99: 13200 ms), latency samples: 3464080
two traffics test : committed: 99.98813654650876 txn/s, latency: 1949.142857142857 ms, (p50: 1800 ms, p90: 2000 ms, p99: 8200 ms), latency samples: 1820
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.208, avg: 0.202", "QsPosToProposal: max: 0.362, avg: 0.256", "ConsensusProposalToOrdered: max: 0.385, avg: 0.352", "ConsensusOrderedToCommit: max: 0.405, avg: 0.394", "ConsensusProposalToCommit: max: 0.759, avg: 0.746"]
Max round gap was 1 [limit 4] at version 1740095. Max no progress secs was 4.890568 [limit 15] at version 1740095.
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite compat on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 31c5fa05b648dccd084e0b5c306cbb2d6b52bd28

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on 31c5fa05b648dccd084e0b5c306cbb2d6b52bd28

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite compat success on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 31c5fa05b648dccd084e0b5c306cbb2d6b52bd28

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 31c5fa05b648dccd084e0b5c306cbb2d6b52bd28 (PR)
1. Check liveness of validators at old version: 01b24e7e3548382dd25440b39a0438a993387f12
compatibility::simple-validator-upgrade::liveness-check : committed: 5332.082605677927 txn/s, latency: 6098.4758618860715 ms, (p50: 5700 ms, p90: 9200 ms, p99: 14200 ms), latency samples: 188540
2. Upgrading first Validator to new version: 31c5fa05b648dccd084e0b5c306cbb2d6b52bd28
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1807.1403995763844 txn/s, latency: 15772.682780358327 ms, (p50: 19600 ms, p90: 22800 ms, p99: 23200 ms), latency samples: 90420
3. Upgrading rest of first batch to new version: 31c5fa05b648dccd084e0b5c306cbb2d6b52bd28
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1733.3771633398937 txn/s, latency: 16304.702831627097 ms, (p50: 19000 ms, p90: 22400 ms, p99: 22800 ms), latency samples: 91820
4. upgrading second batch to new version: 31c5fa05b648dccd084e0b5c306cbb2d6b52bd28
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 2917.966706836343 txn/s, latency: 10144.674373484237 ms, (p50: 9900 ms, p90: 12000 ms, p99: 13900 ms), latency samples: 123700
5. check swarm health
Compatibility test for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 31c5fa05b648dccd084e0b5c306cbb2d6b52bd28 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite realistic_env_max_load success on 31c5fa05b648dccd084e0b5c306cbb2d6b52bd28

two traffics test: inner traffic : committed: 7647.692652012464 txn/s, latency: 5113.959790195026 ms, (p50: 5100 ms, p90: 6000 ms, p99: 10200 ms), latency samples: 3315460
two traffics test : committed: 99.99558146376593 txn/s, latency: 1849.6294444444445 ms, (p50: 1800 ms, p90: 2100 ms, p99: 5300 ms), latency samples: 1800
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.205, avg: 0.202", "QsPosToProposal: max: 0.287, avg: 0.264", "ConsensusProposalToOrdered: max: 0.379, avg: 0.368", "ConsensusOrderedToCommit: max: 0.387, avg: 0.377", "ConsensusProposalToCommit: max: 0.754, avg: 0.745"]
Max round gap was 1 [limit 4] at version 1675238. Max no progress secs was 4.603827 [limit 15] at version 1675238.
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on de83741b55d0bd69099956dde97871b7456bdcf3

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite compat on 01b24e7e3548382dd25440b39a0438a993387f12 ==> de83741b55d0bd69099956dde97871b7456bdcf3

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite realistic_env_max_load success on de83741b55d0bd69099956dde97871b7456bdcf3

two traffics test: inner traffic : committed: 8306.665591342193 txn/s, latency: 4710.872728487366 ms, (p50: 4500 ms, p90: 5600 ms, p99: 10200 ms), latency samples: 3592540
two traffics test : committed: 100.01173852481348 txn/s, latency: 1876.6277777777777 ms, (p50: 1800 ms, p90: 2100 ms, p99: 5700 ms), latency samples: 1800
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.208, avg: 0.202", "QsPosToProposal: max: 0.273, avg: 0.237", "ConsensusProposalToOrdered: max: 0.317, avg: 0.312", "ConsensusOrderedToCommit: max: 0.399, avg: 0.389", "ConsensusProposalToCommit: max: 0.709, avg: 0.701"]
Max round gap was 1 [limit 4] at version 1652846. Max no progress secs was 4.648314 [limit 15] at version 1652846.
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite compat success on 01b24e7e3548382dd25440b39a0438a993387f12 ==> de83741b55d0bd69099956dde97871b7456bdcf3

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> de83741b55d0bd69099956dde97871b7456bdcf3 (PR)
1. Check liveness of validators at old version: 01b24e7e3548382dd25440b39a0438a993387f12
compatibility::simple-validator-upgrade::liveness-check : committed: 6695.76824884473 txn/s, latency: 4938.291052046834 ms, (p50: 5000 ms, p90: 8000 ms, p99: 9300 ms), latency samples: 234020
2. Upgrading first Validator to new version: de83741b55d0bd69099956dde97871b7456bdcf3
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1487.7879439485625 txn/s, latency: 19777.354359702797 ms, (p50: 18500 ms, p90: 31000 ms, p99: 33200 ms), latency samples: 91520
3. Upgrading rest of first batch to new version: de83741b55d0bd69099956dde97871b7456bdcf3
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1772.2366386491574 txn/s, latency: 16333.451641526175 ms, (p50: 18100 ms, p90: 22000 ms, p99: 22400 ms), latency samples: 90160
4. upgrading second batch to new version: de83741b55d0bd69099956dde97871b7456bdcf3
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3105.6655044495205 txn/s, latency: 9898.630409050045 ms, (p50: 9900 ms, p90: 12600 ms, p99: 13000 ms), latency samples: 133480
5. check swarm health
Compatibility test for 01b24e7e3548382dd25440b39a0438a993387f12 ==> de83741b55d0bd69099956dde97871b7456bdcf3 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite compat on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 6c8dd4db12fcd21824e4a3cfa93eef69331f2ca1

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on 6c8dd4db12fcd21824e4a3cfa93eef69331f2ca1

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:x: Forge suite realistic_env_max_load failure on 6c8dd4db12fcd21824e4a3cfa93eef69331f2ca1

Forge test runner terminated:
Trailing Log Lines:
  29: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  30: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  31: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  32: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  33: __libc_start_main
  34: _start
Debugging output:
NAME                                   READY   STATUS            RESTARTS   AGE
aptos-node-0-fullnode-eforge84-0       1/1     Running           0          104s
aptos-node-0-validator-0               1/1     Running           0          104s
aptos-node-1-fullnode-eforge84-0       1/1     Running           0          104s
aptos-node-1-validator-0               1/1     Running           0          104s
aptos-node-2-fullnode-eforge84-0       0/1     PodInitializing   0          103s
aptos-node-2-validator-0               1/1     Running           0          104s
aptos-node-3-fullnode-eforge84-0       0/1     PodInitializing   0          103s
aptos-node-3-validator-0               1/1     Running           0          104s
aptos-node-4-fullnode-eforge84-0       1/1     Running           0          104s
aptos-node-4-validator-0               1/1     Running           0          104s
aptos-node-5-validator-0               0/1     ErrImagePull      0          104s
aptos-node-6-validator-0               1/1     Running           0          104s
genesis-aptos-genesis-eforge84-frvtk   0/1     Completed         0          2m14s

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite compat success on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 6c8dd4db12fcd21824e4a3cfa93eef69331f2ca1

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 6c8dd4db12fcd21824e4a3cfa93eef69331f2ca1 (PR)
1. Check liveness of validators at old version: 01b24e7e3548382dd25440b39a0438a993387f12
compatibility::simple-validator-upgrade::liveness-check : committed: 6066.384274317974 txn/s, latency: 5386.643764530829 ms, (p50: 4900 ms, p90: 9000 ms, p99: 15400 ms), latency samples: 215060
2. Upgrading first Validator to new version: 6c8dd4db12fcd21824e4a3cfa93eef69331f2ca1
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1403.8963188886428 txn/s, latency: 20043.80210483433 ms, (p50: 22200 ms, p90: 27600 ms, p99: 29600 ms), latency samples: 73640
3. Upgrading rest of first batch to new version: 6c8dd4db12fcd21824e4a3cfa93eef69331f2ca1
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1512.6281529359376 txn/s, latency: 16369.323574730355 ms, (p50: 18900 ms, p90: 22800 ms, p99: 30000 ms), latency samples: 90860
4. upgrading second batch to new version: 6c8dd4db12fcd21824e4a3cfa93eef69331f2ca1
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3523.1432036756146 txn/s, latency: 8830.441711763031 ms, (p50: 9600 ms, p90: 12700 ms, p99: 13000 ms), latency samples: 140440
5. check swarm health
Compatibility test for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 6c8dd4db12fcd21824e4a3cfa93eef69331f2ca1 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on 6c8dd4db12fcd21824e4a3cfa93eef69331f2ca1

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite realistic_env_max_load success on 6c8dd4db12fcd21824e4a3cfa93eef69331f2ca1

two traffics test: inner traffic : committed: 7840.475612172085 txn/s, latency: 4992.205201465849 ms, (p50: 4800 ms, p90: 5700 ms, p99: 10500 ms), latency samples: 3394620
two traffics test : committed: 99.90186543700366 txn/s, latency: 1774.632183908046 ms, (p50: 1800 ms, p90: 2000 ms, p99: 2400 ms), latency samples: 1740
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.203, avg: 0.201", "QsPosToProposal: max: 0.277, avg: 0.254", "ConsensusProposalToOrdered: max: 0.363, avg: 0.355", "ConsensusOrderedToCommit: max: 0.404, avg: 0.392", "ConsensusProposalToCommit: max: 0.756, avg: 0.746"]
Max round gap was 1 [limit 4] at version 1531161. Max no progress secs was 4.387996 [limit 15] at version 1531161.
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on 37ab5dfff26b6a714d213ba80a2e6eaf79c8bdf1

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite compat on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 37ab5dfff26b6a714d213ba80a2e6eaf79c8bdf1

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite compat success on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 37ab5dfff26b6a714d213ba80a2e6eaf79c8bdf1

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 37ab5dfff26b6a714d213ba80a2e6eaf79c8bdf1 (PR)
1. Check liveness of validators at old version: 01b24e7e3548382dd25440b39a0438a993387f12
compatibility::simple-validator-upgrade::liveness-check : committed: 6133.889687708981 txn/s, latency: 5163.312686929108 ms, (p50: 4800 ms, p90: 9300 ms, p99: 10800 ms), latency samples: 247420
2. Upgrading first Validator to new version: 37ab5dfff26b6a714d213ba80a2e6eaf79c8bdf1
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1734.9761230345064 txn/s, latency: 15954.242513134852 ms, (p50: 19100 ms, p90: 22000 ms, p99: 23000 ms), latency samples: 91360
3. Upgrading rest of first batch to new version: 37ab5dfff26b6a714d213ba80a2e6eaf79c8bdf1
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1832.8124967998108 txn/s, latency: 15734.43910019561 ms, (p50: 18800 ms, p90: 22300 ms, p99: 22600 ms), latency samples: 92020
4. upgrading second batch to new version: 37ab5dfff26b6a714d213ba80a2e6eaf79c8bdf1
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3182.1259158233233 txn/s, latency: 9767.613915984895 ms, (p50: 10100 ms, p90: 13900 ms, p99: 15000 ms), latency samples: 127120
5. check swarm health
Compatibility test for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 37ab5dfff26b6a714d213ba80a2e6eaf79c8bdf1 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite realistic_env_max_load success on 37ab5dfff26b6a714d213ba80a2e6eaf79c8bdf1

two traffics test: inner traffic : committed: 8053.472243040777 txn/s, latency: 4863.222877065759 ms, (p50: 4700 ms, p90: 5700 ms, p99: 10700 ms), latency samples: 3480560
two traffics test : committed: 99.95949709598301 txn/s, latency: 1750.7380952380952 ms, (p50: 1800 ms, p90: 2000 ms, p99: 2100 ms), latency samples: 1680
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.215, avg: 0.203", "QsPosToProposal: max: 0.302, avg: 0.253", "ConsensusProposalToOrdered: max: 0.347, avg: 0.327", "ConsensusOrderedToCommit: max: 0.399, avg: 0.389", "ConsensusProposalToCommit: max: 0.725, avg: 0.716"]
Max round gap was 1 [limit 4] at version 1786687. Max no progress secs was 4.76486 [limit 15] at version 1786687.
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite compat on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 86e379a2352630a2bbf4b051b51be270c671c13b

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on 86e379a2352630a2bbf4b051b51be270c671c13b

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:x: Forge suite realistic_env_max_load failure on 86e379a2352630a2bbf4b051b51be270c671c13b

Test Failed: start emitter job

Caused by:
    0: Tried executing 10 txns, request counters: "success 10, failed submit [0], failed wait [10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10], by client: [(10, 0, 40): http://aptos-node-4-fullnode.forge-e2e-pr-13285.svc:8080/v1/]\n[(0, 0, 40): http://aptos-node-0-fullnode.forge-e2e-pr-13285.svc:8080/v1/]\n[(0, 0, 40): http://aptos-node-1-fullnode.forge-e2e-pr-13285.svc:8080/v1/]\n[(0, 0, 40): http://aptos-node-2-fullnode.forge-e2e-pr-13285.svc:8080/v1/]\n[(0, 0, 20): http://aptos-node-3-fullnode.forge-e2e-pr-13285.svc:8080/v1/]"
    1: Unknown error Ledger on endpoint (http://aptos-node-1-fullnode.forge-e2e-pr-13285.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (a2ca953b) might still succeed.

Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:565:25
   1: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/result.rs:1963:27
   2: aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiReliableTransactionSubmitter::submit_check_and_retry::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:127:28
   3: <futures_util::future::maybe_done::MaybeDone<Fut> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/maybe_done.rs:95:38
   4: <futures_util::future::join_all::JoinAll<F> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/join_all.rs:143:24
   5: <aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiReliableTransactionSubmitter as aptos_transaction_generator_lib::ReliableTransactionSubmitter>::execute_transactions_with_counter::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:309:10
   6: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
   7: aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_and_fund_seed_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:434:18
   8: aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_and_fund_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:326:14
   9: aptos_transaction_emitter_lib::emitter::create_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:1195:14
  10: aptos_transaction_emitter_lib::emitter::TxnEmitter::start_job::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:717:10
  11: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:63
  12: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:107:5
  13: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:73:5
  14: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:31
  15: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/blocking.rs:66:9
  16: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:87:13
  17: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/runtime.rs:65:16
  18: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:86:9
  19: tokio::runtime::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/runtime.rs:350:50
  20: <dyn aptos_testcases::NetworkLoadTest>::network_load_test
             at ./testsuite/testcases/src/lib.rs:276:23
  21: <dyn aptos_testcases::NetworkLoadTest as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/lib.rs:180:30
  22: <aptos_testcases::CompositeNetworkTest as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/lib.rs:499:9
  23: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:598:42
  24: aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:666:11
  25: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:598:30
  26: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
  27: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
  28: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
  29: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
  30: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
  31: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
  32: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  33: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  34: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  35: std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
  36: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  37: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  38: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  39: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  40: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  41: __libc_start_main
  42: _start
Trailing Log Lines:
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  38: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  39: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  40: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  41: __libc_start_main
  42: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:292"},"thread_name":"main","hostname":"forge-e2e-pr-13285-1715950712-86e379a2352630a2bbf4b051b51be270c","timestamp":"2024-05-17T13:06:21.336155Z","message":"Deleting namespace forge-e2e-pr-13285: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:400"},"thread_name":"main","hostname":"forge-e2e-pr-13285-1715950712-86e379a2352630a2bbf4b051b51be270c","timestamp":"2024-05-17T13:06:21.336183Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-13285"}

failures:
    CompositeNetworkTest

test result: FAILED. 0 passed; 1 failed; 0 filtered out
Failed to run tests:
Tests Failed

Error: Tests Failed

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:83:36
   1: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:618:13
   2: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
   3: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
   4: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
   5: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
   6: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
   7: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
   8: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
   9: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  10: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  11: std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
  12: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  13: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  14: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  15: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  16: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  17: __libc_start_main
  18: _start
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-fullnode-eforge146-0       1/1     Running     0          6m57s
aptos-node-0-validator-0                1/1     Running     0          6m57s
aptos-node-1-fullnode-eforge146-0       1/1     Running     0          6m57s
aptos-node-1-validator-0                1/1     Running     0          6m57s
aptos-node-2-fullnode-eforge146-0       1/1     Running     0          6m57s
aptos-node-2-validator-0                1/1     Running     0          6m57s
aptos-node-3-fullnode-eforge146-0       1/1     Running     0          6m57s
aptos-node-3-validator-0                1/1     Running     0          6m57s
aptos-node-4-fullnode-eforge146-0       1/1     Running     0          6m56s
aptos-node-4-validator-0                1/1     Running     0          6m56s
aptos-node-5-validator-0                1/1     Running     0          6m57s
aptos-node-6-validator-0                1/1     Running     0          6m57s
genesis-aptos-genesis-eforge146-6w2r6   0/1     Completed   0          7m28s

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite compat success on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 86e379a2352630a2bbf4b051b51be270c671c13b

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 86e379a2352630a2bbf4b051b51be270c671c13b (PR)
1. Check liveness of validators at old version: 01b24e7e3548382dd25440b39a0438a993387f12
compatibility::simple-validator-upgrade::liveness-check : committed: 6223.65433262448 txn/s, latency: 5109.587285591767 ms, (p50: 4800 ms, p90: 8700 ms, p99: 10800 ms), latency samples: 233200
2. Upgrading first Validator to new version: 86e379a2352630a2bbf4b051b51be270c671c13b
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1434.433216479275 txn/s, latency: 20845.66487838145 ms, (p50: 19200 ms, p90: 31100 ms, p99: 31400 ms), latency samples: 87980
3. Upgrading rest of first batch to new version: 86e379a2352630a2bbf4b051b51be270c671c13b
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1406.12112480369 txn/s, latency: 18635.213701630568 ms, (p50: 18900 ms, p90: 27400 ms, p99: 28700 ms), latency samples: 82180
4. upgrading second batch to new version: 86e379a2352630a2bbf4b051b51be270c671c13b
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3558.7525559366063 txn/s, latency: 8685.822693248885 ms, (p50: 9500 ms, p90: 12200 ms, p99: 12700 ms), latency samples: 138940
5. check swarm health
Compatibility test for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 86e379a2352630a2bbf4b051b51be270c671c13b passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on 79237f66b4d7893b38ec0c1402792068dcb72d9e

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite compat on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 79237f66b4d7893b38ec0c1402792068dcb72d9e

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:x: Forge suite realistic_env_max_load failure on 79237f66b4d7893b38ec0c1402792068dcb72d9e

Test Failed: test NetworkLoadTest

Caused by:
    0: Tried executing 10 txns, request counters: "success 0, failed submit [0], failed wait [10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10], by client: [(0, 0, 30): http://aptos-node-0-fullnode.forge-e2e-pr-13285.svc:8080/v1/]\n[(0, 0, 10): http://aptos-node-1-fullnode.forge-e2e-pr-13285.svc:8080/v1/]\n[(0, 0, 40): http://aptos-node-2-fullnode.forge-e2e-pr-13285.svc:8080/v1/]\n[(0, 0, 40): http://aptos-node-3-fullnode.forge-e2e-pr-13285.svc:8080/v1/]\n[(0, 0, 60): http://aptos-node-4-fullnode.forge-e2e-pr-13285.svc:8080/v1/]"
    1: Unknown error Ledger on endpoint (http://aptos-node-0-fullnode.forge-e2e-pr-13285.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (a9345c6d) might still succeed.

Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:565:25
   1: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/result.rs:1963:27
   2: aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiReliableTransactionSubmitter::submit_check_and_retry::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:127:28
   3: <futures_util::future::maybe_done::MaybeDone<Fut> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/maybe_done.rs:95:38
   4: <futures_util::future::join_all::JoinAll<F> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/join_all.rs:143:24
   5: <aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiReliableTransactionSubmitter as aptos_transaction_generator_lib::ReliableTransactionSubmitter>::execute_transactions_with_counter::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:309:10
   6: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
   7: aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_and_fund_seed_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:434:18
   8: aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_and_fund_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:326:14
   9: aptos_transaction_emitter_lib::emitter::create_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:1195:14
  10: aptos_transaction_emitter_lib::emitter::TxnEmitter::start_job::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:717:10
  11: aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for_impl::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:827:14
  12: aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:859:14
  13: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:63
  14: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:107:5
  15: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:73:5
  16: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:31
  17: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/blocking.rs:66:9
  18: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:87:13
  19: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/runtime.rs:65:16
  20: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:86:9
  21: tokio::runtime::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/runtime.rs:350:50
  22: <aptos_testcases::two_traffics_test::TwoTrafficsTest as aptos_testcases::NetworkLoadTest>::test
             at ./testsuite/testcases/src/two_traffics_test.rs:52:21
  23: <dyn aptos_testcases::NetworkLoadTest>::network_load_test
             at ./testsuite/testcases/src/lib.rs:311:13
  24: <dyn aptos_testcases::NetworkLoadTest as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/lib.rs:180:30
  25: <aptos_testcases::CompositeNetworkTest as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/lib.rs:499:9
  26: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:598:42
  27: aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:666:11
  28: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:598:30
  29: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
  30: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
  31: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
  32: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
  33: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
  34: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
  35: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  36: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  37: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  38: std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
  39: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  40: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  41: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  42: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  43: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  44: __libc_start_main
  45: _start
Trailing Log Lines:
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  41: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  42: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  43: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  44: __libc_start_main
  45: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:292"},"thread_name":"main","hostname":"forge-e2e-pr-13285-1715956953-79237f66b4d7893b38ec0c1402792068d","timestamp":"2024-05-17T14:50:45.616415Z","message":"Deleting namespace forge-e2e-pr-13285: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:400"},"thread_name":"main","hostname":"forge-e2e-pr-13285-1715956953-79237f66b4d7893b38ec0c1402792068d","timestamp":"2024-05-17T14:50:45.616439Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-13285"}

failures:
    CompositeNetworkTest

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:83:36
   1: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:618:13
   2: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
   3: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
   4: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
   5: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
   6: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
   7: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
   8: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
   9: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  10: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  11: std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
  12: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  13: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  14: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  15: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  16: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  17: __libc_start_main
  18: _start
Debugging output:
NAME                                   READY   STATUS      RESTARTS   AGE
aptos-node-0-fullnode-eforge23-0       1/1     Running     0          7m21s
aptos-node-0-validator-0               1/1     Running     0          7m21s
aptos-node-1-fullnode-eforge23-0       1/1     Running     0          7m19s
aptos-node-1-validator-0               1/1     Running     0          7m20s
aptos-node-2-fullnode-eforge23-0       1/1     Running     0          7m21s
aptos-node-2-validator-0               1/1     Running     0          7m20s
aptos-node-3-fullnode-eforge23-0       1/1     Running     0          7m20s
aptos-node-3-validator-0               1/1     Running     0          7m20s
aptos-node-4-fullnode-eforge23-0       1/1     Running     0          7m21s
aptos-node-4-validator-0               1/1     Running     0          7m20s
aptos-node-5-validator-0               1/1     Running     0          7m20s
aptos-node-6-validator-0               1/1     Running     0          7m21s
genesis-aptos-genesis-eforge23-jrdjc   0/1     Completed   0          7m47s

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite compat success on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 79237f66b4d7893b38ec0c1402792068dcb72d9e

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 79237f66b4d7893b38ec0c1402792068dcb72d9e (PR)
1. Check liveness of validators at old version: 01b24e7e3548382dd25440b39a0438a993387f12
compatibility::simple-validator-upgrade::liveness-check : committed: 5792.266408158122 txn/s, latency: 5701.712445630684 ms, (p50: 5100 ms, p90: 8300 ms, p99: 14900 ms), latency samples: 202320
2. Upgrading first Validator to new version: 79237f66b4d7893b38ec0c1402792068dcb72d9e
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1718.4253466842868 txn/s, latency: 16780.18652888985 ms, (p50: 18700 ms, p90: 23600 ms, p99: 23900 ms), latency samples: 92420
3. Upgrading rest of first batch to new version: 79237f66b4d7893b38ec0c1402792068dcb72d9e
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1681.5118140355703 txn/s, latency: 17079.61906187625 ms, (p50: 18800 ms, p90: 23300 ms, p99: 23600 ms), latency samples: 90180
4. upgrading second batch to new version: 79237f66b4d7893b38ec0c1402792068dcb72d9e
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 2915.677395207241 txn/s, latency: 10408.951336427657 ms, (p50: 9900 ms, p90: 12500 ms, p99: 13000 ms), latency samples: 124960
5. check swarm health
Compatibility test for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 79237f66b4d7893b38ec0c1402792068dcb72d9e passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite compat on 01b24e7e3548382dd25440b39a0438a993387f12 ==> b99e75a2375508e1f9e52110b1d962bb74ad9c28

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on b99e75a2375508e1f9e52110b1d962bb74ad9c28

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite realistic_env_max_load success on b99e75a2375508e1f9e52110b1d962bb74ad9c28

two traffics test: inner traffic : committed: 7866.961696140545 txn/s, latency: 4986.33614704722 ms, (p50: 4800 ms, p90: 5700 ms, p99: 10500 ms), latency samples: 3396460
two traffics test : committed: 99.95045978521036 txn/s, latency: 1847.364835164835 ms, (p50: 1800 ms, p90: 2100 ms, p99: 3200 ms), latency samples: 1820
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.206, avg: 0.202", "QsPosToProposal: max: 0.262, avg: 0.252", "ConsensusProposalToOrdered: max: 0.356, avg: 0.322", "ConsensusOrderedToCommit: max: 0.414, avg: 0.403", "ConsensusProposalToCommit: max: 0.738, avg: 0.725"]
Max round gap was 1 [limit 4] at version 1666506. Max no progress secs was 4.59404 [limit 15] at version 1666506.
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite compat success on 01b24e7e3548382dd25440b39a0438a993387f12 ==> b99e75a2375508e1f9e52110b1d962bb74ad9c28

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> b99e75a2375508e1f9e52110b1d962bb74ad9c28 (PR)
1. Check liveness of validators at old version: 01b24e7e3548382dd25440b39a0438a993387f12
compatibility::simple-validator-upgrade::liveness-check : committed: 5833.388639495023 txn/s, latency: 5274.068661569558 ms, (p50: 4900 ms, p90: 9200 ms, p99: 14800 ms), latency samples: 227580
2. Upgrading first Validator to new version: b99e75a2375508e1f9e52110b1d962bb74ad9c28
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1265.1194565140206 txn/s, latency: 20578.63368121442 ms, (p50: 25800 ms, p90: 29000 ms, p99: 29700 ms), latency samples: 73780
3. Upgrading rest of first batch to new version: b99e75a2375508e1f9e52110b1d962bb74ad9c28
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1742.7472959093595 txn/s, latency: 16587.782800710163 ms, (p50: 18300 ms, p90: 24400 ms, p99: 24700 ms), latency samples: 90120
4. upgrading second batch to new version: b99e75a2375508e1f9e52110b1d962bb74ad9c28
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 2941.5687819140794 txn/s, latency: 10246.890352959797 ms, (p50: 9900 ms, p90: 12300 ms, p99: 12700 ms), latency samples: 126360
5. check swarm health
Compatibility test for 01b24e7e3548382dd25440b39a0438a993387f12 ==> b99e75a2375508e1f9e52110b1d962bb74ad9c28 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite compat on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 0722006bdb3b8a85ae5829e2a40c2f273a55c8d4

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on 0722006bdb3b8a85ae5829e2a40c2f273a55c8d4

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite compat success on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 0722006bdb3b8a85ae5829e2a40c2f273a55c8d4

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 0722006bdb3b8a85ae5829e2a40c2f273a55c8d4 (PR)
1. Check liveness of validators at old version: 01b24e7e3548382dd25440b39a0438a993387f12
compatibility::simple-validator-upgrade::liveness-check : committed: 5631.314849650034 txn/s, latency: 5300.097025282886 ms, (p50: 5100 ms, p90: 6200 ms, p99: 12800 ms), latency samples: 226240
2. Upgrading first Validator to new version: 0722006bdb3b8a85ae5829e2a40c2f273a55c8d4
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1201.5990512786723 txn/s, latency: 22466.215101214573 ms, (p50: 26900 ms, p90: 30900 ms, p99: 31400 ms), latency samples: 74100
3. Upgrading rest of first batch to new version: 0722006bdb3b8a85ae5829e2a40c2f273a55c8d4
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1808.7738369748058 txn/s, latency: 16003.405538727824 ms, (p50: 18700 ms, p90: 22500 ms, p99: 22700 ms), latency samples: 92440
4. upgrading second batch to new version: 0722006bdb3b8a85ae5829e2a40c2f273a55c8d4
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3600.474446836987 txn/s, latency: 8723.147932042892 ms, (p50: 9800 ms, p90: 12400 ms, p99: 12700 ms), latency samples: 143620
5. check swarm health
Compatibility test for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 0722006bdb3b8a85ae5829e2a40c2f273a55c8d4 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite compat on 01b24e7e3548382dd25440b39a0438a993387f12 ==> f5fe7ec34940aa470c02d7d9d909116830a18589

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on f5fe7ec34940aa470c02d7d9d909116830a18589

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite compat success on 01b24e7e3548382dd25440b39a0438a993387f12 ==> f5fe7ec34940aa470c02d7d9d909116830a18589

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> f5fe7ec34940aa470c02d7d9d909116830a18589 (PR)
1. Check liveness of validators at old version: 01b24e7e3548382dd25440b39a0438a993387f12
compatibility::simple-validator-upgrade::liveness-check : committed: 6472.188577900718 txn/s, latency: 4986.750419744795 ms, (p50: 4800 ms, p90: 8100 ms, p99: 9700 ms), latency samples: 238240
2. Upgrading first Validator to new version: f5fe7ec34940aa470c02d7d9d909116830a18589
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1490.3700076981268 txn/s, latency: 18803.718335603997 ms, (p50: 18100 ms, p90: 29500 ms, p99: 30500 ms), latency samples: 88080
3. Upgrading rest of first batch to new version: f5fe7ec34940aa470c02d7d9d909116830a18589
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1719.8939170413014 txn/s, latency: 16928.40299688057 ms, (p50: 18100 ms, p90: 22000 ms, p99: 22600 ms), latency samples: 89760
4. upgrading second batch to new version: f5fe7ec34940aa470c02d7d9d909116830a18589
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3517.4677150374227 txn/s, latency: 8886.170365265334 ms, (p50: 9600 ms, p90: 12600 ms, p99: 12900 ms), latency samples: 145100
5. check swarm health
Compatibility test for 01b24e7e3548382dd25440b39a0438a993387f12 ==> f5fe7ec34940aa470c02d7d9d909116830a18589 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite realistic_env_max_load success on f5fe7ec34940aa470c02d7d9d909116830a18589

two traffics test: inner traffic : committed: 8461.12621466548 txn/s, latency: 4637.300904021953 ms, (p50: 4500 ms, p90: 5400 ms, p99: 10200 ms), latency samples: 3651460
two traffics test : committed: 99.88045641700829 txn/s, latency: 1925.7823863636363 ms, (p50: 1800 ms, p90: 2100 ms, p99: 6700 ms), latency samples: 1760
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.207, avg: 0.202", "QsPosToProposal: max: 0.253, avg: 0.228", "ConsensusProposalToOrdered: max: 0.344, avg: 0.327", "ConsensusOrderedToCommit: max: 0.411, avg: 0.401", "ConsensusProposalToCommit: max: 0.738, avg: 0.728"]
Max round gap was 1 [limit 4] at version 1789174. Max no progress secs was 4.731448 [limit 15] at version 1789174.
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on efd4b418ad451ea33595e7090d51201d0570aa75

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite compat on 3ffe0986b5fe4acb76544ae7ae85d73b91a6a411 ==> efd4b418ad451ea33595e7090d51201d0570aa75

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite compat success on 3ffe0986b5fe4acb76544ae7ae85d73b91a6a411 ==> efd4b418ad451ea33595e7090d51201d0570aa75

Compatibility test results for 3ffe0986b5fe4acb76544ae7ae85d73b91a6a411 ==> efd4b418ad451ea33595e7090d51201d0570aa75 (PR)
1. Check liveness of validators at old version: 3ffe0986b5fe4acb76544ae7ae85d73b91a6a411
compatibility::simple-validator-upgrade::liveness-check : committed: 6727.791405033736 txn/s, latency: 4934.742757208008 ms, (p50: 4800 ms, p90: 8100 ms, p99: 8800 ms), latency samples: 258740
2. Upgrading first Validator to new version: efd4b418ad451ea33595e7090d51201d0570aa75
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1851.2673726627631 txn/s, latency: 15381.982457278824 ms, (p50: 18800 ms, p90: 21700 ms, p99: 22200 ms), latency samples: 92460
3. Upgrading rest of first batch to new version: efd4b418ad451ea33595e7090d51201d0570aa75
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1847.356539767029 txn/s, latency: 15541.240762748488 ms, (p50: 18200 ms, p90: 22000 ms, p99: 22300 ms), latency samples: 92560
4. upgrading second batch to new version: efd4b418ad451ea33595e7090d51201d0570aa75
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3562.348454723503 txn/s, latency: 8773.528274672188 ms, (p50: 9600 ms, p90: 12700 ms, p99: 12900 ms), latency samples: 144900
5. check swarm health
Compatibility test for 3ffe0986b5fe4acb76544ae7ae85d73b91a6a411 ==> efd4b418ad451ea33595e7090d51201d0570aa75 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite realistic_env_max_load success on efd4b418ad451ea33595e7090d51201d0570aa75

two traffics test: inner traffic : committed: 8861.36945415039 txn/s, latency: 4426.021377858348 ms, (p50: 4200 ms, p90: 4800 ms, p99: 9900 ms), latency samples: 3828260
two traffics test : committed: 99.89258184589475 txn/s, latency: 1825.5230769230768 ms, (p50: 1700 ms, p90: 2000 ms, p99: 6400 ms), latency samples: 1820
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.207, avg: 0.203", "QsPosToProposal: max: 0.238, avg: 0.208", "ConsensusProposalToOrdered: max: 0.312, avg: 0.305", "ConsensusOrderedToCommit: max: 0.404, avg: 0.394", "ConsensusProposalToCommit: max: 0.709, avg: 0.699"]
Max round gap was 1 [limit 4] at version 1960598. Max no progress secs was 5.011463 [limit 15] at version 1960598.
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking