search

aptos-labs / aptos-core

Aptos is a layer 1 blockchain built to support the widespread use of blockchain through better technology and user experience.
https://aptosfoundation.org
Other
5.85k stars 3.54k forks source link

[framework] A few minor fix for dispatchable token #13181

Closed runtian-zhou closed 4 days ago

runtian-zhou commented 2 weeks ago

Description

Fixes a bunch of issues:

  1. Updated move-unit-test to include full error msg on test failure so we can see the detailed error msg.
  2. Fix error msg when the module that contains the hook function doesn't exist.
  3. Fix error msg when invoking FS api on assets with hook registered.
  4. Fix PFS mint to invoke withdraw_internal.
  5. Fix transfer api in dispatchable api to check for the frozen bit.

Type of Change

Which Components or Systems Does This Change Impact?

How Has This Been Tested?

Added tests for each affected cases.

Key Areas to Review

N/A

Checklist

trunk-io[bot] commented 2 weeks ago
⏱️ 12h 18m total CI duration on this PR | Job | Cumulative Duration | Recent Runs | |---|---|---| | [rust-targeted-unit-tests](https://github.com/aptos-labs/aptos-core/actions/runs/9023408271/job/24795146380) | 3h 6m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8931758567/job/24534416228) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8943088721/job/24567067995) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8976396585/job/24653180235) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8989430724/job/24692531420) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9023408271/job/24795146380) (+3 more) | | [rust-move-unit-coverage](https://github.com/aptos-labs/aptos-core/actions/runs/9023408282/job/24795136692) | 2h 40m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8931758556/job/24534408635) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8943088726/job/24567055537) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8976396578/job/24653170980) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8989430734/job/24692512391) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9023408282/job/24795136692) (+3 more) | | [rust-move-tests](https://github.com/aptos-labs/aptos-core/actions/runs/9023408273/job/24795136378) | 1h 33m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8931758566/job/24534408673) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8943088725/job/24567055749) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/8976396581/job/24653170655) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8989430728/job/24692512081) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9023408273/job/24795136378) (+3 more) | | [forge-framework-upgrade-test / forge](https://github.com/aptos-labs/aptos-core/actions/runs/9037728599/job/24840679651) | 1h 10m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9037728599/job/24840679651)  | | [execution-performance / single-node-performance](https://github.com/aptos-labs/aptos-core/actions/runs/9083366112/job/24963463013) | 54m | [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9068845135/job/24918328995) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9083366112/job/24963463013)  | | [rust-lints](https://github.com/aptos-labs/aptos-core/actions/runs/9023408271/job/24795146040) | 53m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8931758567/job/24534415659) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8943088721/job/24567067780) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8976396585/job/24653179824) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8989430724/job/24692530712) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9023408271/job/24795146040) (+3 more) | | [run-tests-main-branch](https://github.com/aptos-labs/aptos-core/actions/runs/9037725288/job/24837361760) | 38m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8943088525/job/24567072512) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8976396336/job/24653186345) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8989430454/job/24692536564) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9023408188/job/24795151062) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9037725288/job/24837361760) (+4 more) | | [rust-smoke-tests](https://github.com/aptos-labs/aptos-core/actions/runs/9068577268/job/24916192971) | 24m | [⬜](https://github.com/aptos-labs/aptos-core/actions/runs/9068577268/job/24916192971)  | | [general-lints](https://github.com/aptos-labs/aptos-core/actions/runs/9023408271/job/24795146201) | 15m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8931758567/job/24534415530) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8943088721/job/24567067571) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8976396585/job/24653180030) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8989430724/job/24692530498) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9023408271/job/24795146201) (+3 more) | | [check-dynamic-deps](https://github.com/aptos-labs/aptos-core/actions/runs/9023408280/job/24795135613) | 11m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8931758565/job/24534408277) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8943088715/job/24567054625) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8976396575/job/24653169790) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8989430740/job/24692511275) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9023408280/job/24795135613) (+3 more) | | [rust-images / rust-all](https://github.com/aptos-labs/aptos-core/actions/runs/9036599386/job/24833810592) | 9m | [⬜](https://github.com/aptos-labs/aptos-core/actions/runs/9036599386/job/24833810592)  | | [cli-e2e-tests / run-cli-tests](https://github.com/aptos-labs/aptos-core/actions/runs/9069581209/job/24930403170) | 7m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9069581209/job/24930403170)  | | [execution-performance / test-target-determinator](https://github.com/aptos-labs/aptos-core/actions/runs/9069581180/job/24930394572) | 6m | [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9036599292/job/24833939728) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9069581180/job/24930394572)  | | [semgrep/ci](https://github.com/aptos-labs/aptos-core/actions/runs/9037725309/job/24837347141) | 4m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8943088730/job/24567054877) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8976396565/job/24653169773) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8989430731/job/24692511065) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9023408283/job/24795135793) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9037725309/job/24837347141) (+4 more) | | [test-target-determinator](https://github.com/aptos-labs/aptos-core/actions/runs/9069581209/job/24930401599) | 3m | [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9069581209/job/24930401599)  | | [file_change_determinator](https://github.com/aptos-labs/aptos-core/actions/runs/9023408188/job/24795142506) | 2m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8931758415/job/24534412090) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8943088525/job/24567064099) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8976396336/job/24653176577) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8989430454/job/24692520850) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9023408188/job/24795142506) (+3 more) | | [file_change_determinator](https://github.com/aptos-labs/aptos-core/actions/runs/9023408271/job/24795136507) | 2m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8931758567/job/24534408748) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8943088721/job/24567055414) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8976396585/job/24653170732) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8989430724/job/24692512313) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9023408271/job/24795136507) (+3 more) | | [permission-check](https://github.com/aptos-labs/aptos-core/actions/runs/9023408188/job/24795135614) | 29s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8931758415/job/24534408153) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8943088525/job/24567054159) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8976396336/job/24653169426) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8989430454/job/24692510261) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9023408188/job/24795135614) (+3 more) | | [permission-check](https://github.com/aptos-labs/aptos-core/actions/runs/9037728563/job/24837358406) | 27s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8976396341/job/24653169459) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8989430459/job/24692510371) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9023408185/job/24795135708) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9036683032/job/24834067946) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9037728563/job/24837358406) (+5 more) | | [permission-check](https://github.com/aptos-labs/aptos-core/actions/runs/9023408186/job/24795135332) | 26s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8931758413/job/24534407921) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8943088527/job/24567053911) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8976396335/job/24653169137) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8989430471/job/24692510051) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9023408186/job/24795135332) (+3 more) | | [check](https://github.com/aptos-labs/aptos-core/actions/runs/9037725320/job/24837347656) | 22s | [⬜](https://github.com/aptos-labs/aptos-core/actions/runs/9037725320/job/24837347656)  | | [permission-check](https://github.com/aptos-labs/aptos-core/actions/runs/9023408187/job/24795135338) | 22s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8931758408/job/24534407927) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8943088526/job/24567053883) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8976396323/job/24653169176) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/8989430457/job/24692510045) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9023408187/job/24795135338) (+3 more) |

🚨 2 jobs on the last run were significantly faster/slower than expected

Job Duration vs 7d avg Delta
rust-targeted-unit-tests 24m 18m +35%
run-tests-main-branch 10s 4m -96%

settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io

codecov[bot] commented 2 weeks ago

Codecov Report

Attention: Patch coverage is 96.00000% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 57.6%. Comparing base (7719765) to head (bd69098). Report is 3 commits behind head on main.

Files Patch % Lines
...rd_party/move/move-compiler-v2/src/plan_builder.rs 0.0% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #13181 +/- ## ======================================= Coverage 57.6% 57.6% ======================================= Files 834 834 Lines 198396 198409 +13 ======================================= + Hits 114325 114339 +14 + Misses 84071 84070 -1 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

github-actions[bot] commented 1 week ago

Forge is running suite framework_upgrade on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 09b8f54862b600693ed771d77de0eeefc8ba8b28

github-actions[bot] commented 1 week ago

Forge is running suite compat on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 09b8f54862b600693ed771d77de0eeefc8ba8b28

github-actions[bot] commented 1 week ago

Forge is running suite realistic_env_max_load on 09b8f54862b600693ed771d77de0eeefc8ba8b28

github-actions[bot] commented 1 week ago

:white_check_mark: Forge suite compat success on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 09b8f54862b600693ed771d77de0eeefc8ba8b28

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 09b8f54862b600693ed771d77de0eeefc8ba8b28 (PR)
1. Check liveness of validators at old version: 01b24e7e3548382dd25440b39a0438a993387f12
compatibility::simple-validator-upgrade::liveness-check : committed: 6658.525954242049 txn/s, latency: 4964.7840657928555 ms, (p50: 4800 ms, p90: 8400 ms, p99: 9300 ms), latency samples: 233460
2. Upgrading first Validator to new version: 09b8f54862b600693ed771d77de0eeefc8ba8b28
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1703.6619440821903 txn/s, latency: 17001.26070873139 ms, (p50: 19200 ms, p90: 22500 ms, p99: 22700 ms), latency samples: 90020
3. Upgrading rest of first batch to new version: 09b8f54862b600693ed771d77de0eeefc8ba8b28
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1816.5512334383936 txn/s, latency: 15772.217867277886 ms, (p50: 18700 ms, p90: 22200 ms, p99: 22500 ms), latency samples: 91620
4. upgrading second batch to new version: 09b8f54862b600693ed771d77de0eeefc8ba8b28
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3425.4720263640847 txn/s, latency: 8673.488599862732 ms, (p50: 9900 ms, p90: 11400 ms, p99: 12600 ms), latency samples: 145700
5. check swarm health
Compatibility test for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 09b8f54862b600693ed771d77de0eeefc8ba8b28 passed
Test Ok
github-actions[bot] commented 1 week ago

:white_check_mark: Forge suite realistic_env_max_load success on 09b8f54862b600693ed771d77de0eeefc8ba8b28

two traffics test: inner traffic : committed: 7977.34994663365 txn/s, latency: 4918.2868462624065 ms, (p50: 4800 ms, p90: 5700 ms, p99: 11700 ms), latency samples: 3443660
two traffics test : committed: 99.97896382171866 txn/s, latency: 2051.002247191011 ms, (p50: 1900 ms, p90: 2100 ms, p99: 9700 ms), latency samples: 1780
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.207, avg: 0.201", "QsPosToProposal: max: 0.304, avg: 0.236", "ConsensusProposalToOrdered: max: 0.435, avg: 0.422", "ConsensusOrderedToCommit: max: 0.377, avg: 0.366", "ConsensusProposalToCommit: max: 0.801, avg: 0.788"]
Max round gap was 1 [limit 4] at version 1692504. Max no progress secs was 4.696348 [limit 15] at version 1692504.
Test Ok
github-actions[bot] commented 1 week ago

:white_check_mark: Forge suite framework_upgrade success on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 09b8f54862b600693ed771d77de0eeefc8ba8b28

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 09b8f54862b600693ed771d77de0eeefc8ba8b28 (PR)
Upgrade the nodes to version: 09b8f54862b600693ed771d77de0eeefc8ba8b28
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1297.10573553904 txn/s, submitted: 1299.5734219954443 txn/s, failed submission: 2.4676864564042615 txn/s, expired: 2.4676864564042615 txn/s, latency: 2338.5310878588725 ms, (p50: 1800 ms, p90: 3900 ms, p99: 5800 ms), latency samples: 115640
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1193.2801269476684 txn/s, submitted: 1195.8752399165146 txn/s, failed submission: 2.5951129688462538 txn/s, expired: 2.5951129688462538 txn/s, latency: 2896.2573645709767 ms, (p50: 2400 ms, p90: 5000 ms, p99: 8400 ms), latency samples: 101160
5. check swarm health
Compatibility test for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 09b8f54862b600693ed771d77de0eeefc8ba8b28 passed
Upgrade the remaining nodes to version: 09b8f54862b600693ed771d77de0eeefc8ba8b28
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1231.7735989082778 txn/s, submitted: 1233.8222868066698 txn/s, failed submission: 2.048687898392146 txn/s, expired: 2.048687898392146 txn/s, latency: 2809.683212058212 ms, (p50: 2400 ms, p90: 5100 ms, p99: 6600 ms), latency samples: 96200
Test Ok
github-actions[bot] commented 1 week ago

Forge is running suite framework_upgrade on 01b24e7e3548382dd25440b39a0438a993387f12 ==> ddcf5129ee9816559a7894188406bcb20f08abb7

github-actions[bot] commented 1 week ago

Forge is running suite realistic_env_max_load on ddcf5129ee9816559a7894188406bcb20f08abb7

github-actions[bot] commented 1 week ago

Forge is running suite compat on 01b24e7e3548382dd25440b39a0438a993387f12 ==> ddcf5129ee9816559a7894188406bcb20f08abb7

github-actions[bot] commented 1 week ago

:white_check_mark: Forge suite compat success on 01b24e7e3548382dd25440b39a0438a993387f12 ==> ddcf5129ee9816559a7894188406bcb20f08abb7

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> ddcf5129ee9816559a7894188406bcb20f08abb7 (PR)
1. Check liveness of validators at old version: 01b24e7e3548382dd25440b39a0438a993387f12
compatibility::simple-validator-upgrade::liveness-check : committed: 6030.4756609008045 txn/s, latency: 5226.7996875 ms, (p50: 4800 ms, p90: 7800 ms, p99: 11400 ms), latency samples: 233600
2. Upgrading first Validator to new version: ddcf5129ee9816559a7894188406bcb20f08abb7
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1852.9501610681132 txn/s, latency: 15540.39741117851 ms, (p50: 19700 ms, p90: 21700 ms, p99: 22300 ms), latency samples: 92320
3. Upgrading rest of first batch to new version: ddcf5129ee9816559a7894188406bcb20f08abb7
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1751.9590690204298 txn/s, latency: 15756.468864666524 ms, (p50: 18900 ms, p90: 22000 ms, p99: 22900 ms), latency samples: 92660
4. upgrading second batch to new version: ddcf5129ee9816559a7894188406bcb20f08abb7
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 2814.6215951021227 txn/s, latency: 11029.500224698904 ms, (p50: 11700 ms, p90: 13500 ms, p99: 15600 ms), latency samples: 111260
5. check swarm health
Compatibility test for 01b24e7e3548382dd25440b39a0438a993387f12 ==> ddcf5129ee9816559a7894188406bcb20f08abb7 passed
Test Ok
github-actions[bot] commented 1 week ago

:white_check_mark: Forge suite realistic_env_max_load success on ddcf5129ee9816559a7894188406bcb20f08abb7

two traffics test: inner traffic : committed: 8558.919898080505 txn/s, latency: 4573.529554478712 ms, (p50: 4400 ms, p90: 5400 ms, p99: 9900 ms), latency samples: 3702180
two traffics test : committed: 99.99251959156015 txn/s, latency: 1814.3777777777777 ms, (p50: 1800 ms, p90: 2000 ms, p99: 2100 ms), latency samples: 1800
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.208, avg: 0.202", "QsPosToProposal: max: 0.223, avg: 0.201", "ConsensusProposalToOrdered: max: 0.473, avg: 0.413", "ConsensusOrderedToCommit: max: 0.357, avg: 0.340", "ConsensusProposalToCommit: max: 0.769, avg: 0.753"]
Max round gap was 1 [limit 4] at version 1692772. Max no progress secs was 5.139476 [limit 15] at version 1692772.
Test Ok
github-actions[bot] commented 1 week ago

:x: Forge suite framework_upgrade failure on 01b24e7e3548382dd25440b39a0438a993387f12 ==> ddcf5129ee9816559a7894188406bcb20f08abb7

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> ddcf5129ee9816559a7894188406bcb20f08abb7 (PR)
Upgrade the nodes to version: ddcf5129ee9816559a7894188406bcb20f08abb7
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1353.6601334632398 txn/s, submitted: 1356.584777261648 txn/s, failed submission: 2.924643798408196 txn/s, expired: 2.924643798408196 txn/s, latency: 2226.3558833305633 ms, (p50: 1800 ms, p90: 3900 ms, p99: 5100 ms), latency samples: 120340
Test Failed: API error: Unknown error Transaction expired. It is guaranteed it will not be committed on chain.

Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:565:25
   1: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/result.rs:1963:27
   2: aptos_release_builder::validate::NetworkConfig::execute_proposal::{{closure}}
             at ./aptos-move/aptos-release-builder/src/validate.rs:364:9
   3: aptos_release_builder::validate::NetworkConfig::submit_and_execute_multi_step_proposal::{{closure}}
             at ./aptos-move/aptos-release-builder/src/validate.rs:124:64
   4: aptos_release_builder::validate::execute_release::{{closure}}
             at ./aptos-move/aptos-release-builder/src/validate.rs:432:22
   5: aptos_release_builder::validate::validate_config_and_generate_release::{{closure}}
             at ./aptos-move/aptos-release-builder/src/validate.rs:493:6
   6: aptos_release_builder::validate::validate_config::{{closure}}
             at ./aptos-move/aptos-release-builder/src/validate.rs:479:80
   7: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:63
   8: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:107:5
   9: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:73:5
  10: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:31
  11: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/blocking.rs:66:9
  12: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:87:13
  13: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/runtime.rs:65:16
  14: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:86:9
  15: tokio::runtime::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/runtime.rs:350:50
  16: <aptos_testcases::framework_upgrade::FrameworkUpgrade as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/framework_upgrade.rs:112:9
  17: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:598:42
  18: aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:666:11
  19: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:598:30
  20: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
  21: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
  22: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
  23: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
  24: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
  25: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
  26: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  27: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  28: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  29: std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
  30: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  31: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  32: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  33: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  34: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  35: __libc_start_main
  36: _start
Trailing Log Lines:
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  32: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  33: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  34: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  35: __libc_start_main
  36: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:292"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-13181-1715373823-01b24e7e3548382dd25","timestamp":"2024-05-10T21:40:11.834234Z","message":"Deleting namespace forge-framework-upgrade-pr-13181: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:400"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-13181-1715373823-01b24e7e3548382dd25","timestamp":"2024-05-10T21:40:11.834258Z","message":"aptos-node resources for Forge removed in namespace: forge-framework-upgrade-pr-13181"}

failures:
    framework_upgrade::framework-upgrade

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:83:36
   1: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:618:13
   2: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
   3: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
   4: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
   5: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
   6: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
   7: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
   8: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
   9: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  10: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  11: std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
  12: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  13: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  14: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  15: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  16: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  17: __libc_start_main
  18: _start
Debugging output:
NAME                                   READY   STATUS      RESTARTS   AGE
aptos-node-0-validator-0               1/1     Running     0          52m
aptos-node-1-validator-0               1/1     Running     0          52m
aptos-node-2-validator-0               1/1     Running     0          55m
aptos-node-3-validator-0               1/1     Running     0          55m
genesis-aptos-genesis-eforge31-9t898   0/1     Completed   0          56m
github-actions[bot] commented 1 week ago

Forge is running suite framework_upgrade on 01b24e7e3548382dd25440b39a0438a993387f12 ==> ddcf5129ee9816559a7894188406bcb20f08abb7

github-actions[bot] commented 1 week ago

:white_check_mark: Forge suite framework_upgrade success on 01b24e7e3548382dd25440b39a0438a993387f12 ==> ddcf5129ee9816559a7894188406bcb20f08abb7

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> ddcf5129ee9816559a7894188406bcb20f08abb7 (PR)
Upgrade the nodes to version: ddcf5129ee9816559a7894188406bcb20f08abb7
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1206.8148005245157 txn/s, submitted: 1209.4793410224538 txn/s, failed submission: 2.664540497938213 txn/s, expired: 2.664540497938213 txn/s, latency: 2504.802014719411 ms, (p50: 2100 ms, p90: 4200 ms, p99: 5700 ms), latency samples: 108700
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1257.2334611497122 txn/s, submitted: 1260.1661787156454 txn/s, failed submission: 2.932717565933296 txn/s, expired: 2.932717565933296 txn/s, latency: 2438.4805849632153 ms, (p50: 2100 ms, p90: 3900 ms, p99: 5400 ms), latency samples: 111460
5. check swarm health
Compatibility test for 01b24e7e3548382dd25440b39a0438a993387f12 ==> ddcf5129ee9816559a7894188406bcb20f08abb7 passed
Upgrade the remaining nodes to version: ddcf5129ee9816559a7894188406bcb20f08abb7
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1534.6220553927847 txn/s, submitted: 1535.7249099061823 txn/s, failed submission: 1.1028545133976175 txn/s, expired: 1.1028545133976175 txn/s, latency: 2579.243065037729 ms, (p50: 2100 ms, p90: 4200 ms, p99: 6300 ms), latency samples: 111320
Test Ok
github-actions[bot] commented 5 days ago

Forge is running suite realistic_env_max_load on 017aa7097c4a8c08a2df10d0d0d24cf2864667ba

github-actions[bot] commented 5 days ago

Forge is running suite compat on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 017aa7097c4a8c08a2df10d0d0d24cf2864667ba

github-actions[bot] commented 5 days ago

Forge is running suite framework_upgrade on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 2aa5b32bf645188113ec998f37be54daec55dd52

github-actions[bot] commented 5 days ago

Forge is running suite realistic_env_max_load on 2aa5b32bf645188113ec998f37be54daec55dd52

github-actions[bot] commented 5 days ago

Forge is running suite compat on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 2aa5b32bf645188113ec998f37be54daec55dd52

github-actions[bot] commented 5 days ago

:white_check_mark: Forge suite compat success on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 2aa5b32bf645188113ec998f37be54daec55dd52

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 2aa5b32bf645188113ec998f37be54daec55dd52 (PR)
1. Check liveness of validators at old version: 01b24e7e3548382dd25440b39a0438a993387f12
compatibility::simple-validator-upgrade::liveness-check : committed: 6102.40026798433 txn/s, latency: 5108.983204024214 ms, (p50: 4800 ms, p90: 9300 ms, p99: 12000 ms), latency samples: 234580
2. Upgrading first Validator to new version: 2aa5b32bf645188113ec998f37be54daec55dd52
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1817.5622885234543 txn/s, latency: 15774.982185562445 ms, (p50: 19300 ms, p90: 22700 ms, p99: 23100 ms), latency samples: 90320
3. Upgrading rest of first batch to new version: 2aa5b32bf645188113ec998f37be54daec55dd52
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1669.882813654887 txn/s, latency: 17238.549013741136 ms, (p50: 19100 ms, p90: 23800 ms, p99: 24100 ms), latency samples: 90240
4. upgrading second batch to new version: 2aa5b32bf645188113ec998f37be54daec55dd52
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3390.2604357322703 txn/s, latency: 8932.044130554032 ms, (p50: 9800 ms, p90: 12400 ms, p99: 13000 ms), latency samples: 145840
5. check swarm health
Compatibility test for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 2aa5b32bf645188113ec998f37be54daec55dd52 passed
Test Ok
github-actions[bot] commented 5 days ago

:white_check_mark: Forge suite realistic_env_max_load success on 2aa5b32bf645188113ec998f37be54daec55dd52

two traffics test: inner traffic : committed: 7382.752822240214 txn/s, latency: 5300.629081313624 ms, (p50: 5100 ms, p90: 6300 ms, p99: 10800 ms), latency samples: 3198480
two traffics test : committed: 100.0683185429065 txn/s, latency: 1970.5393258426966 ms, (p50: 1900 ms, p90: 2200 ms, p99: 2700 ms), latency samples: 1780
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.211, avg: 0.203", "QsPosToProposal: max: 0.281, avg: 0.268", "ConsensusProposalToOrdered: max: 0.479, avg: 0.439", "ConsensusOrderedToCommit: max: 0.396, avg: 0.371", "ConsensusProposalToCommit: max: 0.843, avg: 0.810"]
Max round gap was 1 [limit 4] at version 1616250. Max no progress secs was 6.111604 [limit 15] at version 1616250.
Test Ok
github-actions[bot] commented 5 days ago

Forge is running suite compat on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 65c3f4cb68e8680f27067f944ca6abf165a2a1bf

github-actions[bot] commented 5 days ago

Forge is running suite realistic_env_max_load on 65c3f4cb68e8680f27067f944ca6abf165a2a1bf

github-actions[bot] commented 5 days ago

:white_check_mark: Forge suite compat success on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 65c3f4cb68e8680f27067f944ca6abf165a2a1bf

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 65c3f4cb68e8680f27067f944ca6abf165a2a1bf (PR)
1. Check liveness of validators at old version: 01b24e7e3548382dd25440b39a0438a993387f12
compatibility::simple-validator-upgrade::liveness-check : committed: 4793 txn/s, latency: 5749 ms, (p50: 4900 ms, p90: 10200 ms, p99: 12500 ms), latency samples: 215720
2. Upgrading first Validator to new version: 65c3f4cb68e8680f27067f944ca6abf165a2a1bf
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1781 txn/s, latency: 16044 ms, (p50: 19000 ms, p90: 21900 ms, p99: 22500 ms), latency samples: 92640
3. Upgrading rest of first batch to new version: 65c3f4cb68e8680f27067f944ca6abf165a2a1bf
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1693 txn/s, latency: 16471 ms, (p50: 19000 ms, p90: 21700 ms, p99: 22800 ms), latency samples: 88040
4. upgrading second batch to new version: 65c3f4cb68e8680f27067f944ca6abf165a2a1bf
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3456 txn/s, latency: 8704 ms, (p50: 9800 ms, p90: 11700 ms, p99: 12300 ms), latency samples: 145160
5. check swarm health
Compatibility test for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 65c3f4cb68e8680f27067f944ca6abf165a2a1bf passed
Test Ok
github-actions[bot] commented 5 days ago

:white_check_mark: Forge suite realistic_env_max_load success on 65c3f4cb68e8680f27067f944ca6abf165a2a1bf

two traffics test: inner traffic : committed: 7856 txn/s, latency: 5001 ms, (p50: 4800 ms, p90: 5900 ms, p99: 10400 ms), latency samples: 3386240
two traffics test : committed: 100 txn/s, latency: 1852 ms, (p50: 1800 ms, p90: 2100 ms, p99: 2500 ms), latency samples: 1800
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.204, avg: 0.201", "QsPosToProposal: max: 0.244, avg: 0.231", "ConsensusProposalToOrdered: max: 0.447, avg: 0.425", "ConsensusOrderedToCommit: max: 0.387, avg: 0.373", "ConsensusProposalToCommit: max: 0.813, avg: 0.798"]
Max round gap was 1 [limit 4] at version 1618744. Max no progress secs was 4.577882 [limit 15] at version 1618744.
Test Ok
github-actions[bot] commented 4 days ago

Forge is running suite realistic_env_max_load on 44ed82233a6efae349a369cd81d6f538765193e1

github-actions[bot] commented 4 days ago

Forge is running suite framework_upgrade on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 44ed82233a6efae349a369cd81d6f538765193e1

github-actions[bot] commented 4 days ago

Forge is running suite compat on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 44ed82233a6efae349a369cd81d6f538765193e1

github-actions[bot] commented 4 days ago

Forge is running suite compat on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 2abd7d226f1e0919c14a5ddc2995781965e2dc8e

github-actions[bot] commented 4 days ago

Forge is running suite realistic_env_max_load on 2abd7d226f1e0919c14a5ddc2995781965e2dc8e

github-actions[bot] commented 4 days ago

Forge is running suite framework_upgrade on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 2abd7d226f1e0919c14a5ddc2995781965e2dc8e

github-actions[bot] commented 4 days ago

:white_check_mark: Forge suite compat success on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 2abd7d226f1e0919c14a5ddc2995781965e2dc8e

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 2abd7d226f1e0919c14a5ddc2995781965e2dc8e (PR)
1. Check liveness of validators at old version: 01b24e7e3548382dd25440b39a0438a993387f12
compatibility::simple-validator-upgrade::liveness-check : committed: 6132.139941907226 txn/s, latency: 5123.150721543848 ms, (p50: 4800 ms, p90: 9600 ms, p99: 12300 ms), latency samples: 234220
2. Upgrading first Validator to new version: 2abd7d226f1e0919c14a5ddc2995781965e2dc8e
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1677.106504668515 txn/s, latency: 17104.516745019027 ms, (p50: 18800 ms, p90: 23200 ms, p99: 23500 ms), latency samples: 89340
3. Upgrading rest of first batch to new version: 2abd7d226f1e0919c14a5ddc2995781965e2dc8e
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1822.125131331941 txn/s, latency: 15888.724096515905 ms, (p50: 18400 ms, p90: 22400 ms, p99: 22800 ms), latency samples: 92420
4. upgrading second batch to new version: 2abd7d226f1e0919c14a5ddc2995781965e2dc8e
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3509.9048126725775 txn/s, latency: 8960.577362394813 ms, (p50: 9600 ms, p90: 12800 ms, p99: 13000 ms), latency samples: 144980
5. check swarm health
Compatibility test for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 2abd7d226f1e0919c14a5ddc2995781965e2dc8e passed
Test Ok
github-actions[bot] commented 4 days ago

:white_check_mark: Forge suite realistic_env_max_load success on 2abd7d226f1e0919c14a5ddc2995781965e2dc8e

two traffics test: inner traffic : committed: 7710.607903044687 txn/s, latency: 5079.991468512178 ms, (p50: 5100 ms, p90: 5700 ms, p99: 10200 ms), latency samples: 3334940
two traffics test : committed: 99.99976228454348 txn/s, latency: 1891.529120879121 ms, (p50: 1900 ms, p90: 2100 ms, p99: 2300 ms), latency samples: 1820
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.205, avg: 0.201", "QsPosToProposal: max: 0.254, avg: 0.232", "ConsensusProposalToOrdered: max: 0.467, avg: 0.431", "ConsensusOrderedToCommit: max: 0.411, avg: 0.396", "ConsensusProposalToCommit: max: 0.846, avg: 0.826"]
Max round gap was 1 [limit 4] at version 1656419. Max no progress secs was 4.670751 [limit 15] at version 1656419.
Test Ok
github-actions[bot] commented 4 days ago

:x: Forge suite framework_upgrade failure on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 2abd7d226f1e0919c14a5ddc2995781965e2dc8e

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 2abd7d226f1e0919c14a5ddc2995781965e2dc8e (PR)
Upgrade the nodes to version: 2abd7d226f1e0919c14a5ddc2995781965e2dc8e
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1320.7122569012245 txn/s, submitted: 1323.2231167052191 txn/s, failed submission: 2.510859803994723 txn/s, expired: 2.510859803994723 txn/s, latency: 2430.5519529899757 ms, (p50: 2100 ms, p90: 4200 ms, p99: 5800 ms), latency samples: 115720
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1254.8464959156954 txn/s, submitted: 1257.4707827666866 txn/s, failed submission: 2.624286850991346 txn/s, expired: 2.624286850991346 txn/s, latency: 2417.54820494946 ms, (p50: 2100 ms, p90: 4100 ms, p99: 5400 ms), latency samples: 114760
5. check swarm health
Compatibility test for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 2abd7d226f1e0919c14a5ddc2995781965e2dc8e passed
Upgrade the remaining nodes to version: 2abd7d226f1e0919c14a5ddc2995781965e2dc8e
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 217.96944512303028 txn/s, submitted: 218.5076412838279 txn/s, failed submission: 0.5381961607976056 txn/s, expired: 0.5381961607976056 txn/s, latency: 2573.6038081671413 ms, (p50: 2100 ms, p90: 4400 ms, p99: 6600 ms), latency samples: 105300
Test Failed: Waiting for nodes to catch up to target version and epoch (None, Some(1)) timed out after 20 seconds, current status: Err(Unknown error error sending request for url (http://aptos-node-2-validator.forge-framework-upgrade-pr-13181.svc:8080/v1/): error trying to connect: dns error: failed to lookup address information: Name or service not known

Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:565:25
   1: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/result.rs:1963:27
   2: aptos_forge::interface::swarm::wait_for_all_nodes_to_catchup_to_target_version_or_epoch::{{closure}}::{{closure}}::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:419:49
   3: <F as futures_core::future::TryFuture>::try_poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/future.rs:82:9
   4: <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/try_future/into_future.rs:34:9
   5: <F as futures_core::future::TryFuture>::try_poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/future.rs:82:9
   6: <futures_util::future::try_maybe_done::TryMaybeDone<Fut> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/try_maybe_done.rs:79:57
   7: <F as futures_core::future::TryFuture>::try_poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/future.rs:82:9
   8: <futures_util::future::try_join_all::TryJoinAll<F> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/try_join_all.rs:163:32
   9: aptos_forge::interface::swarm::wait_for_all_nodes_to_catchup_to_target_version_or_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:426:14
  10: aptos_forge::interface::swarm::wait_for_all_nodes_to_catchup_to_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:400:6
  11: aptos_forge::interface::swarm::SwarmExt::wait_for_all_nodes_to_change_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:306:92
  12: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
  13: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:63
  14: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:107:5
  15: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:73:5
  16: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:31
  17: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/blocking.rs:66:9
  18: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:87:13
  19: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/runtime.rs:65:16
  20: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:86:9
  21: tokio::runtime::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/runtime.rs:350:50
  22: aptos_forge::interface::swarm::SwarmExt::fork_check
             at ./testsuite/forge/src/interface/swarm.rs:210:9
  23: <aptos_testcases::framework_upgrade::FrameworkUpgrade as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/framework_upgrade.rs:168:9
  24: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:598:42
  25: aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:666:11
  26: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:598:30
  27: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
  28: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
  29: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
  30: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
  31: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
  32: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
  33: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  34: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  35: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  36: std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
  37: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  38: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  39: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  40: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  41: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  42: __libc_start_main
  43: _start)

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:83:36
   1: aptos_forge::interface::swarm::wait_for_all_nodes_to_catchup_to_target_version_or_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:467:24
   2: aptos_forge::interface::swarm::wait_for_all_nodes_to_catchup_to_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:400:6
   3: aptos_forge::interface::swarm::SwarmExt::wait_for_all_nodes_to_change_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:306:92
   4: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
   5: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:63
   6: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:107:5
   7: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:73:5
   8: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:31
   9: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/blocking.rs:66:9
  10: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:87:13
  11: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/runtime.rs:65:16
  12: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:86:9
  13: tokio::runtime::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/runtime.rs:350:50
  14: aptos_forge::interface::swarm::SwarmExt::fork_check
             at ./testsuite/forge/src/interface/swarm.rs:210:9
  15: <aptos_testcases::framework_upgrade::FrameworkUpgrade as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/framework_upgrade.rs:168:9
  16: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:598:42
  17: aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:666:11
  18: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:598:30
  19: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
  20: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
  21: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
  22: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
  23: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
  24: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
  25: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  26: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  27: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  28: std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
  29: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  30: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  31: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  32: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  33: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  34: __libc_start_main
  35: _start
Trailing Log Lines:
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  31: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  32: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  33: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  34: __libc_start_main
  35: _start

Swarm logs can be found here: See fgi output for more information.
thread 'main' panicked at testsuite/forge/src/backend/k8s/swarm.rs:696:18:
called `Result::unwrap()` on an `Err` value: ApiError: namespaces "forge-framework-upgrade-pr-13181" not found: NotFound (ErrorResponse { status: "Failure", message: "namespaces \"forge-framework-upgrade-pr-13181\" not found", reason: "NotFound", code: 404 })

Caused by:
    namespaces "forge-framework-upgrade-pr-13181" not found: NotFound

Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:565:25
   1: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/result.rs:1963:27
   2: aptos_forge::backend::k8s::cluster_helper::delete_k8s_cluster::{{closure}}
             at ./testsuite/forge/src/backend/k8s/cluster_helper.rs:289:13
   3: aptos_forge::backend::k8s::cluster_helper::uninstall_testnet_resources::{{closure}}
             at ./testsuite/forge/src/backend/k8s/cluster_helper.rs:399:48
   4: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:63
   5: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:107:5
   6: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:73:5
   7: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:31
   8: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/blocking.rs:66:9
   9: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:87:13
  10: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/runtime.rs:65:16
  11: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:86:9
  12: tokio::runtime::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/runtime.rs:350:50
  13: <aptos_forge::backend::k8s::swarm::K8sSwarm as core::ops::drop::Drop>::drop
             at ./testsuite/forge/src/backend/k8s/swarm.rs:694:13
  14: core::ptr::drop_in_place<aptos_forge::backend::k8s::swarm::K8sSwarm>
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ptr/mod.rs:498:1
  15: core::ptr::drop_in_place<alloc::boxed::Box<dyn aptos_forge::interface::swarm::Swarm>>
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ptr/mod.rs:498:1
  16: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:611:9
  17: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
  18: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
  19: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
  20: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
  21: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
  22: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
  23: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  24: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  25: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  26: std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
  27: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  28: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  29: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  30: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  31: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  32: __libc_start_main
  33: _start
stack backtrace:
   0: rust_begin_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/panicking.rs:72:14
   2: core::result::unwrap_failed
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/result.rs:1653:5
   3: core::result::Result<T,E>::unwrap
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/result.rs:1077:23
   4: <aptos_forge::backend::k8s::swarm::K8sSwarm as core::ops::drop::Drop>::drop
             at ./testsuite/forge/src/backend/k8s/swarm.rs:694:13
   5: core::ptr::drop_in_place<aptos_forge::backend::k8s::swarm::K8sSwarm>
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ptr/mod.rs:498:1
   6: core::ptr::drop_in_place<alloc::boxed::Box<dyn aptos_forge::interface::swarm::Swarm>>
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ptr/mod.rs:498:1
   7: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:611:9
   8: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
   9: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
  10: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Debugging output:
github-actions[bot] commented 4 days ago

Forge is running suite compat on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 857b09f302634b5083489513079bf51d3ec9b9ae

github-actions[bot] commented 4 days ago

Forge is running suite framework_upgrade on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 857b09f302634b5083489513079bf51d3ec9b9ae

github-actions[bot] commented 4 days ago

Forge is running suite realistic_env_max_load on 857b09f302634b5083489513079bf51d3ec9b9ae

github-actions[bot] commented 4 days ago

:white_check_mark: Forge suite compat success on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 857b09f302634b5083489513079bf51d3ec9b9ae

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 857b09f302634b5083489513079bf51d3ec9b9ae (PR)
1. Check liveness of validators at old version: 01b24e7e3548382dd25440b39a0438a993387f12
compatibility::simple-validator-upgrade::liveness-check : committed: 6803.865701248323 txn/s, latency: 4847.538653174022 ms, (p50: 4800 ms, p90: 8100 ms, p99: 8700 ms), latency samples: 246060
2. Upgrading first Validator to new version: 857b09f302634b5083489513079bf51d3ec9b9ae
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1802.3345778749476 txn/s, latency: 16080.912100154423 ms, (p50: 18600 ms, p90: 23000 ms, p99: 23300 ms), latency samples: 90660
3. Upgrading rest of first batch to new version: 857b09f302634b5083489513079bf51d3ec9b9ae
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1719.525218431725 txn/s, latency: 16093.4430989011 ms, (p50: 19600 ms, p90: 22500 ms, p99: 22900 ms), latency samples: 91000
4. upgrading second batch to new version: 857b09f302634b5083489513079bf51d3ec9b9ae
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 2775.6245703839068 txn/s, latency: 10730.13551607445 ms, (p50: 9900 ms, p90: 13200 ms, p99: 14500 ms), latency samples: 118200
5. check swarm health
Compatibility test for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 857b09f302634b5083489513079bf51d3ec9b9ae passed
Test Ok
github-actions[bot] commented 4 days ago

:white_check_mark: Forge suite realistic_env_max_load success on 857b09f302634b5083489513079bf51d3ec9b9ae

two traffics test: inner traffic : committed: 7475.40038139032 txn/s, latency: 5224.04868766809 ms, (p50: 5100 ms, p90: 6000 ms, p99: 10500 ms), latency samples: 3242320
two traffics test : committed: 99.96078931524985 txn/s, latency: 1842.3444444444444 ms, (p50: 1800 ms, p90: 2100 ms, p99: 2400 ms), latency samples: 1800
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.209, avg: 0.201", "QsPosToProposal: max: 0.278, avg: 0.252", "ConsensusProposalToOrdered: max: 0.457, avg: 0.433", "ConsensusOrderedToCommit: max: 0.393, avg: 0.375", "ConsensusProposalToCommit: max: 0.826, avg: 0.808"]
Max round gap was 1 [limit 4] at version 1493077. Max no progress secs was 4.727255 [limit 15] at version 1493077.
Test Ok
github-actions[bot] commented 4 days ago

Forge is running suite framework_upgrade on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 4ae5307be10e4be223ee084034422062126f6c25

github-actions[bot] commented 4 days ago

Forge is running suite realistic_env_max_load on 4ae5307be10e4be223ee084034422062126f6c25

github-actions[bot] commented 4 days ago

Forge is running suite compat on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 4ae5307be10e4be223ee084034422062126f6c25

github-actions[bot] commented 4 days ago

Forge is running suite framework_upgrade on 01b24e7e3548382dd25440b39a0438a993387f12 ==> bd69098db3c0196c41925b6ff4d557b2f74e167f

github-actions[bot] commented 4 days ago

Forge is running suite compat on 01b24e7e3548382dd25440b39a0438a993387f12 ==> bd69098db3c0196c41925b6ff4d557b2f74e167f

github-actions[bot] commented 4 days ago

Forge is running suite realistic_env_max_load on bd69098db3c0196c41925b6ff4d557b2f74e167f

github-actions[bot] commented 4 days ago

:white_check_mark: Forge suite compat success on 01b24e7e3548382dd25440b39a0438a993387f12 ==> bd69098db3c0196c41925b6ff4d557b2f74e167f

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> bd69098db3c0196c41925b6ff4d557b2f74e167f (PR)
1. Check liveness of validators at old version: 01b24e7e3548382dd25440b39a0438a993387f12
compatibility::simple-validator-upgrade::liveness-check : committed: 6502.17215268427 txn/s, latency: 5000.47191020754 ms, (p50: 4800 ms, p90: 8700 ms, p99: 10300 ms), latency samples: 236100
2. Upgrading first Validator to new version: bd69098db3c0196c41925b6ff4d557b2f74e167f
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1833.0118954595491 txn/s, latency: 15583.23072368421 ms, (p50: 19400 ms, p90: 22300 ms, p99: 22800 ms), latency samples: 91200
3. Upgrading rest of first batch to new version: bd69098db3c0196c41925b6ff4d557b2f74e167f
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1833.2161816307948 txn/s, latency: 15662.82356818673 ms, (p50: 18600 ms, p90: 22200 ms, p99: 22600 ms), latency samples: 92540
4. upgrading second batch to new version: bd69098db3c0196c41925b6ff4d557b2f74e167f
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3222.6315557107355 txn/s, latency: 9637.292558463683 ms, (p50: 9900 ms, p90: 12700 ms, p99: 14100 ms), latency samples: 129140
5. check swarm health
Compatibility test for 01b24e7e3548382dd25440b39a0438a993387f12 ==> bd69098db3c0196c41925b6ff4d557b2f74e167f passed
Test Ok
github-actions[bot] commented 4 days ago

:white_check_mark: Forge suite realistic_env_max_load success on bd69098db3c0196c41925b6ff4d557b2f74e167f

two traffics test: inner traffic : committed: 8458.44724367445 txn/s, latency: 4638.225689709977 ms, (p50: 4500 ms, p90: 5400 ms, p99: 9900 ms), latency samples: 3649360
two traffics test : committed: 99.907951124701 txn/s, latency: 1940.608888888889 ms, (p50: 1800 ms, p90: 2100 ms, p99: 6000 ms), latency samples: 1800
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.204, avg: 0.201", "QsPosToProposal: max: 0.234, avg: 0.211", "ConsensusProposalToOrdered: max: 0.427, avg: 0.401", "ConsensusOrderedToCommit: max: 0.366, avg: 0.354", "ConsensusProposalToCommit: max: 0.764, avg: 0.755"]
Max round gap was 1 [limit 4] at version 1641645. Max no progress secs was 4.843901 [limit 15] at version 1641645.
Test Ok
github-actions[bot] commented 4 days ago

:white_check_mark: Forge suite framework_upgrade success on 01b24e7e3548382dd25440b39a0438a993387f12 ==> bd69098db3c0196c41925b6ff4d557b2f74e167f

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> bd69098db3c0196c41925b6ff4d557b2f74e167f (PR)
Upgrade the nodes to version: bd69098db3c0196c41925b6ff4d557b2f74e167f
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1215.446499268706 txn/s, submitted: 1217.6673041275926 txn/s, failed submission: 2.2208048588867273 txn/s, expired: 2.2208048588867273 txn/s, latency: 2458.463968572995 ms, (p50: 2100 ms, p90: 4200 ms, p99: 5700 ms), latency samples: 109460
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1209.8918473392112 txn/s, submitted: 1211.9245198473413 txn/s, failed submission: 2.0326725081300916 txn/s, expired: 2.0326725081300916 txn/s, latency: 2632.3316688445025 ms, (p50: 2100 ms, p90: 4200 ms, p99: 7900 ms), latency samples: 107140
5. check swarm health
Compatibility test for 01b24e7e3548382dd25440b39a0438a993387f12 ==> bd69098db3c0196c41925b6ff4d557b2f74e167f passed
Upgrade the remaining nodes to version: bd69098db3c0196c41925b6ff4d557b2f74e167f
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1300.141861002158 txn/s, submitted: 1302.3438325394218 txn/s, failed submission: 2.2019715372637227 txn/s, expired: 2.2019715372637227 txn/s, latency: 2529.077446368084 ms, (p50: 2100 ms, p90: 4200 ms, p99: 6000 ms), latency samples: 106280
Test Ok