search

aptos-labs / aptos-core

Aptos is a layer 1 blockchain built to support the widespread use of blockchain through better technology and user experience.
https://aptosfoundation.org
Other
5.86k stars 3.54k forks source link

[Draft] Add fetch block manager to sync blocks in a separate thread #13262

Closed vusirikala closed 2 days ago

vusirikala commented 2 weeks ago

Description

Add fetch block manager to sync blocks in a separate thread

Type of Change

Which Components or Systems Does This Change Impact?

How Has This Been Tested?

Key Areas to Review

Checklist

trunk-io[bot] commented 2 weeks ago
⏱️ 21h 55m total CI duration on this PR | Job | Cumulative Duration | Recent Runs | |---|---|---| | [indexer-grpc-e2e-tests / test-indexer-grpc-docker-compose](https://github.com/aptos-labs/aptos-core/actions/runs/9128632522/job/25102024768) | 6h 8m | [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9058581189/job/24885103485) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9067565295/job/24913541922) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9069547446/job/24919804624) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9071026081/job/24924291755) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9128632522/job/25102024768)  | | [forge-e2e-test / forge](https://github.com/aptos-labs/aptos-core/actions/runs/9128632522/job/25102024497) | 4h 58m | [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9058581189/job/24885104106) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9067565295/job/24913542687) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9069547446/job/24919804415) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9071026081/job/24924292320) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9128632522/job/25102024497)  | | [rust-targeted-unit-tests](https://github.com/aptos-labs/aptos-core/actions/runs/9128632717/job/25101388174) | 1h 46m | [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9058581080/job/24884490157) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9067565299/job/24912991542) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9069547538/job/24919348502) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9071026142/job/24923960762) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9128632717/job/25101388174) (+1 more) | | [rust-smoke-tests](https://github.com/aptos-labs/aptos-core/actions/runs/9128632717/job/25101388747) | 1h 44m | [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9058581080/job/24884489790) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9067565299/job/24912990830) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9069547538/job/24919347784) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9071026142/job/24923960598) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9128632717/job/25101388747)  | | [forge-compat-test / forge](https://github.com/aptos-labs/aptos-core/actions/runs/9128632522/job/25102024259) | 1h 22m | [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9058581189/job/24885103885) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9067565295/job/24913542435) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9069547446/job/24919804206) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9071026081/job/24924292465) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9128632522/job/25102024259)  | | [rust-images / rust-all](https://github.com/aptos-labs/aptos-core/actions/runs/9128632522/job/25101389706) | 1h 8m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9058581189/job/24884492881) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9067565295/job/24912995173) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9069547446/job/24919350172) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9071026081/job/24923961603) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9128632522/job/25101389706)  | | [rust-move-tests](https://github.com/aptos-labs/aptos-core/actions/runs/9128632687/job/25101374742) | 1h 5m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9058581082/job/24884479186) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9067565300/job/24912979431) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9069547551/job/24919337912) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9071026135/job/24923948293) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9128632687/job/25101374742) (+2 more) | | [cli-e2e-tests / run-cli-tests](https://github.com/aptos-labs/aptos-core/actions/runs/9128632522/job/25102023592) | 46m | [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9058581189/job/24885103041) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9067565295/job/24913541429) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9069547446/job/24919804014) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9071026081/job/24924291911) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9128632522/job/25102023592)  | | [rust-lints](https://github.com/aptos-labs/aptos-core/actions/runs/9128632717/job/25101387559) | 45m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9058581080/job/24884489331) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9067565299/job/24912990569) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9069547538/job/24919347271) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9071026142/job/24923959995) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9128632717/job/25101387559) (+1 more) | | [rust-build-cached-packages](https://github.com/aptos-labs/aptos-core/actions/runs/9128632717/job/25101388486) | 25m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9058581080/job/24884489983) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9067565299/job/24912991749) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9069547538/job/24919348044) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9071026142/job/24923960181) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9128632717/job/25101388486)  | | [run-tests-main-branch](https://github.com/aptos-labs/aptos-core/actions/runs/9128632484/job/25101392350) | 23m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9058581124/job/24884496505) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9067565212/job/24912999240) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9069547411/job/24919352065) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9071026055/job/24923960794) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9128632484/job/25101392350)  | | [check](https://github.com/aptos-labs/aptos-core/actions/runs/9128632684/job/25101374687) | 21m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9058581057/job/24884477768) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9067565304/job/24912979248) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9069547547/job/24919338111) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9071026143/job/24923948286) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9128632684/job/25101374687)  | | [test-target-determinator](https://github.com/aptos-labs/aptos-core/actions/runs/9128632522/job/25101381459) | 16m | [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9058581189/job/24884486578) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9067565295/job/24912987655) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9069547446/job/24919343609) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9071026081/job/24923953581) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9128632522/job/25101381459) (+1 more) | | [execution-performance / test-target-determinator](https://github.com/aptos-labs/aptos-core/actions/runs/9128632727/job/25101374771) | 16m | [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9058581109/job/24884478023) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9067565354/job/24912979451) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9069547567/job/24919337729) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9071026157/job/24923948126) [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/9128632727/job/25101374771)  | | [check-dynamic-deps](https://github.com/aptos-labs/aptos-core/actions/runs/9128632692/job/25101374097) | 11m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9058581075/job/24884477345) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9067565308/job/24912978402) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9069547545/job/24919337262) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9071026144/job/24923947783) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9128632692/job/25101374097) (+2 more) | | [general-lints](https://github.com/aptos-labs/aptos-core/actions/runs/9128632717/job/25101387884) | 9m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9058581080/job/24884489609) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9067565299/job/24912991288) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9069547538/job/24919347541) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9071026142/job/24923959760) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9128632717/job/25101387884) (+1 more) | | [node-api-compatibility-tests / node-api-compatibility-tests](https://github.com/aptos-labs/aptos-core/actions/runs/9128632522/job/25102025068) | 4m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9058581189/job/24885103661) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9067565295/job/24913541679) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9069547446/job/24919803796) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9071026081/job/24924292052) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9128632522/job/25102025068)  | | [semgrep/ci](https://github.com/aptos-labs/aptos-core/actions/runs/9128632675/job/25101374043) | 3m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9058581061/job/24884477196) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9067565301/job/24912978285) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9069547546/job/24919337322) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9071026147/job/24923947835) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9128632675/job/25101374043) (+2 more) | | [file_change_determinator](https://github.com/aptos-labs/aptos-core/actions/runs/9128632717/job/25101375026) | 1m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9058581080/job/24884480924) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9067565299/job/24912979207) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9069547538/job/24919337953) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9071026142/job/24923948153) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9128632717/job/25101375026) (+2 more) | | [file_change_determinator](https://github.com/aptos-labs/aptos-core/actions/runs/9128632484/job/25101379457) | 1m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9058581124/job/24884487495) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9067565212/job/24912987702) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9069547411/job/24919342246) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9071026055/job/24923952654) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9128632484/job/25101379457) (+2 more) | | [file_change_determinator](https://github.com/aptos-labs/aptos-core/actions/runs/9128632522/job/25101381893) | 55s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9058581189/job/24884486374) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9067565295/job/24912987148) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9069547446/job/24919343171) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9071026081/job/24923954064) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9128632522/job/25101381893) (+1 more) | | [permission-check](https://github.com/aptos-labs/aptos-core/actions/runs/9128632510/job/25101373419) | 22s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9058581126/job/24884477553) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9067565215/job/24912978023) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9069547417/job/24919336617) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9071026056/job/24923947429) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9128632510/job/25101373419) (+2 more) | | [determine-docker-build-metadata](https://github.com/aptos-labs/aptos-core/actions/runs/9128632522/job/25101381669) | 20s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9058581189/job/24884486798) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9067565295/job/24912987423) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9069547446/job/24919343395) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9071026081/job/24923953869) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9128632522/job/25101381669) (+1 more) | | [permission-check](https://github.com/aptos-labs/aptos-core/actions/runs/9128632484/job/25101373600) | 17s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9058581124/job/24884482313) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9067565212/job/24912978568) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9069547411/job/24919336957) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9071026055/job/24923947633) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9128632484/job/25101373600) (+2 more) | | [permission-check](https://github.com/aptos-labs/aptos-core/actions/runs/9128632489/job/25101373402) | 17s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9058581120/job/24884477537) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9067565206/job/24912977993) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9069547412/job/24919336610) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9071026061/job/24923947448) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9128632489/job/25101373402) (+2 more) | | [permission-check](https://github.com/aptos-labs/aptos-core/actions/runs/9128632488/job/25101373595) | 16s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9058581131/job/24884478391) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9067565209/job/24912978515) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9069547409/job/24919337068) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9071026054/job/24923947625) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9128632488/job/25101373595) (+2 more) | | [permission-check](https://github.com/aptos-labs/aptos-core/actions/runs/9128632522/job/25101374253) | 13s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9058581189/job/24884481619) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9067565295/job/24912979781) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9069547446/job/24919337672) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9071026081/job/24923948125) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/9128632522/job/25101374253) (+1 more) |

🚨 4 jobs on the last run were significantly faster/slower than expected

Job Duration vs 7d avg Delta
check-dynamic-deps 4m 1m +200%
rust-build-cached-packages 8m 5m +74%
cli-e2e-tests / run-cli-tests 10m 7m +48%
forge-compat-test / forge 18m 14m +30%

settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io

github-actions[bot] commented 2 weeks ago

Forge is running suite compat on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 1f7b0dc77082bcaffb28024671fffc07bb853188

github-actions[bot] commented 2 weeks ago

Forge is running suite realistic_env_max_load on 1f7b0dc77082bcaffb28024671fffc07bb853188

github-actions[bot] commented 2 weeks ago

:x: Forge suite compat failure on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 1f7b0dc77082bcaffb28024671fffc07bb853188

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 1f7b0dc77082bcaffb28024671fffc07bb853188 (PR)
1. Check liveness of validators at old version: 01b24e7e3548382dd25440b39a0438a993387f12
compatibility::simple-validator-upgrade::liveness-check : committed: 6640 txn/s, latency: 5044 ms, (p50: 4800 ms, p90: 8900 ms, p99: 10100 ms), latency samples: 232400
2. Upgrading first Validator to new version: 1f7b0dc77082bcaffb28024671fffc07bb853188
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 137 txn/s, submitted: 396 txn/s, failed submission: 83 txn/s, expired: 258 txn/s, latency: 61626 ms, (p50: 66900 ms, p90: 67100 ms, p99: 67200 ms), latency samples: 13848
3. Upgrading rest of first batch to new version: 1f7b0dc77082bcaffb28024671fffc07bb853188
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 40 txn/s, submitted: 150 txn/s, expired: 109 txn/s, latency: 4848 ms, (p50: 2700 ms, p90: 6400 ms, p99: 20700 ms), latency samples: 13571
Test Failed: Waiting for nodes to catch up to target version and epoch (None, Some(20)) timed out after 61 seconds, current status: Err(Unknown error error sending request for url (http://aptos-node-1-validator.forge-compat-pr-13262.svc:8080/v1/): error trying to connect: tcp connect error: Connection refused (os error 111)

Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:565:25
   1: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/result.rs:1963:27
   2: aptos_forge::interface::swarm::wait_for_all_nodes_to_catchup_to_target_version_or_epoch::{{closure}}::{{closure}}::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:419:49
   3: <F as futures_core::future::TryFuture>::try_poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/future.rs:82:9
   4: <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/try_future/into_future.rs:34:9
   5: <F as futures_core::future::TryFuture>::try_poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/future.rs:82:9
   6: <futures_util::future::try_maybe_done::TryMaybeDone<Fut> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/try_maybe_done.rs:79:57
   7: <F as futures_core::future::TryFuture>::try_poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/future.rs:82:9
   8: <futures_util::future::try_join_all::TryJoinAll<F> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/try_join_all.rs:163:32
   9: aptos_forge::interface::swarm::wait_for_all_nodes_to_catchup_to_target_version_or_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:426:14
  10: aptos_forge::interface::swarm::wait_for_all_nodes_to_catchup_to_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:400:6
  11: aptos_forge::interface::swarm::SwarmExt::wait_for_all_nodes_to_change_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:306:92
  12: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
  13: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:63
  14: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:107:5
  15: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:73:5
  16: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:31
  17: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/blocking.rs:66:9
  18: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:87:13
  19: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/runtime.rs:65:16
  20: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:86:9
  21: tokio::runtime::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/runtime.rs:350:50
  22: aptos_forge::interface::swarm::SwarmExt::fork_check
             at ./testsuite/forge/src/interface/swarm.rs:210:9
  23: <aptos_testcases::compatibility_test::SimpleValidatorUpgrade as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/compatibility_test.rs:105:9
  24: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:598:42
  25: aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:666:11
  26: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:598:30
  27: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
  28: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
  29: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
  30: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
  31: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
  32: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
  33: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  34: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  35: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  36: std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
  37: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  38: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  39: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  40: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  41: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  42: __libc_start_main
  43: _start)

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:83:36
   1: aptos_forge::interface::swarm::wait_for_all_nodes_to_catchup_to_target_version_or_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:467:24
   2: aptos_forge::interface::swarm::wait_for_all_nodes_to_catchup_to_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:400:6
   3: aptos_forge::interface::swarm::SwarmExt::wait_for_all_nodes_to_change_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:306:92
   4: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
   5: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:63
   6: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:107:5
   7: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:73:5
   8: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:31
   9: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/blocking.rs:66:9
  10: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:87:13
  11: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/runtime.rs:65:16
  12: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:86:9
  13: tokio::runtime::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/runtime.rs:350:50
  14: aptos_forge::interface::swarm::SwarmExt::fork_check
             at ./testsuite/forge/src/interface/swarm.rs:210:9
  15: <aptos_testcases::compatibility_test::SimpleValidatorUpgrade as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/compatibility_test.rs:105:9
  16: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:598:42
  17: aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:666:11
  18: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:598:30
  19: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
  20: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
  21: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
  22: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
  23: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
  24: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
  25: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  26: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  27: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  28: std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
  29: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  30: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  31: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  32: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  33: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  34: __libc_start_main
  35: _start
Trailing Log Lines:
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  31: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  32: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  33: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  34: __libc_start_main
  35: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:292"},"thread_name":"main","hostname":"forge-compat-pr-13262-1715584701-01b24e7e3548382dd25440b39a0438","timestamp":"2024-05-13T07:34:16.573427Z","message":"Deleting namespace forge-compat-pr-13262: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:400"},"thread_name":"main","hostname":"forge-compat-pr-13262-1715584701-01b24e7e3548382dd25440b39a0438","timestamp":"2024-05-13T07:34:16.573455Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-13262"}

failures:
    compatibility::simple-validator-upgrade

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:83:36
   1: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:618:13
   2: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
   3: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
   4: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
   5: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
   6: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
   7: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
   8: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
   9: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  10: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  11: std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
  12: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  13: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  14: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  15: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  16: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  17: __libc_start_main
  18: _start
Debugging output:
NAME                                    READY   STATUS             RESTARTS        AGE
aptos-node-0-validator-0                0/1     CrashLoopBackOff   5 (2m44s ago)   7m38s
aptos-node-1-validator-0                0/1     CrashLoopBackOff   6 (2m ago)      11m
aptos-node-2-validator-0                1/1     Running            0               15m
aptos-node-3-validator-0                1/1     Running            0               15m
genesis-aptos-genesis-eforge198-dgqwf   0/1     Completed          0               15m
github-actions[bot] commented 2 weeks ago

:x: Forge suite realistic_env_max_load failure on 1f7b0dc77082bcaffb28024671fffc07bb853188

Forge test runner terminated:
Trailing Log Lines:
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:672"},"thread_name":"main","hostname":"forge-e2e-pr-13262-1715584708-1f7b0dc77082bcaffb28024671fffc07b","timestamp":"2024-05-13T11:17:17.405903Z","message":"Node fullnode-2 unhealthy: Unknown error error sending request for url (http://aptos-node-2-fullnode.forge-e2e-pr-13262.svc:8080/v1/): error trying to connect: dns error: failed to lookup address information: Name or service not known"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:672"},"thread_name":"main","hostname":"forge-e2e-pr-13262-1715584708-1f7b0dc77082bcaffb28024671fffc07b","timestamp":"2024-05-13T11:17:27.508419Z","message":"Node fullnode-2 unhealthy: Unknown error error sending request for url (http://aptos-node-2-fullnode.forge-e2e-pr-13262.svc:8080/v1/): error trying to connect: dns error: failed to lookup address information: Name or service not known"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:672"},"thread_name":"main","hostname":"forge-e2e-pr-13262-1715584708-1f7b0dc77082bcaffb28024671fffc07b","timestamp":"2024-05-13T11:17:37.543030Z","message":"Node fullnode-2 unhealthy: Unknown error error sending request for url (http://aptos-node-2-fullnode.forge-e2e-pr-13262.svc:8080/v1/): error trying to connect: dns error: failed to lookup address information: Name or service not known"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:672"},"thread_name":"main","hostname":"forge-e2e-pr-13262-1715584708-1f7b0dc77082bcaffb28024671fffc07b","timestamp":"2024-05-13T11:17:47.578607Z","message":"Node fullnode-2 unhealthy: Unknown error error sending request for url (http://aptos-node-2-fullnode.forge-e2e-pr-13262.svc:8080/v1/): error trying to connect: dns error: failed to lookup address information: Name or service not known"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:672"},"thread_name":"main","hostname":"forge-e2e-pr-13262-1715584708-1f7b0dc77082bcaffb28024671fffc07b","timestamp":"2024-05-13T11:17:57.698689Z","message":"Node fullnode-2 unhealthy: Unknown error error sending request for url (http://aptos-node-2-fullnode.forge-e2e-pr-13262.svc:8080/v1/): error trying to connect: dns error: failed to lookup address information: Name or service not known"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:672"},"thread_name":"main","hostname":"forge-e2e-pr-13262-1715584708-1f7b0dc77082bcaffb28024671fffc07b","timestamp":"2024-05-13T11:18:07.737044Z","message":"Node fullnode-2 unhealthy: Unknown error error sending request for url (http://aptos-node-2-fullnode.forge-e2e-pr-13262.svc:8080/v1/): error trying to connect: dns error: failed to lookup address information: Name or service not known"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:672"},"thread_name":"main","hostname":"forge-e2e-pr-13262-1715584708-1f7b0dc77082bcaffb28024671fffc07b","timestamp":"2024-05-13T11:18:17.770026Z","message":"Node fullnode-2 unhealthy: Unknown error error sending request for url (http://aptos-node-2-fullnode.forge-e2e-pr-13262.svc:8080/v1/): error trying to connect: dns error: failed to lookup address information: Name or service not known"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:672"},"thread_name":"main","hostname":"forge-e2e-pr-13262-1715584708-1f7b0dc77082bcaffb28024671fffc07b","timestamp":"2024-05-13T11:18:27.883878Z","message":"Node fullnode-2 unhealthy: Unknown error error sending request for url (http://aptos-node-2-fullnode.forge-e2e-pr-13262.svc:8080/v1/): error trying to connect: dns error: failed to lookup address information: Name or service not known"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:672"},"thread_name":"main","hostname":"forge-e2e-pr-13262-1715584708-1f7b0dc77082bcaffb28024671fffc07b","timestamp":"2024-05-13T11:18:37.924295Z","message":"Node fullnode-2 unhealthy: Unknown error error sending request for url (http://aptos-node-2-fullnode.forge-e2e-pr-13262.svc:8080/v1/): error trying to connect: dns error: failed to lookup address information: Name or service not known"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:672"},"thread_name":"main","hostname":"forge-e2e-pr-13262-1715584708-1f7b0dc77082bcaffb28024671fffc07b","timestamp":"2024-05-13T11:18:47.959368Z","message":"Node fullnode-2 unhealthy: Unknown error error sending request for url (http://aptos-node-2-fullnode.forge-e2e-pr-13262.svc:8080/v1/): error trying to connect: dns error: failed to lookup address information: Name or service not known"}
Debugging output:
Failed to get debugging output: error: You must be logged in to the server (Unauthorized)
github-actions[bot] commented 2 weeks ago

Forge is running suite realistic_env_max_load on d66810aaf844e445271b3a31af308797ce13cd4e

github-actions[bot] commented 2 weeks ago

Forge is running suite compat on 01b24e7e3548382dd25440b39a0438a993387f12 ==> d66810aaf844e445271b3a31af308797ce13cd4e

github-actions[bot] commented 2 weeks ago

:white_check_mark: Forge suite compat success on 01b24e7e3548382dd25440b39a0438a993387f12 ==> d66810aaf844e445271b3a31af308797ce13cd4e

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> d66810aaf844e445271b3a31af308797ce13cd4e (PR)
1. Check liveness of validators at old version: 01b24e7e3548382dd25440b39a0438a993387f12
compatibility::simple-validator-upgrade::liveness-check : committed: 6958 txn/s, latency: 4786 ms, (p50: 4800 ms, p90: 7600 ms, p99: 8100 ms), latency samples: 243540
2. Upgrading first Validator to new version: d66810aaf844e445271b3a31af308797ce13cd4e
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1556 txn/s, latency: 16215 ms, (p50: 18900 ms, p90: 22300 ms, p99: 29800 ms), latency samples: 91820
3. Upgrading rest of first batch to new version: d66810aaf844e445271b3a31af308797ce13cd4e
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1721 txn/s, latency: 16050 ms, (p50: 19000 ms, p90: 22400 ms, p99: 22900 ms), latency samples: 91260
4. upgrading second batch to new version: d66810aaf844e445271b3a31af308797ce13cd4e
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3676 txn/s, latency: 8652 ms, (p50: 9900 ms, p90: 11800 ms, p99: 12400 ms), latency samples: 143380
5. check swarm health
Compatibility test for 01b24e7e3548382dd25440b39a0438a993387f12 ==> d66810aaf844e445271b3a31af308797ce13cd4e passed
Test Ok
github-actions[bot] commented 2 weeks ago

:x: Forge suite realistic_env_max_load failure on d66810aaf844e445271b3a31af308797ce13cd4e

two traffics test: inner traffic : committed: 8708 txn/s, latency: 4509 ms, (p50: 4200 ms, p90: 5100 ms, p99: 9900 ms), latency samples: 3753180
two traffics test : committed: 100 txn/s, latency: 1838 ms, (p50: 1800 ms, p90: 2100 ms, p99: 4400 ms), latency samples: 1660
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.208, avg: 0.203", "QsPosToProposal: max: 0.187, avg: 0.174", "ConsensusProposalToOrdered: max: 0.410, avg: 0.372", "ConsensusOrderedToCommit: max: 0.365, avg: 0.350", "ConsensusProposalToCommit: max: 0.739, avg: 0.721"]
Test Failed: check for success

Caused by:
    error!() count in validator logs was 1691, and must be 0

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:83:36
   1: aptos_forge::success_criteria::SuccessCriteriaChecker::check_no_errors::{{closure}}
             at ./testsuite/forge/src/success_criteria.rs:552:13
   2: aptos_forge::success_criteria::SuccessCriteriaChecker::check_for_success::{{closure}}
             at ./testsuite/forge/src/success_criteria.rs:309:42
   3: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:63
   4: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:107:5
   5: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:73:5
   6: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:31
   7: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/blocking.rs:66:9
   8: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:87:13
   9: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/runtime.rs:65:16
  10: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:86:9
  11: tokio::runtime::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/runtime.rs:350:50
  12: aptos_forge::interface::network::NetworkContext::check_for_success
             at ./testsuite/forge/src/interface/network.rs:71:9
  13: <dyn aptos_testcases::NetworkLoadTest as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/lib.rs:229:13
  14: <aptos_testcases::CompositeNetworkTest as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/lib.rs:499:9
  15: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:598:42
  16: aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:666:11
  17: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:598:30
  18: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
  19: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
  20: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
  21: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
  22: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
  23: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
  24: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  25: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  26: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  27: std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
  28: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  29: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  30: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  31: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  32: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  33: __libc_start_main
  34: _start
Trailing Log Lines:
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  30: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  31: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  32: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  33: __libc_start_main
  34: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:292"},"thread_name":"main","hostname":"forge-e2e-pr-13262-1715623983-d66810aaf844e445271b3a31af308797c","timestamp":"2024-05-13T18:25:17.291095Z","message":"Deleting namespace forge-e2e-pr-13262: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:400"},"thread_name":"main","hostname":"forge-e2e-pr-13262-1715623983-d66810aaf844e445271b3a31af308797c","timestamp":"2024-05-13T18:25:17.291115Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-13262"}
Failed to run tests:
Tests Failed

failures:
    CompositeNetworkTest

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Error: Tests Failed

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:83:36
   1: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:618:13
   2: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
   3: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
   4: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
   5: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
   6: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
   7: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
   8: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
   9: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  10: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  11: std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
  12: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  13: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  14: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  15: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  16: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  17: __libc_start_main
  18: _start
Debugging output:
NAME                                   READY   STATUS      RESTARTS   AGE
aptos-node-0-fullnode-eforge91-0       1/1     Running     0          10m
aptos-node-0-validator-0               1/1     Running     0          10m
aptos-node-1-fullnode-eforge91-0       1/1     Running     0          10m
aptos-node-1-validator-0               1/1     Running     0          10m
aptos-node-2-fullnode-eforge91-0       1/1     Running     0          10m
aptos-node-2-validator-0               1/1     Running     0          10m
aptos-node-3-fullnode-eforge91-0       1/1     Running     0          10m
aptos-node-3-validator-0               1/1     Running     0          10m
aptos-node-4-fullnode-eforge91-0       1/1     Running     0          10m
aptos-node-4-validator-0               1/1     Running     0          10m
aptos-node-5-validator-0               1/1     Running     0          10m
aptos-node-6-validator-0               1/1     Running     0          10m
genesis-aptos-genesis-eforge91-9mtck   0/1     Completed   0          11m
github-actions[bot] commented 2 weeks ago

Forge is running suite realistic_env_max_load on a63c4d266e667056b53491cb87ae7fc3c8d1d616

github-actions[bot] commented 2 weeks ago

Forge is running suite compat on 01b24e7e3548382dd25440b39a0438a993387f12 ==> a63c4d266e667056b53491cb87ae7fc3c8d1d616

github-actions[bot] commented 2 weeks ago

:x: Forge suite realistic_env_max_load failure on a63c4d266e667056b53491cb87ae7fc3c8d1d616

two traffics test: inner traffic : committed: 8086 txn/s, latency: 4852 ms, (p50: 4800 ms, p90: 5700 ms, p99: 10400 ms), latency samples: 3493200
two traffics test : committed: 100 txn/s, latency: 1826 ms, (p50: 1800 ms, p90: 2000 ms, p99: 2300 ms), latency samples: 1760
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.207, avg: 0.202", "QsPosToProposal: max: 0.237, avg: 0.225", "ConsensusProposalToOrdered: max: 0.468, avg: 0.403", "ConsensusOrderedToCommit: max: 0.390, avg: 0.372", "ConsensusProposalToCommit: max: 0.786, avg: 0.775"]
Test Failed: check for success

Caused by:
    error!() count in validator logs was 2, and must be 0

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:83:36
   1: aptos_forge::success_criteria::SuccessCriteriaChecker::check_no_errors::{{closure}}
             at ./testsuite/forge/src/success_criteria.rs:552:13
   2: aptos_forge::success_criteria::SuccessCriteriaChecker::check_for_success::{{closure}}
             at ./testsuite/forge/src/success_criteria.rs:309:42
   3: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:63
   4: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:107:5
   5: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:73:5
   6: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:31
   7: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/blocking.rs:66:9
   8: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:87:13
   9: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/runtime.rs:65:16
  10: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:86:9
  11: tokio::runtime::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/runtime.rs:350:50
  12: aptos_forge::interface::network::NetworkContext::check_for_success
             at ./testsuite/forge/src/interface/network.rs:71:9
  13: <dyn aptos_testcases::NetworkLoadTest as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/lib.rs:229:13
  14: <aptos_testcases::CompositeNetworkTest as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/lib.rs:499:9
  15: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:598:42
  16: aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:666:11
  17: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:598:30
  18: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
  19: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
  20: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
  21: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
  22: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
  23: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
  24: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  25: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  26: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  27: std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
  28: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  29: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  30: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  31: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  32: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  33: __libc_start_main
  34: _start
Trailing Log Lines:
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  30: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  31: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  32: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  33: __libc_start_main
  34: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:292"},"thread_name":"main","hostname":"forge-e2e-pr-13262-1715633921-a63c4d266e667056b53491cb87ae7fc3c","timestamp":"2024-05-13T21:10:24.300849Z","message":"Deleting namespace forge-e2e-pr-13262: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:400"},"thread_name":"main","hostname":"forge-e2e-pr-13262-1715633921-a63c4d266e667056b53491cb87ae7fc3c","timestamp":"2024-05-13T21:10:24.300880Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-13262"}

failures:
    CompositeNetworkTest

Failed to run tests:
Tests Failed
test result: FAILED. 0 passed; 1 failed; 0 filtered out

Error: Tests Failed

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:83:36
   1: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:618:13
   2: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
   3: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
   4: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
   5: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
   6: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
   7: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
   8: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
   9: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  10: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  11: std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
  12: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  13: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  14: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  15: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  16: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  17: __libc_start_main
  18: _start
Debugging output:
NAME                                READY   STATUS    RESTARTS   AGE
aptos-node-0-fullnode-eforge108-0   1/1     Running   0          10m
aptos-node-0-validator-0            1/1     Running   0          10m
aptos-node-1-fullnode-eforge108-0   1/1     Running   0          10m
aptos-node-1-validator-0            1/1     Running   0          10m
aptos-node-2-fullnode-eforge108-0   1/1     Running   0          10m
aptos-node-2-validator-0            1/1     Running   0          10m
aptos-node-3-fullnode-eforge108-0   1/1     Running   0          10m
aptos-node-3-validator-0            1/1     Running   0          10m
aptos-node-4-fullnode-eforge108-0   1/1     Running   0          10m
aptos-node-4-validator-0            1/1     Running   0          10m
aptos-node-5-validator-0            1/1     Running   0          10m
aptos-node-6-validator-0            1/1     Running   0          10m
github-actions[bot] commented 2 weeks ago

:x: Forge suite compat failure on 01b24e7e3548382dd25440b39a0438a993387f12 ==> a63c4d266e667056b53491cb87ae7fc3c8d1d616

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> a63c4d266e667056b53491cb87ae7fc3c8d1d616 (PR)
1. Check liveness of validators at old version: 01b24e7e3548382dd25440b39a0438a993387f12
compatibility::simple-validator-upgrade::liveness-check : committed: 6781 txn/s, latency: 4910 ms, (p50: 4800 ms, p90: 8000 ms, p99: 9500 ms), latency samples: 237340
2. Upgrading first Validator to new version: a63c4d266e667056b53491cb87ae7fc3c8d1d616
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1743 txn/s, latency: 15835 ms, (p50: 19000 ms, p90: 22000 ms, p99: 22700 ms), latency samples: 92380
3. Upgrading rest of first batch to new version: a63c4d266e667056b53491cb87ae7fc3c8d1d616
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1554 txn/s, latency: 17232 ms, (p50: 20600 ms, p90: 26500 ms, p99: 27200 ms), latency samples: 79280
4. upgrading second batch to new version: a63c4d266e667056b53491cb87ae7fc3c8d1d616
Test Failed: Tried executing 10 txns, request counters: "success 0, failed submit [0], failed wait [10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10], by client: [(0, 0, 100): http://aptos-node-3-validator.forge-compat-pr-13262.svc:8080/v1/]\n[(0, 0, 80): http://aptos-node-2-validator.forge-compat-pr-13262.svc:8080/v1/]"

Caused by:
    Unknown error Ledger on endpoint (http://aptos-node-3-validator.forge-compat-pr-13262.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (9b6afe90) might still succeed.

Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:565:25
   1: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/result.rs:1963:27
   2: aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiReliableTransactionSubmitter::submit_check_and_retry::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:127:28
   3: <futures_util::future::maybe_done::MaybeDone<Fut> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/maybe_done.rs:95:38
   4: <futures_util::future::join_all::JoinAll<F> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/join_all.rs:143:24
   5: <aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiReliableTransactionSubmitter as aptos_transaction_generator_lib::ReliableTransactionSubmitter>::execute_transactions_with_counter::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:309:10
   6: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
   7: aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_and_fund_seed_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:434:18
   8: aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_and_fund_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:326:14
   9: aptos_transaction_emitter_lib::emitter::create_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:1195:14
  10: aptos_transaction_emitter_lib::emitter::TxnEmitter::start_job::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:717:10
  11: aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for_impl::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:827:14
  12: aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:859:14
  13: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:63
  14: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:107:5
  15: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:73:5
  16: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:31
  17: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/blocking.rs:66:9
  18: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:87:13
  19: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/runtime.rs:65:16
  20: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:86:9
  21: tokio::runtime::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/runtime.rs:350:50
  22: aptos_testcases::generate_traffic
             at ./testsuite/testcases/src/lib.rs:105:17
  23: <aptos_testcases::compatibility_test::SimpleValidatorUpgrade as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/compatibility_test.rs:114:24
  24: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:598:42
  25: aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:666:11
  26: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:598:30
  27: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
  28: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
  29: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
  30: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
  31: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
  32: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
  33: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  34: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  35: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  36: std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
  37: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  38: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  39: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  40: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  41: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  42: __libc_start_main
  43: _start
Trailing Log Lines:
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  39: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  40: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  41: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  42: __libc_start_main
  43: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:292"},"thread_name":"main","hostname":"forge-compat-pr-13262-1715633930-01b24e7e3548382dd25440b39a0438","timestamp":"2024-05-13T21:13:10.513468Z","message":"Deleting namespace forge-compat-pr-13262: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:400"},"thread_name":"main","hostname":"forge-compat-pr-13262-1715633930-01b24e7e3548382dd25440b39a0438","timestamp":"2024-05-13T21:13:10.513490Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-13262"}

failures:
    compatibility::simple-validator-upgrade

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:83:36
   1: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:618:13
   2: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
   3: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
   4: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
   5: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
   6: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
   7: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
   8: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
   9: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  10: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  11: std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
  12: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  13: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  14: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  15: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  16: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  17: __libc_start_main
  18: _start
Debugging output:
NAME                                   READY   STATUS      RESTARTS   AGE
aptos-node-0-validator-0               1/1     Running     0          7m55s
aptos-node-1-validator-0               1/1     Running     0          11m
aptos-node-2-validator-0               1/1     Running     0          5m26s
aptos-node-3-validator-0               1/1     Running     0          4m44s
genesis-aptos-genesis-eforge14-j7l6t   0/1     Completed   0          13m
github-actions[bot] commented 2 weeks ago

Forge is running suite compat on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 7d033a754f793d3206ce78e2a07b2d97539c74ab

github-actions[bot] commented 2 weeks ago

Forge is running suite realistic_env_max_load on 7d033a754f793d3206ce78e2a07b2d97539c74ab

github-actions[bot] commented 2 weeks ago

:white_check_mark: Forge suite realistic_env_max_load success on 7d033a754f793d3206ce78e2a07b2d97539c74ab

two traffics test: inner traffic : committed: 8395 txn/s, latency: 4683 ms, (p50: 4500 ms, p90: 5400 ms, p99: 10200 ms), latency samples: 3618520
two traffics test : committed: 100 txn/s, latency: 1978 ms, (p50: 1900 ms, p90: 2100 ms, p99: 8700 ms), latency samples: 1800
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.208, avg: 0.202", "QsPosToProposal: max: 0.238, avg: 0.217", "ConsensusProposalToOrdered: max: 0.443, avg: 0.399", "ConsensusOrderedToCommit: max: 0.372, avg: 0.361", "ConsensusProposalToCommit: max: 0.769, avg: 0.760"]
Max round gap was 1 [limit 4] at version 1836618. Max no progress secs was 4.981166 [limit 15] at version 1836618.
Test Ok
github-actions[bot] commented 2 weeks ago

:x: Forge suite compat failure on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 7d033a754f793d3206ce78e2a07b2d97539c74ab

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 7d033a754f793d3206ce78e2a07b2d97539c74ab (PR)
1. Check liveness of validators at old version: 01b24e7e3548382dd25440b39a0438a993387f12
compatibility::simple-validator-upgrade::liveness-check : committed: 6524 txn/s, latency: 5005 ms, (p50: 4800 ms, p90: 7800 ms, p99: 8900 ms), latency samples: 254460
2. Upgrading first Validator to new version: 7d033a754f793d3206ce78e2a07b2d97539c74ab
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1452 txn/s, latency: 18690 ms, (p50: 21400 ms, p90: 29500 ms, p99: 30300 ms), latency samples: 74100
3. Upgrading rest of first batch to new version: 7d033a754f793d3206ce78e2a07b2d97539c74ab
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1441 txn/s, latency: 18717 ms, (p50: 23100 ms, p90: 29400 ms, p99: 30500 ms), latency samples: 73540
4. upgrading second batch to new version: 7d033a754f793d3206ce78e2a07b2d97539c74ab
Test Failed: Tried executing 10 txns, request counters: "success 0, failed submit [0], failed wait [10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10], by client: [(0, 0, 70): http://aptos-node-2-validator.forge-compat-pr-13262.svc:8080/v1/]\n[(0, 0, 110): http://aptos-node-3-validator.forge-compat-pr-13262.svc:8080/v1/]"

Caused by:
    Unknown error Ledger on endpoint (http://aptos-node-2-validator.forge-compat-pr-13262.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (bccef9f9) might still succeed.

Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:565:25
   1: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/result.rs:1963:27
   2: aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiReliableTransactionSubmitter::submit_check_and_retry::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:127:28
   3: <futures_util::future::maybe_done::MaybeDone<Fut> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/maybe_done.rs:95:38
   4: <futures_util::future::join_all::JoinAll<F> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/join_all.rs:143:24
   5: <aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiReliableTransactionSubmitter as aptos_transaction_generator_lib::ReliableTransactionSubmitter>::execute_transactions_with_counter::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:309:10
   6: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
   7: aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_and_fund_seed_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:434:18
   8: aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_and_fund_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:326:14
   9: aptos_transaction_emitter_lib::emitter::create_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:1195:14
  10: aptos_transaction_emitter_lib::emitter::TxnEmitter::start_job::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:717:10
  11: aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for_impl::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:827:14
  12: aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:859:14
  13: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:63
  14: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:107:5
  15: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:73:5
  16: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:31
  17: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/blocking.rs:66:9
  18: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:87:13
  19: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/runtime.rs:65:16
  20: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:86:9
  21: tokio::runtime::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/runtime.rs:350:50
  22: aptos_testcases::generate_traffic
             at ./testsuite/testcases/src/lib.rs:105:17
  23: <aptos_testcases::compatibility_test::SimpleValidatorUpgrade as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/compatibility_test.rs:114:24
  24: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:598:42
  25: aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:666:11
  26: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:598:30
  27: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
  28: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
  29: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
  30: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
  31: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
  32: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
  33: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  34: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  35: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  36: std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
  37: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  38: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  39: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  40: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  41: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  42: __libc_start_main
  43: _start
Trailing Log Lines:
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  39: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  40: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  41: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  42: __libc_start_main
  43: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:292"},"thread_name":"main","hostname":"forge-compat-pr-13262-1715642553-01b24e7e3548382dd25440b39a0438","timestamp":"2024-05-13T23:36:25.401669Z","message":"Deleting namespace forge-compat-pr-13262: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:400"},"thread_name":"main","hostname":"forge-compat-pr-13262-1715642553-01b24e7e3548382dd25440b39a0438","timestamp":"2024-05-13T23:36:25.401703Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-13262"}
Failed to run tests:
Tests Failed

failures:
    compatibility::simple-validator-upgrade

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Error: Tests Failed

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:83:36
   1: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:618:13
   2: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
   3: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
   4: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
   5: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
   6: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
   7: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
   8: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
   9: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  10: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  11: std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
  12: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  13: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  14: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  15: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  16: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  17: __libc_start_main
  18: _start
Debugging output:
NAME                       READY   STATUS    RESTARTS   AGE
aptos-node-0-validator-0   1/1     Running   0          7m51s
aptos-node-1-validator-0   1/1     Running   0          9m50s
aptos-node-2-validator-0   1/1     Running   0          5m20s
aptos-node-3-validator-0   1/1     Running   0          4m35s
github-actions[bot] commented 2 weeks ago

Forge is running suite realistic_env_max_load on 25ff0defeabf53dc5f3ba8cd48cdc95dc5a03fac

github-actions[bot] commented 2 weeks ago

Forge is running suite compat on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 25ff0defeabf53dc5f3ba8cd48cdc95dc5a03fac

github-actions[bot] commented 2 weeks ago

:white_check_mark: Forge suite realistic_env_max_load success on 25ff0defeabf53dc5f3ba8cd48cdc95dc5a03fac

two traffics test: inner traffic : committed: 7809 txn/s, latency: 5012 ms, (p50: 4800 ms, p90: 5700 ms, p99: 10500 ms), latency samples: 3381540
two traffics test : committed: 100 txn/s, latency: 1921 ms, (p50: 1900 ms, p90: 2200 ms, p99: 3900 ms), latency samples: 1760
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.209, avg: 0.202", "QsPosToProposal: max: 0.276, avg: 0.253", "ConsensusProposalToOrdered: max: 0.461, avg: 0.410", "ConsensusOrderedToCommit: max: 0.362, avg: 0.350", "ConsensusProposalToCommit: max: 0.772, avg: 0.760"]
Max round gap was 1 [limit 4] at version 1700075. Max no progress secs was 4.614867 [limit 15] at version 1700075.
Test Ok
github-actions[bot] commented 2 weeks ago

:x: Forge suite compat failure on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 25ff0defeabf53dc5f3ba8cd48cdc95dc5a03fac

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 25ff0defeabf53dc5f3ba8cd48cdc95dc5a03fac (PR)
1. Check liveness of validators at old version: 01b24e7e3548382dd25440b39a0438a993387f12
compatibility::simple-validator-upgrade::liveness-check : committed: 5575 txn/s, latency: 5901 ms, (p50: 5100 ms, p90: 8400 ms, p99: 17300 ms), latency samples: 195140
2. Upgrading first Validator to new version: 25ff0defeabf53dc5f3ba8cd48cdc95dc5a03fac
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1498 txn/s, latency: 17310 ms, (p50: 19500 ms, p90: 28000 ms, p99: 29600 ms), latency samples: 88400
3. Upgrading rest of first batch to new version: 25ff0defeabf53dc5f3ba8cd48cdc95dc5a03fac
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1513 txn/s, latency: 17337 ms, (p50: 18600 ms, p90: 30800 ms, p99: 31200 ms), latency samples: 92300
4. upgrading second batch to new version: 25ff0defeabf53dc5f3ba8cd48cdc95dc5a03fac
Test Failed: Tried executing 10 txns, request counters: "success 0, failed submit [0], failed wait [10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10], by client: [(0, 0, 90): http://aptos-node-3-validator.forge-compat-pr-13262.svc:8080/v1/]\n[(0, 0, 90): http://aptos-node-2-validator.forge-compat-pr-13262.svc:8080/v1/]"

Caused by:
    Unknown error Ledger on endpoint (http://aptos-node-3-validator.forge-compat-pr-13262.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (b13d747f) might still succeed.

Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:565:25
   1: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/result.rs:1963:27
   2: aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiReliableTransactionSubmitter::submit_check_and_retry::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:127:28
   3: <futures_util::future::maybe_done::MaybeDone<Fut> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/maybe_done.rs:95:38
   4: <futures_util::future::join_all::JoinAll<F> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/join_all.rs:143:24
   5: <aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiReliableTransactionSubmitter as aptos_transaction_generator_lib::ReliableTransactionSubmitter>::execute_transactions_with_counter::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:309:10
   6: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
   7: aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_and_fund_seed_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:434:18
   8: aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_and_fund_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:326:14
   9: aptos_transaction_emitter_lib::emitter::create_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:1195:14
  10: aptos_transaction_emitter_lib::emitter::TxnEmitter::start_job::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:717:10
  11: aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for_impl::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:827:14
  12: aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:859:14
  13: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:63
  14: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:107:5
  15: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:73:5
  16: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:31
  17: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/blocking.rs:66:9
  18: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:87:13
  19: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/runtime.rs:65:16
  20: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:86:9
  21: tokio::runtime::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/runtime.rs:350:50
  22: aptos_testcases::generate_traffic
             at ./testsuite/testcases/src/lib.rs:105:17
  23: <aptos_testcases::compatibility_test::SimpleValidatorUpgrade as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/compatibility_test.rs:114:24
  24: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:598:42
  25: aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:666:11
  26: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:598:30
  27: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
  28: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
  29: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
  30: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
  31: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
  32: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
  33: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  34: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  35: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  36: std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
  37: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  38: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  39: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  40: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  41: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  42: __libc_start_main
  43: _start
Trailing Log Lines:
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  39: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  40: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  41: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  42: __libc_start_main
  43: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:292"},"thread_name":"main","hostname":"forge-compat-pr-13262-1715951775-01b24e7e3548382dd25440b39a0438","timestamp":"2024-05-17T13:32:29.429353Z","message":"Deleting namespace forge-compat-pr-13262: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:400"},"thread_name":"main","hostname":"forge-compat-pr-13262-1715951775-01b24e7e3548382dd25440b39a0438","timestamp":"2024-05-17T13:32:29.429403Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-13262"}
Failed to run tests:
Tests Failed

failures:
    compatibility::simple-validator-upgrade

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Error: Tests Failed

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:83:36
   1: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:618:13
   2: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
   3: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
   4: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
   5: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
   6: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
   7: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
   8: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
   9: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  10: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  11: std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
  12: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  13: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  14: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  15: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  16: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  17: __libc_start_main
  18: _start
Debugging output:
NAME                       READY   STATUS    RESTARTS   AGE
aptos-node-0-validator-0   1/1     Running   0          7m49s
aptos-node-1-validator-0   1/1     Running   0          10m
aptos-node-2-validator-0   1/1     Running   0          5m31s
aptos-node-3-validator-0   1/1     Running   0          4m44s