[TEST] randomness - Githubissues

aptos-labs / aptos-core

Aptos is a layer 1 blockchain built to support the widespread use of blockchain through better technology and user experience.

https://aptosfoundation.org

Other

5.86k stars 3.54k forks source link

[TEST] randomness #8994

Closed danielxiangzl closed 10 months ago

danielxiangzl commented 11 months ago

Description

Test Plan

github-actions[bot] commented 11 months ago

Forge is running suite `realistic_env_max_load` on `cc27e824bce47efa788d79aaaac94c9aedd813e6`

github-actions[bot] commented 11 months ago

Forge is running suite `compat` on `aptos-node-v1.4.4` ==> `cc27e824bce47efa788d79aaaac94c9aedd813e6`

github-actions[bot] commented 11 months ago

Forge is running suite `framework_upgrade` on `aptos-node-v1.4.4` ==> `cc27e824bce47efa788d79aaaac94c9aedd813e6`

github-actions[bot] commented 11 months ago

:x: Forge suite `compat` failure on `aptos-node-v1.4.4` ==> `cc27e824bce47efa788d79aaaac94c9aedd813e6`

Compatibility test results for aptos-node-v1.4.4 ==> cc27e824bce47efa788d79aaaac94c9aedd813e6 (PR)
1. Check liveness of validators at old version: aptos-node-v1.4.4
compatibility::simple-validator-upgrade::liveness-check : committed: 7032 txn/s, latency: 4697 ms, (p50: 4600 ms, p90: 6700 ms, p99: 8200 ms), latency samples: 246140
2. Upgrading first Validator to new version: cc27e824bce47efa788d79aaaac94c9aedd813e6
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1739 txn/s, latency: 18728 ms, (p50: 17000 ms, p90: 25500 ms, p99: 30100 ms), latency samples: 80000
3. Upgrading rest of first batch to new version: cc27e824bce47efa788d79aaaac94c9aedd813e6
Test Failed: Tried executing 10 txns, request counters: "success 0, failed submit [0], failed wait [10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10], by client: [(0, 0, 180): http://aptos-node-0-validator.forge-compat-pr-8994.svc:8080/v1/]"

Caused by:
    Unknown error Ledger on endpoint (http://aptos-node-0-validator.forge-compat-pr-8994.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (3f57717d) might still succeed.

Stack backtrace:
   0: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/result.rs:2002:27
      aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiReliableTransactionSubmitter::submit_check_and_retry::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:119:28
   1: <futures_util::future::maybe_done::MaybeDone<Fut> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.24/src/future/maybe_done.rs:95:38
      <futures_util::future::join_all::JoinAll<F> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.24/src/future/join_all.rs:142:24
   2: <aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiReliableTransactionSubmitter as aptos_transaction_generator_lib::ReliableTransactionSubmitter>::execute_transactions_with_counter::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:273:9
   3: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/future/future.rs:125:9
      aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_and_fund_seed_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:314:17
      aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:187:13
   4: aptos_transaction_emitter_lib::emitter::TxnEmitter::start_job::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:608:13
   5: aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for_impl::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:712:13
      aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:744:13
      tokio::park::thread::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/park/thread.rs:267:54
      tokio::coop::with_budget::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:102:9
      std::thread::local::LocalKey<T>::try_with
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/thread/local.rs:252:16
      std::thread::local::LocalKey<T>::with
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/thread/local.rs:228:9
   6: tokio::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:95:5
      tokio::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:72:5
      tokio::park::thread::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/park/thread.rs:267:31
   7: tokio::runtime::enter::Enter::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/enter.rs:152:13
      tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/scheduler/multi_thread/mod.rs:79:9
   8: tokio::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/mod.rs:492:44
   9: aptos_testcases::generate_traffic
             at ./testsuite/testcases/src/lib.rs:99:17
  10: <aptos_testcases::compatibility_test::SimpleValidatorUpgrade as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/compatibility_test.rs:93:24
  11: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:482:42
      aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:550:11
      aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:482:30
  12: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:380:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:306:21
  13: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
  14: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
  15: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  16: main
  17: __libc_start_main
  18: _start
Trailing Log Lines:
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  16: main
  17: __libc_start_main
  18: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:291"},"thread_name":"main","hostname":"forge-compat-pr-8994-1688708130-aptos-node-v1-4-4","timestamp":"2023-07-07T05:47:29.172886Z","message":"Deleting namespace forge-compat-pr-8994: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:399"},"thread_name":"main","hostname":"forge-compat-pr-8994-1688708130-aptos-node-v1-4-4","timestamp":"2023-07-07T05:47:29.172917Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-8994"}
Failed to run tests:
Tests Failed

failures:
    compatibility::simple-validator-upgrade

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Error: Tests Failed

Stack backtrace:
   0: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:502:13
   1: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:380:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:306:21
   2: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
   3: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
   4: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
   5: main
   6: __libc_start_main
   7: _start
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-validator-0                1/1     Running     0          4m57s
aptos-node-1-validator-0                1/1     Running     0          9m13s
aptos-node-2-validator-0                1/1     Running     0          11m
aptos-node-3-validator-0                1/1     Running     0          11m
aptos-node-4-validator-0                1/1     Running     0          11m
genesis-aptos-genesis-eforge233-r79v6   0/1     Completed   0          11m

Grafana dashboard
Humio Logs
Test runner output
Test run is land-blocking

github-actions[bot] commented 11 months ago

:x: Forge suite `framework_upgrade` failure on `aptos-node-v1.4.4` ==> `cc27e824bce47efa788d79aaaac94c9aedd813e6`

Compatibility test results for aptos-node-v1.4.4 ==> cc27e824bce47efa788d79aaaac94c9aedd813e6 (PR)
Upgrade the nodes to version: cc27e824bce47efa788d79aaaac94c9aedd813e6
Test Failed: Waiting for nodes to catch up to target version and epoch (Some(351), None) timed out after 10 seconds, current status: [("validator-0", 341, 4), ("validator-1", 341, 4), ("validator-2", 351, 4), ("validator-3", 351, 4), ("validator-4", 351, 4)]

Stack backtrace:
   0: aptos_forge::interface::swarm::wait_for_all_nodes_to_catchup_to_target_version_or_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:451:24
   1: aptos_forge::interface::swarm::wait_for_all_nodes_to_catchup_to_version::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:368:5
   2: aptos_forge::interface::swarm::SwarmExt::wait_for_all_nodes_to_catchup_to_version::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:254:9
   3: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/future/future.rs:125:9
      tokio::park::thread::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/park/thread.rs:267:54
      tokio::coop::with_budget::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:102:9
      std::thread::local::LocalKey<T>::try_with
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/thread/local.rs:252:16
      std::thread::local::LocalKey<T>::with
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/thread/local.rs:228:9
      tokio::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:95:5
      tokio::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:72:5
      tokio::park::thread::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/park/thread.rs:267:31
   4: tokio::runtime::enter::Enter::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/enter.rs:152:13
      tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/scheduler/multi_thread/mod.rs:79:9
   5: tokio::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/mod.rs:492:44
   6: aptos_forge::interface::swarm::SwarmExt::fork_check
             at ./testsuite/forge/src/interface/swarm.rs:232:9
   7: <aptos_testcases::framework_upgrade::FrameworkUpgrade as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/framework_upgrade.rs:58:9
   8: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:482:42
      aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:550:11
      aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:482:30
   9: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:380:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:306:21
  10: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
  11: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
  12: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  13: main
  14: __libc_start_main
  15: _start
Trailing Log Lines:
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  13: main
  14: __libc_start_main
  15: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:291"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-8994-1688708151-aptos-node-v1-4-4","timestamp":"2023-07-07T05:48:16.250914Z","message":"Deleting namespace forge-framework-upgrade-pr-8994: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:399"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-8994-1688708151-aptos-node-v1-4-4","timestamp":"2023-07-07T05:48:16.250982Z","message":"aptos-node resources for Forge removed in namespace: forge-framework-upgrade-pr-8994"}
Failed to run tests:
Tests Failed

failures:
    framework_upgrade::framework-upgrade

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Error: Tests Failed

Stack backtrace:
   0: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:502:13
   1: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:380:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:306:21
   2: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
   3: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
   4: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
   5: main
   6: __libc_start_main
   7: _start
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-validator-0                1/1     Running     0          4m15s
aptos-node-1-validator-0                1/1     Running     0          3m25s
aptos-node-2-validator-0                1/1     Running     0          2m35s
aptos-node-3-validator-0                1/1     Running     0          104s
aptos-node-4-validator-0                1/1     Running     0          54s
genesis-aptos-genesis-eforge139-mp762   0/1     Completed   0          10m

Grafana dashboard
Humio Logs
Test runner output
Test run is land-blocking

github-actions[bot] commented 11 months ago

:x: Forge suite `realistic_env_max_load` failure on `cc27e824bce47efa788d79aaaac94c9aedd813e6`

two traffics test: inner traffic : committed: 2337 txn/s, submitted: 2339 txn/s, expired: 1 txn/s, latency: 15899 ms, (p50: 13800 ms, p90: 28200 ms, p99: 47300 ms), latency samples: 1091480
Test Failed: test NetworkLoadTest

Caused by:
    TPS requirement for inner traffic failed. Average TPS 2337, minimum TPS requirement 5000. Full stats: committed: 2337 txn/s, submitted: 2339 txn/s, expired: 1 txn/s, latency: 15899 ms, (p50: 13800 ms, p90: 28200 ms, p99: 47300 ms), latency samples: 1091480

Stack backtrace:
   0: aptos_forge::success_criteria::SuccessCriteriaChecker::check_tps
             at ./testsuite/forge/src/success_criteria.rs:288:13
   1: aptos_forge::success_criteria::SuccessCriteriaChecker::check_throughput
             at ./testsuite/forge/src/success_criteria.rs:341:9
      aptos_forge::success_criteria::SuccessCriteriaChecker::check_core_for_success
             at ./testsuite/forge/src/success_criteria.rs:101:9
   2: <aptos_testcases::two_traffics_test::TwoTrafficsTest as aptos_testcases::NetworkLoadTest>::test
             at ./testsuite/testcases/src/two_traffics_test.rs:69:9
   3: <dyn aptos_testcases::NetworkLoadTest>::network_load_test
             at ./testsuite/testcases/src/lib.rs:283:13
   4: <dyn aptos_testcases::NetworkLoadTest as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/lib.rs:174:87
   5: <aptos_testcases::CompositeNetworkTest as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/lib.rs:396:9
   6: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:482:42
      aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:550:11
      aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:482:30
   7: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:380:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:306:21
   8: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
   9: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
  10: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  11: main
  12: __libc_start_main
  13: _start
Trailing Log Lines:
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  11: main
  12: __libc_start_main
  13: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:291"},"thread_name":"main","hostname":"forge-e2e-pr-8994-1688708126-cc27e824bce47efa788d79aaaac94c9aed","timestamp":"2023-07-07T05:53:53.544466Z","message":"Deleting namespace forge-e2e-pr-8994: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:399"},"thread_name":"main","hostname":"forge-e2e-pr-8994-1688708126-cc27e824bce47efa788d79aaaac94c9aed","timestamp":"2023-07-07T05:53:53.544492Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-8994"}
Failed to run tests:
Tests Failed

failures:
    CompositeNetworkTest

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Error: Tests Failed

Stack backtrace:
   0: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:502:13
   1: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:380:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:306:21
   2: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
   3: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
   4: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
   5: main
   6: __libc_start_main
   7: _start
Debugging output:
NAME                                   READY   STATUS      RESTARTS   AGE
aptos-node-0-fullnode-eforge55-0       1/1     Running     0          15m
aptos-node-0-validator-0               1/1     Running     0          15m
aptos-node-1-fullnode-eforge55-0       1/1     Running     0          15m
aptos-node-1-validator-0               1/1     Running     0          15m
aptos-node-10-validator-0              1/1     Running     0          15m
aptos-node-11-validator-0              1/1     Running     0          15m
aptos-node-12-validator-0              1/1     Running     0          15m
aptos-node-13-validator-0              1/1     Running     0          15m
aptos-node-14-validator-0              1/1     Running     0          15m
aptos-node-15-validator-0              1/1     Running     0          15m
aptos-node-16-validator-0              1/1     Running     0          15m
aptos-node-17-validator-0              1/1     Running     0          15m
aptos-node-18-validator-0              1/1     Running     0          15m
aptos-node-19-validator-0              1/1     Running     0          15m
aptos-node-2-fullnode-eforge55-0       1/1     Running     0          15m
aptos-node-2-validator-0               1/1     Running     0          15m
aptos-node-3-fullnode-eforge55-0       1/1     Running     0          15m
aptos-node-3-validator-0               1/1     Running     0          15m
aptos-node-4-fullnode-eforge55-0       1/1     Running     0          15m
aptos-node-4-validator-0               1/1     Running     0          15m
aptos-node-5-fullnode-eforge55-0       1/1     Running     0          15m
aptos-node-5-validator-0               1/1     Running     0          15m
aptos-node-6-fullnode-eforge55-0       1/1     Running     0          15m
aptos-node-6-validator-0               1/1     Running     0          15m
aptos-node-7-fullnode-eforge55-0       1/1     Running     0          15m
aptos-node-7-validator-0               1/1     Running     0          15m
aptos-node-8-fullnode-eforge55-0       1/1     Running     0          15m
aptos-node-8-validator-0               1/1     Running     0          15m
aptos-node-9-fullnode-eforge55-0       1/1     Running     0          15m
aptos-node-9-validator-0               1/1     Running     0          15m
genesis-aptos-genesis-eforge55-msrw2   0/1     Completed   0          18m

Grafana dashboard
Humio Logs
Test runner output
Test run is land-blocking

github-actions[bot] commented 11 months ago

Forge is running suite `framework_upgrade` on `aptos-node-v1.4.4` ==> `6b244f1a448ca211cc94c61d0b38f9abe7abd599`

github-actions[bot] commented 11 months ago

Forge is running suite `compat` on `aptos-node-v1.4.4` ==> `6b244f1a448ca211cc94c61d0b38f9abe7abd599`

github-actions[bot] commented 11 months ago

Forge is running suite `realistic_env_max_load` on `6b244f1a448ca211cc94c61d0b38f9abe7abd599`

github-actions[bot] commented 11 months ago

:x: Forge suite `compat` failure on `aptos-node-v1.4.4` ==> `6b244f1a448ca211cc94c61d0b38f9abe7abd599`

Compatibility test results for aptos-node-v1.4.4 ==> 6b244f1a448ca211cc94c61d0b38f9abe7abd599 (PR)
1. Check liveness of validators at old version: aptos-node-v1.4.4
compatibility::simple-validator-upgrade::liveness-check : committed: 6552 txn/s, latency: 4927 ms, (p50: 4900 ms, p90: 6700 ms, p99: 8600 ms), latency samples: 242440
2. Upgrading first Validator to new version: 6b244f1a448ca211cc94c61d0b38f9abe7abd599
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1448 txn/s, latency: 22804 ms, (p50: 28000 ms, p90: 30000 ms, p99: 30000 ms), latency samples: 50700
3. Upgrading rest of first batch to new version: 6b244f1a448ca211cc94c61d0b38f9abe7abd599
Test Failed: Tried executing 10 txns, request counters: "success 0, failed submit [0], failed wait [10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10], by client: [(0, 0, 180): http://aptos-node-0-validator.forge-compat-pr-8994.svc:8080/v1/]"

Caused by:
    Unknown error Ledger on endpoint (http://aptos-node-0-validator.forge-compat-pr-8994.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (3ee99ae8) might still succeed.

Stack backtrace:
   0: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/result.rs:2002:27
      aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiReliableTransactionSubmitter::submit_check_and_retry::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:119:28
   1: <futures_util::future::maybe_done::MaybeDone<Fut> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.24/src/future/maybe_done.rs:95:38
      <futures_util::future::join_all::JoinAll<F> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.24/src/future/join_all.rs:142:24
   2: <aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiReliableTransactionSubmitter as aptos_transaction_generator_lib::ReliableTransactionSubmitter>::execute_transactions_with_counter::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:273:9
   3: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/future/future.rs:125:9
      aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_and_fund_seed_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:314:17
      aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:187:13
   4: aptos_transaction_emitter_lib::emitter::TxnEmitter::start_job::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:608:13
   5: aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for_impl::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:712:13
      aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:744:13
      tokio::park::thread::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/park/thread.rs:267:54
      tokio::coop::with_budget::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:102:9
      std::thread::local::LocalKey<T>::try_with
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/thread/local.rs:252:16
      std::thread::local::LocalKey<T>::with
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/thread/local.rs:228:9
   6: tokio::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:95:5
      tokio::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:72:5
      tokio::park::thread::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/park/thread.rs:267:31
   7: tokio::runtime::enter::Enter::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/enter.rs:152:13
      tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/scheduler/multi_thread/mod.rs:79:9
   8: tokio::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/mod.rs:492:44
   9: aptos_testcases::generate_traffic
             at ./testsuite/testcases/src/lib.rs:99:17
  10: <aptos_testcases::compatibility_test::SimpleValidatorUpgrade as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/compatibility_test.rs:93:24
  11: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:482:42
      aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:550:11
      aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:482:30
  12: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:380:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:306:21
  13: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
  14: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
  15: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  16: main
  17: __libc_start_main
  18: _start
Trailing Log Lines:
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  16: main
  17: __libc_start_main
  18: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:291"},"thread_name":"main","hostname":"forge-compat-pr-8994-1688754656-aptos-node-v1-4-4","timestamp":"2023-07-07T18:45:08.436117Z","message":"Deleting namespace forge-compat-pr-8994: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:399"},"thread_name":"main","hostname":"forge-compat-pr-8994-1688754656-aptos-node-v1-4-4","timestamp":"2023-07-07T18:45:08.436158Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-8994"}

failures:
    compatibility::simple-validator-upgrade

test result: FAILED. 0 passed; 1 failed; 0 filtered out
Failed to run tests:
Tests Failed

Error: Tests Failed

Stack backtrace:
   0: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:502:13
   1: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:380:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:306:21
   2: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
   3: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
   4: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
   5: main
   6: __libc_start_main
   7: _start
Debugging output:
NAME                                   READY   STATUS      RESTARTS   AGE
aptos-node-0-validator-0               1/1     Running     0          4m50s
aptos-node-1-validator-0               1/1     Running     0          9m24s
aptos-node-2-validator-0               1/1     Running     0          13m
aptos-node-3-validator-0               1/1     Running     0          13m
aptos-node-4-validator-0               1/1     Running     0          13m
genesis-aptos-genesis-eforge97-s8q4f   0/1     Completed   0          13m

Grafana dashboard
Humio Logs
Test runner output
Test run is land-blocking

github-actions[bot] commented 11 months ago

:x: Forge suite `realistic_env_max_load` failure on `6b244f1a448ca211cc94c61d0b38f9abe7abd599`

two traffics test: inner traffic : committed: 1745 txn/s, submitted: 1746 txn/s, expired: 1 txn/s, latency: 21532 ms, (p50: 16900 ms, p90: 44600 ms, p99: 69400 ms), latency samples: 794100
Test Failed: test NetworkLoadTest

Caused by:
    TPS requirement for inner traffic failed. Average TPS 1745, minimum TPS requirement 5000. Full stats: committed: 1745 txn/s, submitted: 1746 txn/s, expired: 1 txn/s, latency: 21532 ms, (p50: 16900 ms, p90: 44600 ms, p99: 69400 ms), latency samples: 794100

Stack backtrace:
   0: aptos_forge::success_criteria::SuccessCriteriaChecker::check_tps
             at ./testsuite/forge/src/success_criteria.rs:288:13
   1: aptos_forge::success_criteria::SuccessCriteriaChecker::check_throughput
             at ./testsuite/forge/src/success_criteria.rs:341:9
      aptos_forge::success_criteria::SuccessCriteriaChecker::check_core_for_success
             at ./testsuite/forge/src/success_criteria.rs:101:9
   2: <aptos_testcases::two_traffics_test::TwoTrafficsTest as aptos_testcases::NetworkLoadTest>::test
             at ./testsuite/testcases/src/two_traffics_test.rs:69:9
   3: <dyn aptos_testcases::NetworkLoadTest>::network_load_test
             at ./testsuite/testcases/src/lib.rs:283:13
   4: <dyn aptos_testcases::NetworkLoadTest as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/lib.rs:174:87
   5: <aptos_testcases::CompositeNetworkTest as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/lib.rs:396:9
   6: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:482:42
      aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:550:11
      aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:482:30
   7: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:380:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:306:21
   8: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
   9: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
  10: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  11: main
  12: __libc_start_main
  13: _start
Trailing Log Lines:
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  11: main
  12: __libc_start_main
  13: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:291"},"thread_name":"main","hostname":"forge-e2e-pr-8994-1688754680-6b244f1a448ca211cc94c61d0b38f9abe7","timestamp":"2023-07-07T18:48:12.962279Z","message":"Deleting namespace forge-e2e-pr-8994: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:399"},"thread_name":"main","hostname":"forge-e2e-pr-8994-1688754680-6b244f1a448ca211cc94c61d0b38f9abe7","timestamp":"2023-07-07T18:48:12.962312Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-8994"}
Failed to run tests:
Tests Failed

failures:
    CompositeNetworkTest

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Error: Tests Failed

Stack backtrace:
   0: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:502:13
   1: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:380:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:306:21
   2: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
   3: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
   4: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
   5: main
   6: __libc_start_main
   7: _start
Debugging output:
NAME                                   READY   STATUS      RESTARTS   AGE
aptos-node-0-fullnode-eforge73-0       1/1     Running     0          14m
aptos-node-0-validator-0               1/1     Running     0          14m
aptos-node-1-fullnode-eforge73-0       1/1     Running     0          14m
aptos-node-1-validator-0               1/1     Running     0          14m
aptos-node-10-validator-0              1/1     Running     0          14m
aptos-node-11-validator-0              1/1     Running     0          14m
aptos-node-12-validator-0              1/1     Running     0          14m
aptos-node-13-validator-0              1/1     Running     0          14m
aptos-node-14-validator-0              1/1     Running     0          14m
aptos-node-15-validator-0              1/1     Running     0          14m
aptos-node-16-validator-0              1/1     Running     0          14m
aptos-node-17-validator-0              1/1     Running     0          14m
aptos-node-18-validator-0              1/1     Running     0          14m
aptos-node-19-validator-0              1/1     Running     0          14m
aptos-node-2-fullnode-eforge73-0       1/1     Running     0          14m
aptos-node-2-validator-0               1/1     Running     0          14m
aptos-node-3-fullnode-eforge73-0       1/1     Running     0          14m
aptos-node-3-validator-0               1/1     Running     0          14m
aptos-node-4-fullnode-eforge73-0       1/1     Running     0          14m
aptos-node-4-validator-0               1/1     Running     0          14m
aptos-node-5-fullnode-eforge73-0       1/1     Running     0          14m
aptos-node-5-validator-0               1/1     Running     0          14m
aptos-node-6-fullnode-eforge73-0       1/1     Running     0          14m
aptos-node-6-validator-0               1/1     Running     0          14m
aptos-node-7-fullnode-eforge73-0       1/1     Running     0          14m
aptos-node-7-validator-0               1/1     Running     0          14m
aptos-node-8-fullnode-eforge73-0       1/1     Running     0          14m
aptos-node-8-validator-0               1/1     Running     0          14m
aptos-node-9-fullnode-eforge73-0       1/1     Running     0          14m
aptos-node-9-validator-0               1/1     Running     0          14m
genesis-aptos-genesis-eforge73-7rbbw   0/1     Completed   0          16m

Grafana dashboard
Humio Logs
Test runner output
Test run is land-blocking

github-actions[bot] commented 11 months ago

:white_check_mark: Forge suite `framework_upgrade` success on `aptos-node-v1.4.4` ==> `6b244f1a448ca211cc94c61d0b38f9abe7abd599`

Compatibility test results for aptos-node-v1.4.4 ==> 6b244f1a448ca211cc94c61d0b38f9abe7abd599 (PR)
Upgrade the nodes to version: 6b244f1a448ca211cc94c61d0b38f9abe7abd599
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1521 txn/s, latency: 13049 ms, (p50: 13200 ms, p90: 16200 ms, p99: 51000 ms), latency samples: 108000
5. check swarm health
Compatibility test for aptos-node-v1.4.4 ==> 6b244f1a448ca211cc94c61d0b38f9abe7abd599 passed
Test Ok

Grafana dashboard
Humio Logs
Test runner output
Test run is land-blocking

github-actions[bot] commented 11 months ago

Forge is running suite `compat` on `aptos-node-v1.4.4` ==> `f43bbfd2ac47f9481810b89c4fc118d57f6819ec`

github-actions[bot] commented 11 months ago

Forge is running suite `realistic_env_max_load` on `f43bbfd2ac47f9481810b89c4fc118d57f6819ec`

github-actions[bot] commented 11 months ago

Forge is running suite `framework_upgrade` on `aptos-node-v1.4.4` ==> `f43bbfd2ac47f9481810b89c4fc118d57f6819ec`

github-actions[bot] commented 11 months ago

:x: Forge suite `compat` failure on `aptos-node-v1.4.4` ==> `f43bbfd2ac47f9481810b89c4fc118d57f6819ec`

Compatibility test results for aptos-node-v1.4.4 ==> f43bbfd2ac47f9481810b89c4fc118d57f6819ec (PR)
1. Check liveness of validators at old version: aptos-node-v1.4.4
compatibility::simple-validator-upgrade::liveness-check : committed: 6406 txn/s, latency: 4961 ms, (p50: 4900 ms, p90: 7200 ms, p99: 9200 ms), latency samples: 237040
2. Upgrading first Validator to new version: f43bbfd2ac47f9481810b89c4fc118d57f6819ec
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1699 txn/s, latency: 20827 ms, (p50: 22900 ms, p90: 23700 ms, p99: 31700 ms), latency samples: 79860
3. Upgrading rest of first batch to new version: f43bbfd2ac47f9481810b89c4fc118d57f6819ec
Test Failed: Tried executing 10 txns, request counters: "success 0, failed submit [0], failed wait [10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10], by client: [(0, 0, 180): http://aptos-node-0-validator.forge-compat-pr-8994.svc:8080/v1/]"

Caused by:
    Unknown error Ledger on endpoint (http://aptos-node-0-validator.forge-compat-pr-8994.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (4ab8b120) might still succeed.

Stack backtrace:
   0: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/result.rs:2002:27
      aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiReliableTransactionSubmitter::submit_check_and_retry::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:119:28
   1: <futures_util::future::maybe_done::MaybeDone<Fut> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.24/src/future/maybe_done.rs:95:38
      <futures_util::future::join_all::JoinAll<F> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.24/src/future/join_all.rs:142:24
   2: <aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiReliableTransactionSubmitter as aptos_transaction_generator_lib::ReliableTransactionSubmitter>::execute_transactions_with_counter::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:273:9
   3: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/future/future.rs:125:9
      aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_and_fund_seed_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:314:17
      aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:187:13
   4: aptos_transaction_emitter_lib::emitter::TxnEmitter::start_job::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:608:13
   5: aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for_impl::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:712:13
      aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:744:13
      tokio::park::thread::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/park/thread.rs:267:54
      tokio::coop::with_budget::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:102:9
      std::thread::local::LocalKey<T>::try_with
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/thread/local.rs:252:16
      std::thread::local::LocalKey<T>::with
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/thread/local.rs:228:9
   6: tokio::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:95:5
      tokio::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:72:5
      tokio::park::thread::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/park/thread.rs:267:31
   7: tokio::runtime::enter::Enter::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/enter.rs:152:13
      tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/scheduler/multi_thread/mod.rs:79:9
   8: tokio::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/mod.rs:492:44
   9: aptos_testcases::generate_traffic
             at ./testsuite/testcases/src/lib.rs:99:17
  10: <aptos_testcases::compatibility_test::SimpleValidatorUpgrade as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/compatibility_test.rs:93:24
  11: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:482:42
      aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:550:11
      aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:482:30
  12: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:380:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:306:21
  13: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
  14: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
  15: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  16: main
  17: __libc_start_main
  18: _start
Trailing Log Lines:
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  16: main
  17: __libc_start_main
  18: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:291"},"thread_name":"main","hostname":"forge-compat-pr-8994-1688764324-aptos-node-v1-4-4","timestamp":"2023-07-07T21:26:03.448862Z","message":"Deleting namespace forge-compat-pr-8994: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:399"},"thread_name":"main","hostname":"forge-compat-pr-8994-1688764324-aptos-node-v1-4-4","timestamp":"2023-07-07T21:26:03.448892Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-8994"}
Failed to run tests:
Tests Failed

failures:
    compatibility::simple-validator-upgrade

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Error: Tests Failed

Stack backtrace:
   0: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:502:13
   1: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:380:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:306:21
   2: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
   3: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
   4: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
   5: main
   6: __libc_start_main
   7: _start
Debugging output:
NAME                                   READY   STATUS      RESTARTS   AGE
aptos-node-0-validator-0               1/1     Running     0          4m55s
aptos-node-1-validator-0               1/1     Running     0          9m35s
aptos-node-2-validator-0               1/1     Running     0          13m
aptos-node-3-validator-0               1/1     Running     0          13m
aptos-node-4-validator-0               1/1     Running     0          13m
genesis-aptos-genesis-eforge75-bpjjj   0/1     Completed   0          13m

Grafana dashboard
Humio Logs
Test runner output
Test run is land-blocking

github-actions[bot] commented 11 months ago

:white_check_mark: Forge suite `realistic_env_max_load` success on `f43bbfd2ac47f9481810b89c4fc118d57f6819ec`

two traffics test: inner traffic : committed: 5646 txn/s, latency: 6880 ms, (p50: 6600 ms, p90: 8400 ms, p99: 13200 ms), latency samples: 2478660
two traffics test : committed: 100 txn/s, latency: 2824 ms, (p50: 2600 ms, p90: 3300 ms, p99: 7600 ms), latency samples: 1860
Max round gap was 1 [limit 4] at version 766280. Max no progress secs was 4.015051 [limit 10] at version 766280.
Test Ok

Grafana dashboard
Humio Logs
Test runner output
Test run is land-blocking

github-actions[bot] commented 11 months ago

Forge is running suite `framework_upgrade` on `aptos-node-v1.4.4` ==> `fa1935f9083286b3c8f57f1bccb9881c9708c0ff`

github-actions[bot] commented 11 months ago

Forge is running suite `realistic_env_max_load` on `fa1935f9083286b3c8f57f1bccb9881c9708c0ff`

github-actions[bot] commented 11 months ago

Forge is running suite `compat` on `aptos-node-v1.4.4` ==> `fa1935f9083286b3c8f57f1bccb9881c9708c0ff`

github-actions[bot] commented 11 months ago

:x: Forge suite `realistic_env_max_load` failure on `fa1935f9083286b3c8f57f1bccb9881c9708c0ff`

two traffics test: inner traffic : committed: 3897 txn/s, latency: 9974 ms, (p50: 9900 ms, p90: 12600 ms, p99: 15600 ms), latency samples: 1706920
Test Failed: test NetworkLoadTest

Caused by:
    TPS requirement for inner traffic failed. Average TPS 3897, minimum TPS requirement 5000. Full stats: committed: 3897 txn/s, latency: 9974 ms, (p50: 9900 ms, p90: 12600 ms, p99: 15600 ms), latency samples: 1706920

Stack backtrace:
   0: aptos_forge::success_criteria::SuccessCriteriaChecker::check_tps
             at ./testsuite/forge/src/success_criteria.rs:288:13
   1: aptos_forge::success_criteria::SuccessCriteriaChecker::check_throughput
             at ./testsuite/forge/src/success_criteria.rs:341:9
      aptos_forge::success_criteria::SuccessCriteriaChecker::check_core_for_success
             at ./testsuite/forge/src/success_criteria.rs:101:9
   2: <aptos_testcases::two_traffics_test::TwoTrafficsTest as aptos_testcases::NetworkLoadTest>::test
             at ./testsuite/testcases/src/two_traffics_test.rs:69:9
   3: <dyn aptos_testcases::NetworkLoadTest>::network_load_test
             at ./testsuite/testcases/src/lib.rs:283:13
   4: <dyn aptos_testcases::NetworkLoadTest as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/lib.rs:174:87
   5: <aptos_testcases::CompositeNetworkTest as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/lib.rs:396:9
   6: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:482:42
      aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:550:11
      aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:482:30
   7: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:380:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:306:21
   8: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
   9: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
  10: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  11: main
  12: __libc_start_main
  13: _start
Trailing Log Lines:
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  11: main
  12: __libc_start_main
  13: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:291"},"thread_name":"main","hostname":"forge-e2e-pr-8994-1688766449-fa1935f9083286b3c8f57f1bccb9881c97","timestamp":"2023-07-07T22:01:46.190307Z","message":"Deleting namespace forge-e2e-pr-8994: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:399"},"thread_name":"main","hostname":"forge-e2e-pr-8994-1688766449-fa1935f9083286b3c8f57f1bccb9881c97","timestamp":"2023-07-07T22:01:46.190343Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-8994"}
Failed to run tests:
Tests Failed

failures:
    CompositeNetworkTest

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Error: Tests Failed

Stack backtrace:
   0: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:502:13
   1: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:380:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:306:21
   2: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
   3: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
   4: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
   5: main
   6: __libc_start_main
   7: _start
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-validator-0                1/1     Running     0          11m
aptos-node-1-validator-0                1/1     Running     0          11m
aptos-node-10-validator-0               1/1     Running     0          11m
aptos-node-11-validator-0               1/1     Running     0          11m
aptos-node-12-validator-0               1/1     Running     0          11m
aptos-node-13-validator-0               1/1     Running     0          11m
aptos-node-14-validator-0               1/1     Running     0          11m
aptos-node-15-validator-0               1/1     Running     0          11m
aptos-node-16-validator-0               1/1     Running     0          11m
aptos-node-17-validator-0               1/1     Running     0          11m
aptos-node-18-validator-0               1/1     Running     0          11m
aptos-node-19-validator-0               1/1     Running     0          11m
aptos-node-2-validator-0                1/1     Running     0          11m
aptos-node-20-validator-0               1/1     Running     0          11m
aptos-node-21-validator-0               1/1     Running     0          11m
aptos-node-22-validator-0               1/1     Running     0          11m
aptos-node-23-validator-0               1/1     Running     0          11m
aptos-node-24-validator-0               1/1     Running     0          11m
aptos-node-25-validator-0               1/1     Running     0          11m
aptos-node-26-validator-0               1/1     Running     0          11m
aptos-node-27-validator-0               1/1     Running     0          11m
aptos-node-28-validator-0               1/1     Running     0          11m
aptos-node-29-validator-0               1/1     Running     0          11m
aptos-node-3-validator-0                1/1     Running     0          11m
aptos-node-30-validator-0               1/1     Running     0          11m
aptos-node-31-validator-0               1/1     Running     0          11m
aptos-node-32-validator-0               1/1     Running     0          11m
aptos-node-33-validator-0               1/1     Running     0          11m
aptos-node-34-validator-0               1/1     Running     0          11m
aptos-node-35-validator-0               1/1     Running     0          11m
aptos-node-36-validator-0               1/1     Running     0          11m
aptos-node-37-validator-0               1/1     Running     0          11m
aptos-node-38-validator-0               1/1     Running     0          11m
aptos-node-39-validator-0               1/1     Running     0          11m
aptos-node-4-validator-0                1/1     Running     0          11m
aptos-node-40-validator-0               1/1     Running     0          11m
aptos-node-41-validator-0               1/1     Running     0          11m
aptos-node-42-validator-0               1/1     Running     0          11m
aptos-node-43-validator-0               1/1     Running     0          11m
aptos-node-44-validator-0               1/1     Running     0          11m
aptos-node-45-validator-0               1/1     Running     0          11m
aptos-node-46-validator-0               1/1     Running     0          11m
aptos-node-47-validator-0               1/1     Running     0          11m
aptos-node-48-validator-0               1/1     Running     0          11m
aptos-node-49-validator-0               1/1     Running     0          11m
aptos-node-5-validator-0                1/1     Running     0          11m
aptos-node-50-validator-0               1/1     Running     0          11m
aptos-node-51-validator-0               1/1     Running     0          11m
aptos-node-52-validator-0               1/1     Running     0          11m
aptos-node-53-validator-0               1/1     Running     0          11m
aptos-node-54-validator-0               1/1     Running     0          11m
aptos-node-55-validator-0               1/1     Running     0          11m
aptos-node-56-validator-0               1/1     Running     0          11m
aptos-node-57-validator-0               1/1     Running     0          11m
aptos-node-58-validator-0               1/1     Running     0          11m
aptos-node-59-validator-0               1/1     Running     0          11m
aptos-node-6-validator-0                1/1     Running     0          11m
aptos-node-60-validator-0               1/1     Running     0          11m
aptos-node-61-validator-0               1/1     Running     0          11m
aptos-node-62-validator-0               1/1     Running     0          11m
aptos-node-63-validator-0               1/1     Running     0          11m
aptos-node-64-validator-0               1/1     Running     0          11m
aptos-node-65-validator-0               1/1     Running     0          11m
aptos-node-66-validator-0               1/1     Running     0          11m
aptos-node-67-validator-0               1/1     Running     0          11m
aptos-node-68-validator-0               1/1     Running     0          11m
aptos-node-69-validator-0               1/1     Running     0          11m
aptos-node-7-validator-0                1/1     Running     0          11m
aptos-node-70-validator-0               1/1     Running     0          11m
aptos-node-71-validator-0               1/1     Running     0          11m
aptos-node-72-validator-0               1/1     Running     0          11m
aptos-node-73-validator-0               1/1     Running     0          11m
aptos-node-74-validator-0               1/1     Running     0          11m
aptos-node-75-validator-0               1/1     Running     0          11m
aptos-node-76-validator-0               1/1     Running     0          11m
aptos-node-77-validator-0               1/1     Running     0          11m
aptos-node-78-validator-0               1/1     Running     0          11m
aptos-node-79-validator-0               1/1     Running     0          11m
aptos-node-8-validator-0                1/1     Running     0          11m
aptos-node-80-validator-0               1/1     Running     0          11m
aptos-node-81-validator-0               1/1     Running     0          11m
aptos-node-82-validator-0               1/1     Running     0          11m
aptos-node-83-validator-0               1/1     Running     0          11m
aptos-node-84-validator-0               1/1     Running     0          11m
aptos-node-85-validator-0               1/1     Running     0          11m
aptos-node-86-validator-0               1/1     Running     0          11m
aptos-node-87-validator-0               1/1     Running     0          11m
aptos-node-88-validator-0               1/1     Running     0          11m
aptos-node-89-validator-0               1/1     Running     0          11m
aptos-node-9-validator-0                1/1     Running     0          11m
aptos-node-90-validator-0               1/1     Running     0          11m
aptos-node-91-validator-0               1/1     Running     0          11m
aptos-node-92-validator-0               1/1     Running     0          11m
aptos-node-93-validator-0               1/1     Running     0          11m
aptos-node-94-validator-0               1/1     Running     0          11m
aptos-node-95-validator-0               1/1     Running     0          11m
aptos-node-96-validator-0               1/1     Running     0          11m
aptos-node-97-validator-0               1/1     Running     0          11m
aptos-node-98-validator-0               1/1     Running     0          11m
aptos-node-99-validator-0               1/1     Running     0          11m
genesis-aptos-genesis-eforge170-4xx77   0/1     Completed   0          13m

Grafana dashboard
Humio Logs
Test runner output
Test run is land-blocking

github-actions[bot] commented 11 months ago

:x: Forge suite `compat` failure on `aptos-node-v1.4.4` ==> `fa1935f9083286b3c8f57f1bccb9881c9708c0ff`

Compatibility test results for aptos-node-v1.4.4 ==> fa1935f9083286b3c8f57f1bccb9881c9708c0ff (PR)
1. Check liveness of validators at old version: aptos-node-v1.4.4
compatibility::simple-validator-upgrade::liveness-check : committed: 6675 txn/s, latency: 4782 ms, (p50: 4800 ms, p90: 6200 ms, p99: 7800 ms), latency samples: 240320
2. Upgrading first Validator to new version: fa1935f9083286b3c8f57f1bccb9881c9708c0ff
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1052 txn/s, latency: 32369 ms, (p50: 28000 ms, p90: 49600 ms, p99: 49700 ms), latency samples: 56840
3. Upgrading rest of first batch to new version: fa1935f9083286b3c8f57f1bccb9881c9708c0ff
Test Failed: Tried executing 10 txns, request counters: "success 0, failed submit [0], failed wait [10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10], by client: [(0, 0, 180): http://aptos-node-0-validator.forge-compat-pr-8994.svc:8080/v1/]"

Caused by:
    Unknown error Ledger on endpoint (http://aptos-node-0-validator.forge-compat-pr-8994.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (fc77d586) might still succeed.

Stack backtrace:
   0: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/result.rs:2002:27
      aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiReliableTransactionSubmitter::submit_check_and_retry::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:119:28
   1: <futures_util::future::maybe_done::MaybeDone<Fut> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.24/src/future/maybe_done.rs:95:38
      <futures_util::future::join_all::JoinAll<F> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.24/src/future/join_all.rs:142:24
   2: <aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiReliableTransactionSubmitter as aptos_transaction_generator_lib::ReliableTransactionSubmitter>::execute_transactions_with_counter::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:273:9
   3: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/future/future.rs:125:9
      aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_and_fund_seed_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:314:17
      aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:187:13
   4: aptos_transaction_emitter_lib::emitter::TxnEmitter::start_job::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:608:13
   5: aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for_impl::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:712:13
      aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:744:13
      tokio::park::thread::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/park/thread.rs:267:54
      tokio::coop::with_budget::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:102:9
      std::thread::local::LocalKey<T>::try_with
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/thread/local.rs:252:16
      std::thread::local::LocalKey<T>::with
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/thread/local.rs:228:9
   6: tokio::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:95:5
      tokio::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:72:5
      tokio::park::thread::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/park/thread.rs:267:31
   7: tokio::runtime::enter::Enter::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/enter.rs:152:13
      tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/scheduler/multi_thread/mod.rs:79:9
   8: tokio::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/mod.rs:492:44
   9: aptos_testcases::generate_traffic
             at ./testsuite/testcases/src/lib.rs:99:17
  10: <aptos_testcases::compatibility_test::SimpleValidatorUpgrade as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/compatibility_test.rs:93:24
  11: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:482:42
      aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:550:11
      aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:482:30
  12: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:380:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:306:21
  13: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
  14: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
  15: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  16: main
  17: __libc_start_main
  18: _start
Trailing Log Lines:
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  16: main
  17: __libc_start_main
  18: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:291"},"thread_name":"main","hostname":"forge-compat-pr-8994-1688766450-aptos-node-v1-4-4","timestamp":"2023-07-07T22:01:52.343331Z","message":"Deleting namespace forge-compat-pr-8994: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:399"},"thread_name":"main","hostname":"forge-compat-pr-8994-1688766450-aptos-node-v1-4-4","timestamp":"2023-07-07T22:01:52.343367Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-8994"}
Failed to run tests:
Tests Failed

failures:
    compatibility::simple-validator-upgrade

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Error: Tests Failed

Stack backtrace:
   0: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:502:13
   1: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:380:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:306:21
   2: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
   3: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
   4: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
   5: main
   6: __libc_start_main
   7: _start
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-validator-0                1/1     Running     0          4m49s
aptos-node-1-validator-0                1/1     Running     0          9m49s
aptos-node-2-validator-0                1/1     Running     0          11m
aptos-node-3-validator-0                1/1     Running     0          11m
aptos-node-4-validator-0                1/1     Running     0          11m
genesis-aptos-genesis-eforge160-n8kgd   0/1     Completed   0          14m

Grafana dashboard
Humio Logs
Test runner output
Test run is land-blocking

github-actions[bot] commented 11 months ago

:white_check_mark: Forge suite `framework_upgrade` success on `aptos-node-v1.4.4` ==> `fa1935f9083286b3c8f57f1bccb9881c9708c0ff`

Compatibility test results for aptos-node-v1.4.4 ==> fa1935f9083286b3c8f57f1bccb9881c9708c0ff (PR)
Upgrade the nodes to version: fa1935f9083286b3c8f57f1bccb9881c9708c0ff
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 4262 txn/s, latency: 7130 ms, (p50: 6600 ms, p90: 10100 ms, p99: 21900 ms), latency samples: 174760
5. check swarm health
Compatibility test for aptos-node-v1.4.4 ==> fa1935f9083286b3c8f57f1bccb9881c9708c0ff passed
Test Ok

Grafana dashboard
Humio Logs
Test runner output
Test run is land-blocking

github-actions[bot] commented 11 months ago

Forge is running suite `framework_upgrade` on `aptos-node-v1.4.4` ==> `ef09d22d83c12ea0655720822d24e7f1c29b7958`

github-actions[bot] commented 11 months ago

Forge is running suite `realistic_env_max_load` on `ef09d22d83c12ea0655720822d24e7f1c29b7958`

github-actions[bot] commented 11 months ago

Forge is running suite `compat` on `aptos-node-v1.4.4` ==> `ef09d22d83c12ea0655720822d24e7f1c29b7958`

github-actions[bot] commented 11 months ago

:x: Forge suite `compat` failure on `aptos-node-v1.4.4` ==> `ef09d22d83c12ea0655720822d24e7f1c29b7958`

Compatibility test results for aptos-node-v1.4.4 ==> ef09d22d83c12ea0655720822d24e7f1c29b7958 (PR)
1. Check liveness of validators at old version: aptos-node-v1.4.4
compatibility::simple-validator-upgrade::liveness-check : committed: 7379 txn/s, latency: 4492 ms, (p50: 4500 ms, p90: 6200 ms, p99: 7500 ms), latency samples: 258280
2. Upgrading first Validator to new version: ef09d22d83c12ea0655720822d24e7f1c29b7958
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1524 txn/s, latency: 22647 ms, (p50: 25300 ms, p90: 34800 ms, p99: 34900 ms), latency samples: 64020
3. Upgrading rest of first batch to new version: ef09d22d83c12ea0655720822d24e7f1c29b7958
Test Failed: Tried executing 10 txns, request counters: "success 0, failed submit [0], failed wait [10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10], by client: [(0, 0, 180): http://aptos-node-0-validator.forge-compat-pr-8994.svc:8080/v1/]"

Caused by:
    Unknown error Ledger on endpoint (http://aptos-node-0-validator.forge-compat-pr-8994.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (c22eff6d) might still succeed.

Stack backtrace:
   0: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/result.rs:2002:27
      aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiReliableTransactionSubmitter::submit_check_and_retry::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:119:28
   1: <futures_util::future::maybe_done::MaybeDone<Fut> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.24/src/future/maybe_done.rs:95:38
      <futures_util::future::join_all::JoinAll<F> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.24/src/future/join_all.rs:142:24
   2: <aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiReliableTransactionSubmitter as aptos_transaction_generator_lib::ReliableTransactionSubmitter>::execute_transactions_with_counter::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:273:9
   3: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/future/future.rs:125:9
      aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_and_fund_seed_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:314:17
      aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:187:13
   4: aptos_transaction_emitter_lib::emitter::TxnEmitter::start_job::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:608:13
   5: aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for_impl::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:712:13
      aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:744:13
      tokio::park::thread::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/park/thread.rs:267:54
      tokio::coop::with_budget::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:102:9
      std::thread::local::LocalKey<T>::try_with
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/thread/local.rs:252:16
      std::thread::local::LocalKey<T>::with
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/thread/local.rs:228:9
   6: tokio::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:95:5
      tokio::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:72:5
      tokio::park::thread::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/park/thread.rs:267:31
   7: tokio::runtime::enter::Enter::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/enter.rs:152:13
      tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/scheduler/multi_thread/mod.rs:79:9
   8: tokio::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/mod.rs:492:44
   9: aptos_testcases::generate_traffic
             at ./testsuite/testcases/src/lib.rs:99:17
  10: <aptos_testcases::compatibility_test::SimpleValidatorUpgrade as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/compatibility_test.rs:93:24
  11: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:482:42
      aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:550:11
      aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:482:30
  12: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:380:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:306:21
  13: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
  14: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
  15: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  16: main
  17: __libc_start_main
  18: _start
Trailing Log Lines:
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  16: main
  17: __libc_start_main
  18: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:291"},"thread_name":"main","hostname":"forge-compat-pr-8994-1688772842-aptos-node-v1-4-4","timestamp":"2023-07-07T23:49:41.299285Z","message":"Deleting namespace forge-compat-pr-8994: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:399"},"thread_name":"main","hostname":"forge-compat-pr-8994-1688772842-aptos-node-v1-4-4","timestamp":"2023-07-07T23:49:41.299318Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-8994"}
Failed to run tests:
Tests Failed

failures:
    compatibility::simple-validator-upgrade

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Error: Tests Failed

Stack backtrace:
   0: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:502:13
   1: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:380:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:306:21
   2: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
   3: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
   4: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
   5: main
   6: __libc_start_main
   7: _start
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-validator-0                1/1     Running     0          4m50s
aptos-node-1-validator-0                1/1     Running     0          10m
aptos-node-2-validator-0                1/1     Running     0          14m
aptos-node-3-validator-0                1/1     Running     0          14m
aptos-node-4-validator-0                1/1     Running     0          14m
genesis-aptos-genesis-eforge104-s6p4j   0/1     Completed   0          15m

Grafana dashboard
Humio Logs
Test runner output
Test run is land-blocking

github-actions[bot] commented 11 months ago

:white_check_mark: Forge suite `framework_upgrade` success on `aptos-node-v1.4.4` ==> `ef09d22d83c12ea0655720822d24e7f1c29b7958`

Compatibility test results for aptos-node-v1.4.4 ==> ef09d22d83c12ea0655720822d24e7f1c29b7958 (PR)
Upgrade the nodes to version: ef09d22d83c12ea0655720822d24e7f1c29b7958
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 4819 txn/s, latency: 6804 ms, (p50: 7100 ms, p90: 9600 ms, p99: 11100 ms), latency samples: 178320
5. check swarm health
Compatibility test for aptos-node-v1.4.4 ==> ef09d22d83c12ea0655720822d24e7f1c29b7958 passed
Test Ok

Grafana dashboard
Humio Logs
Test runner output
Test run is land-blocking

github-actions[bot] commented 11 months ago

:x: Forge suite `realistic_env_max_load` failure on `ef09d22d83c12ea0655720822d24e7f1c29b7958`

two traffics test: inner traffic : committed: 4000 txn/s, submitted: 4031 txn/s, expired: 31 txn/s, latency: 46059 ms, (p50: 49500 ms, p90: 55900 ms, p99: 61300 ms), latency samples: 1952090
Test Failed: test NetworkLoadTest

Caused by:
    TPS requirement for inner traffic failed. Average TPS 4000, minimum TPS requirement 5000. Full stats: committed: 4000 txn/s, submitted: 4031 txn/s, expired: 31 txn/s, latency: 46059 ms, (p50: 49500 ms, p90: 55900 ms, p99: 61300 ms), latency samples: 1952090

Stack backtrace:
   0: aptos_forge::success_criteria::SuccessCriteriaChecker::check_tps
             at ./testsuite/forge/src/success_criteria.rs:288:13
   1: aptos_forge::success_criteria::SuccessCriteriaChecker::check_throughput
             at ./testsuite/forge/src/success_criteria.rs:341:9
      aptos_forge::success_criteria::SuccessCriteriaChecker::check_core_for_success
             at ./testsuite/forge/src/success_criteria.rs:101:9
   2: <aptos_testcases::two_traffics_test::TwoTrafficsTest as aptos_testcases::NetworkLoadTest>::test
             at ./testsuite/testcases/src/two_traffics_test.rs:69:9
   3: <dyn aptos_testcases::NetworkLoadTest>::network_load_test
             at ./testsuite/testcases/src/lib.rs:283:13
   4: <dyn aptos_testcases::NetworkLoadTest as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/lib.rs:174:87
   5: <aptos_testcases::CompositeNetworkTest as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/lib.rs:396:9
   6: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:482:42
      aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:550:11
      aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:482:30
   7: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:380:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:306:21
   8: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
   9: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
  10: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  11: main
  12: __libc_start_main
  13: _start
Trailing Log Lines:
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  11: main
  12: __libc_start_main
  13: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:291"},"thread_name":"main","hostname":"forge-e2e-pr-8994-1688772823-ef09d22d83c12ea0655720822d24e7f1c2","timestamp":"2023-07-07T23:51:19.134986Z","message":"Deleting namespace forge-e2e-pr-8994: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:399"},"thread_name":"main","hostname":"forge-e2e-pr-8994-1688772823-ef09d22d83c12ea0655720822d24e7f1c2","timestamp":"2023-07-07T23:51:19.135017Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-8994"}

failures:
    CompositeNetworkTest
Failed to run tests:
Tests Failed

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Error: Tests Failed

Stack backtrace:
   0: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:502:13
   1: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:380:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:306:21
   2: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
   3: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
   4: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
   5: main
   6: __libc_start_main
   7: _start
Debugging output:

Grafana dashboard
Humio Logs
Test runner output
Test run is land-blocking

github-actions[bot] commented 11 months ago

Forge is running suite `framework_upgrade` on `aptos-node-v1.4.4` ==> `9969b85afcf11267bcecabeac4f55e4227c22a07`

github-actions[bot] commented 11 months ago

Forge is running suite `realistic_env_max_load` on `9969b85afcf11267bcecabeac4f55e4227c22a07`

github-actions[bot] commented 11 months ago

Forge is running suite `compat` on `aptos-node-v1.4.4` ==> `9969b85afcf11267bcecabeac4f55e4227c22a07`

github-actions[bot] commented 11 months ago

:white_check_mark: Forge suite `realistic_env_max_load` success on `9969b85afcf11267bcecabeac4f55e4227c22a07`

network:multi-region-network-emulation : committed: 8042 txn/s, submitted: 8087 txn/s, expired: 45 txn/s, latency: 18315 ms, (p50: 14500 ms, p90: 27400 ms, p99: 71300 ms), latency samples: 3434161
Max round gap was 1 [limit 4] at version 3964279. Max no progress secs was 3.38132 [limit 10] at version 3964279.
Test Ok

Grafana dashboard
Humio Logs
Test runner output
Test run is land-blocking

github-actions[bot] commented 11 months ago

:x: Forge suite `compat` failure on `aptos-node-v1.4.4` ==> `9969b85afcf11267bcecabeac4f55e4227c22a07`

Compatibility test results for aptos-node-v1.4.4 ==> 9969b85afcf11267bcecabeac4f55e4227c22a07 (PR)
1. Check liveness of validators at old version: aptos-node-v1.4.4
compatibility::simple-validator-upgrade::liveness-check : committed: 6806 txn/s, latency: 4708 ms, (p50: 4600 ms, p90: 6800 ms, p99: 8100 ms), latency samples: 251840
2. Upgrading first Validator to new version: 9969b85afcf11267bcecabeac4f55e4227c22a07
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1233 txn/s, latency: 18391 ms, (p50: 19700 ms, p90: 24700 ms, p99: 25400 ms), latency samples: 76480
3. Upgrading rest of first batch to new version: 9969b85afcf11267bcecabeac4f55e4227c22a07
Test Failed: Tried executing 10 txns, request counters: "success 0, failed submit [0], failed wait [10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10], by client: [(0, 0, 180): http://aptos-node-0-validator.forge-compat-pr-8994.svc:8080/v1/]"

Caused by:
    Unknown error Ledger on endpoint (http://aptos-node-0-validator.forge-compat-pr-8994.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (5bca612a) might still succeed.

Stack backtrace:
   0: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/result.rs:2002:27
      aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiReliableTransactionSubmitter::submit_check_and_retry::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:119:28
   1: <futures_util::future::maybe_done::MaybeDone<Fut> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.24/src/future/maybe_done.rs:95:38
      <futures_util::future::join_all::JoinAll<F> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.24/src/future/join_all.rs:142:24
   2: <aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiReliableTransactionSubmitter as aptos_transaction_generator_lib::ReliableTransactionSubmitter>::execute_transactions_with_counter::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:273:9
   3: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/future/future.rs:125:9
      aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_and_fund_seed_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:314:17
      aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:187:13
   4: aptos_transaction_emitter_lib::emitter::TxnEmitter::start_job::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:608:13
   5: aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for_impl::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:712:13
      aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:744:13
      tokio::park::thread::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/park/thread.rs:267:54
      tokio::coop::with_budget::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:102:9
      std::thread::local::LocalKey<T>::try_with
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/thread/local.rs:252:16
      std::thread::local::LocalKey<T>::with
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/thread/local.rs:228:9
   6: tokio::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:95:5
      tokio::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:72:5
      tokio::park::thread::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/park/thread.rs:267:31
   7: tokio::runtime::enter::Enter::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/enter.rs:152:13
      tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/scheduler/multi_thread/mod.rs:79:9
   8: tokio::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/mod.rs:492:44
   9: aptos_testcases::generate_traffic
             at ./testsuite/testcases/src/lib.rs:99:17
  10: <aptos_testcases::compatibility_test::SimpleValidatorUpgrade as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/compatibility_test.rs:93:24
  11: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:482:42
      aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:550:11
      aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:482:30
  12: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:380:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:306:21
  13: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
  14: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
  15: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  16: main
  17: __libc_start_main
  18: _start
Trailing Log Lines:
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  16: main
  17: __libc_start_main
  18: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:291"},"thread_name":"main","hostname":"forge-compat-pr-8994-1688777820-aptos-node-v1-4-4","timestamp":"2023-07-08T01:12:08.493188Z","message":"Deleting namespace forge-compat-pr-8994: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:399"},"thread_name":"main","hostname":"forge-compat-pr-8994-1688777820-aptos-node-v1-4-4","timestamp":"2023-07-08T01:12:08.493220Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-8994"}
Failed to run tests:
Tests Failed

failures:
    compatibility::simple-validator-upgrade

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Error: Tests Failed

Stack backtrace:
   0: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:502:13
   1: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:380:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:306:21
   2: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
   3: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
   4: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
   5: main
   6: __libc_start_main
   7: _start
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-validator-0                1/1     Running     0          4m47s
aptos-node-1-validator-0                1/1     Running     0          9m47s
aptos-node-2-validator-0                1/1     Running     0          13m
aptos-node-3-validator-0                1/1     Running     0          13m
aptos-node-4-validator-0                1/1     Running     0          13m
genesis-aptos-genesis-eforge170-p59p2   0/1     Completed   0          13m

Grafana dashboard
Humio Logs
Test runner output
Test run is land-blocking

github-actions[bot] commented 11 months ago

:white_check_mark: Forge suite `framework_upgrade` success on `aptos-node-v1.4.4` ==> `9969b85afcf11267bcecabeac4f55e4227c22a07`

Compatibility test results for aptos-node-v1.4.4 ==> 9969b85afcf11267bcecabeac4f55e4227c22a07 (PR)
Upgrade the nodes to version: 9969b85afcf11267bcecabeac4f55e4227c22a07
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 3292 txn/s, latency: 6649 ms, (p50: 6300 ms, p90: 9600 ms, p99: 14700 ms), latency samples: 181100
5. check swarm health
Compatibility test for aptos-node-v1.4.4 ==> 9969b85afcf11267bcecabeac4f55e4227c22a07 passed
Test Ok

Grafana dashboard
Humio Logs
Test runner output
Test run is land-blocking

github-actions[bot] commented 11 months ago

Forge is running suite `realistic_env_max_load` on `3d0127e788868e635bf7c389e9d62ced896ef52a`

github-actions[bot] commented 11 months ago

Forge is running suite `compat` on `aptos-node-v1.4.4` ==> `3d0127e788868e635bf7c389e9d62ced896ef52a`

github-actions[bot] commented 11 months ago

Forge is running suite `framework_upgrade` on `aptos-node-v1.4.4` ==> `3d0127e788868e635bf7c389e9d62ced896ef52a`

github-actions[bot] commented 11 months ago

:x: Forge suite `framework_upgrade` failure on `aptos-node-v1.4.4` ==> `3d0127e788868e635bf7c389e9d62ced896ef52a`

Compatibility test results for aptos-node-v1.4.4 ==> 3d0127e788868e635bf7c389e9d62ced896ef52a (PR)
Upgrade the nodes to version: 3d0127e788868e635bf7c389e9d62ced896ef52a
Test Failed: Waiting for nodes to catch up to target version and epoch (Some(490), None) timed out after 10 seconds, current status: [("validator-0", 370, 4), ("validator-1", 490, 5), ("validator-2", 490, 5), ("validator-3", 488, 5), ("validator-4", 488, 5)]

Stack backtrace:
   0: aptos_forge::interface::swarm::wait_for_all_nodes_to_catchup_to_target_version_or_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:451:24
   1: aptos_forge::interface::swarm::wait_for_all_nodes_to_catchup_to_version::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:368:5
   2: aptos_forge::interface::swarm::SwarmExt::wait_for_all_nodes_to_catchup_to_version::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:254:9
   3: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/future/future.rs:125:9
      tokio::park::thread::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/park/thread.rs:267:54
      tokio::coop::with_budget::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:102:9
      std::thread::local::LocalKey<T>::try_with
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/thread/local.rs:252:16
      std::thread::local::LocalKey<T>::with
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/thread/local.rs:228:9
      tokio::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:95:5
      tokio::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:72:5
      tokio::park::thread::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/park/thread.rs:267:31
   4: tokio::runtime::enter::Enter::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/enter.rs:152:13
      tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/scheduler/multi_thread/mod.rs:79:9
   5: tokio::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/mod.rs:492:44
   6: aptos_forge::interface::swarm::SwarmExt::fork_check
             at ./testsuite/forge/src/interface/swarm.rs:232:9
   7: <aptos_testcases::framework_upgrade::FrameworkUpgrade as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/framework_upgrade.rs:58:9
   8: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:482:42
      aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:550:11
      aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:482:30
   9: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:380:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:306:21
  10: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
  11: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
  12: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  13: main
  14: __libc_start_main
  15: _start
Trailing Log Lines:
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  13: main
  14: __libc_start_main
  15: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:291"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-8994-1688780289-aptos-node-v1-4-4","timestamp":"2023-07-08T01:45:45.918522Z","message":"Deleting namespace forge-framework-upgrade-pr-8994: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:399"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-8994-1688780289-aptos-node-v1-4-4","timestamp":"2023-07-08T01:45:45.918552Z","message":"aptos-node resources for Forge removed in namespace: forge-framework-upgrade-pr-8994"}

failures:
    framework_upgrade::framework-upgrade

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed

Stack backtrace:
   0: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:502:13
   1: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:380:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:306:21
   2: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
   3: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
   4: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
   5: main
   6: __libc_start_main
   7: _start
Debugging output:
NAME                                   READY   STATUS      RESTARTS   AGE
aptos-node-0-validator-0               1/1     Running     0          5m53s
aptos-node-1-validator-0               1/1     Running     0          4m30s
aptos-node-2-validator-0               1/1     Running     0          3m14s
aptos-node-3-validator-0               1/1     Running     0          2m4s
aptos-node-4-validator-0               1/1     Running     0          64s
genesis-aptos-genesis-eforge16-6rlz9   0/1     Completed   0          7m15s

Grafana dashboard
Humio Logs
Test runner output
Test run is land-blocking

github-actions[bot] commented 11 months ago

:x: Forge suite `compat` failure on `aptos-node-v1.4.4` ==> `3d0127e788868e635bf7c389e9d62ced896ef52a`

Compatibility test results for aptos-node-v1.4.4 ==> 3d0127e788868e635bf7c389e9d62ced896ef52a (PR)
1. Check liveness of validators at old version: aptos-node-v1.4.4
compatibility::simple-validator-upgrade::liveness-check : committed: 7238 txn/s, latency: 4599 ms, (p50: 4600 ms, p90: 6300 ms, p99: 7300 ms), latency samples: 246120
2. Upgrading first Validator to new version: 3d0127e788868e635bf7c389e9d62ced896ef52a
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1616 txn/s, latency: 21467 ms, (p50: 19700 ms, p90: 28800 ms, p99: 28800 ms), latency samples: 53340
3. Upgrading rest of first batch to new version: 3d0127e788868e635bf7c389e9d62ced896ef52a
Test Failed: Tried executing 10 txns, request counters: "success 0, failed submit [0], failed wait [10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10], by client: [(0, 0, 180): http://aptos-node-0-validator.forge-compat-pr-8994.svc:8080/v1/]"

Caused by:
    Unknown error Ledger on endpoint (http://aptos-node-0-validator.forge-compat-pr-8994.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (3bab7184) might still succeed.

Stack backtrace:
   0: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/result.rs:2002:27
      aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiReliableTransactionSubmitter::submit_check_and_retry::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:119:28
   1: <futures_util::future::maybe_done::MaybeDone<Fut> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.24/src/future/maybe_done.rs:95:38
      <futures_util::future::join_all::JoinAll<F> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.24/src/future/join_all.rs:142:24
   2: <aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiReliableTransactionSubmitter as aptos_transaction_generator_lib::ReliableTransactionSubmitter>::execute_transactions_with_counter::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:273:9
   3: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/future/future.rs:125:9
      aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_and_fund_seed_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:314:17
      aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:187:13
   4: aptos_transaction_emitter_lib::emitter::TxnEmitter::start_job::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:608:13
   5: aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for_impl::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:712:13
      aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:744:13
      tokio::park::thread::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/park/thread.rs:267:54
      tokio::coop::with_budget::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:102:9
      std::thread::local::LocalKey<T>::try_with
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/thread/local.rs:252:16
      std::thread::local::LocalKey<T>::with
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/thread/local.rs:228:9
   6: tokio::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:95:5
      tokio::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:72:5
      tokio::park::thread::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/park/thread.rs:267:31
   7: tokio::runtime::enter::Enter::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/enter.rs:152:13
      tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/scheduler/multi_thread/mod.rs:79:9
   8: tokio::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/mod.rs:492:44
   9: aptos_testcases::generate_traffic
             at ./testsuite/testcases/src/lib.rs:99:17
  10: <aptos_testcases::compatibility_test::SimpleValidatorUpgrade as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/compatibility_test.rs:93:24
  11: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:482:42
      aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:550:11
      aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:482:30
  12: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:380:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:306:21
  13: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
  14: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
  15: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  16: main
  17: __libc_start_main
  18: _start
Trailing Log Lines:
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  16: main
  17: __libc_start_main
  18: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:291"},"thread_name":"main","hostname":"forge-compat-pr-8994-1688780272-aptos-node-v1-4-4","timestamp":"2023-07-08T01:50:28.142911Z","message":"Deleting namespace forge-compat-pr-8994: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:399"},"thread_name":"main","hostname":"forge-compat-pr-8994-1688780272-aptos-node-v1-4-4","timestamp":"2023-07-08T01:50:28.142935Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-8994"}

failures:
    compatibility::simple-validator-upgrade

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed

Stack backtrace:
   0: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:502:13
   1: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:380:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:306:21
   2: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
   3: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
   4: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
   5: main
   6: __libc_start_main
   7: _start
Debugging output:
NAME                                   READY   STATUS      RESTARTS   AGE
aptos-node-0-validator-0               1/1     Running     0          4m43s
aptos-node-1-validator-0               1/1     Running     0          8m45s
aptos-node-2-validator-0               1/1     Running     0          12m
aptos-node-3-validator-0               1/1     Running     0          12m
aptos-node-4-validator-0               1/1     Running     0          12m
genesis-aptos-genesis-eforge16-wv5ct   0/1     Completed   0          12m

Grafana dashboard
Humio Logs
Test runner output
Test run is land-blocking

github-actions[bot] commented 11 months ago

:x: Forge suite `realistic_env_max_load` failure on `3d0127e788868e635bf7c389e9d62ced896ef52a`

network:multi-region-network-emulation : committed: 6694 txn/s, latency: 22335 ms, (p50: 22000 ms, p90: 24100 ms, p99: 28600 ms), latency samples: 2858420
Test Failed: check for success

Caused by:
    TPS requirement failed. Average TPS 6694, minimum TPS requirement 8000. Full stats: committed: 6694 txn/s, latency: 22335 ms, (p50: 22000 ms, p90: 24100 ms, p99: 28600 ms), latency samples: 2858420

Stack backtrace:
   0: aptos_forge::success_criteria::SuccessCriteriaChecker::check_tps
             at ./testsuite/forge/src/success_criteria.rs:288:13
   1: aptos_forge::success_criteria::SuccessCriteriaChecker::check_throughput
             at ./testsuite/forge/src/success_criteria.rs:341:9
      aptos_forge::success_criteria::SuccessCriteriaChecker::check_for_success::{{closure}}
             at ./testsuite/forge/src/success_criteria.rs:134:9
      tokio::park::thread::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/park/thread.rs:267:54
      tokio::coop::with_budget::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:102:9
      std::thread::local::LocalKey<T>::try_with
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/thread/local.rs:252:16
      std::thread::local::LocalKey<T>::with
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/thread/local.rs:228:9
      tokio::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:95:5
      tokio::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:72:5
      tokio::park::thread::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/park/thread.rs:267:31
   2: tokio::runtime::enter::Enter::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/enter.rs:152:13
      tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/scheduler/multi_thread/mod.rs:79:9
   3: tokio::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/mod.rs:492:44
   4: aptos_forge::interface::network::NetworkContext::check_for_success
             at ./testsuite/forge/src/interface/network.rs:69:9
   5: <dyn aptos_testcases::NetworkLoadTest as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/lib.rs:197:9
   6: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:482:42
      aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:550:11
      aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:482:30
   7: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:380:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:306:21
   8: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
   9: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
  10: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  11: main
  12: __libc_start_main
  13: _start
Trailing Log Lines:
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  11: main
  12: __libc_start_main
  13: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:291"},"thread_name":"main","hostname":"forge-e2e-pr-8994-1688780252-3d0127e788868e635bf7c389e9d62ced89","timestamp":"2023-07-08T01:54:11.116453Z","message":"Deleting namespace forge-e2e-pr-8994: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:399"},"thread_name":"main","hostname":"forge-e2e-pr-8994-1688780252-3d0127e788868e635bf7c389e9d62ced89","timestamp":"2023-07-08T01:54:11.116495Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-8994"}

failures:
    network:multi-region-network-emulation

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed

Stack backtrace:
   0: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:502:13
   1: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:380:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:306:21
   2: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
   3: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
   4: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
   5: main
   6: __libc_start_main
   7: _start
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-fullnode-eforge229-0       1/1     Running     0          13m
aptos-node-0-validator-0                1/1     Running     0          13m
aptos-node-1-fullnode-eforge229-0       1/1     Running     0          13m
aptos-node-1-validator-0                1/1     Running     0          13m
aptos-node-10-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-10-validator-0               1/1     Running     0          13m
aptos-node-11-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-11-validator-0               1/1     Running     0          13m
aptos-node-12-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-12-validator-0               1/1     Running     0          13m
aptos-node-13-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-13-validator-0               1/1     Running     0          13m
aptos-node-14-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-14-validator-0               1/1     Running     0          13m
aptos-node-15-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-15-validator-0               1/1     Running     0          13m
aptos-node-16-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-16-validator-0               1/1     Running     0          13m
aptos-node-17-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-17-validator-0               1/1     Running     0          13m
aptos-node-18-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-18-validator-0               1/1     Running     0          13m
aptos-node-19-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-19-validator-0               1/1     Running     0          13m
aptos-node-2-fullnode-eforge229-0       1/1     Running     0          13m
aptos-node-2-validator-0                1/1     Running     0          13m
aptos-node-20-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-20-validator-0               1/1     Running     0          13m
aptos-node-21-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-21-validator-0               1/1     Running     0          13m
aptos-node-22-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-22-validator-0               1/1     Running     0          13m
aptos-node-23-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-23-validator-0               1/1     Running     0          13m
aptos-node-24-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-24-validator-0               1/1     Running     0          13m
aptos-node-25-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-25-validator-0               1/1     Running     0          13m
aptos-node-26-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-26-validator-0               1/1     Running     0          13m
aptos-node-27-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-27-validator-0               1/1     Running     0          13m
aptos-node-28-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-28-validator-0               1/1     Running     0          13m
aptos-node-29-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-29-validator-0               1/1     Running     0          13m
aptos-node-3-fullnode-eforge229-0       1/1     Running     0          13m
aptos-node-3-validator-0                1/1     Running     0          13m
aptos-node-30-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-30-validator-0               1/1     Running     0          13m
aptos-node-31-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-31-validator-0               1/1     Running     0          13m
aptos-node-32-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-32-validator-0               1/1     Running     0          13m
aptos-node-33-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-33-validator-0               1/1     Running     0          13m
aptos-node-34-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-34-validator-0               1/1     Running     0          13m
aptos-node-35-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-35-validator-0               1/1     Running     0          13m
aptos-node-36-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-36-validator-0               1/1     Running     0          13m
aptos-node-37-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-37-validator-0               1/1     Running     0          13m
aptos-node-38-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-38-validator-0               1/1     Running     0          13m
aptos-node-39-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-39-validator-0               1/1     Running     0          13m
aptos-node-4-fullnode-eforge229-0       1/1     Running     0          13m
aptos-node-4-validator-0                1/1     Running     0          13m
aptos-node-40-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-40-validator-0               1/1     Running     0          13m
aptos-node-41-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-41-validator-0               1/1     Running     0          13m
aptos-node-42-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-42-validator-0               1/1     Running     0          13m
aptos-node-43-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-43-validator-0               1/1     Running     0          13m
aptos-node-44-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-44-validator-0               1/1     Running     0          13m
aptos-node-45-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-45-validator-0               1/1     Running     0          13m
aptos-node-46-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-46-validator-0               1/1     Running     0          13m
aptos-node-47-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-47-validator-0               1/1     Running     0          13m
aptos-node-48-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-48-validator-0               1/1     Running     0          13m
aptos-node-49-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-49-validator-0               1/1     Running     0          13m
aptos-node-5-fullnode-eforge229-0       1/1     Running     0          13m
aptos-node-5-validator-0                1/1     Running     0          13m
aptos-node-50-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-50-validator-0               1/1     Running     0          13m
aptos-node-51-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-51-validator-0               1/1     Running     0          13m
aptos-node-52-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-52-validator-0               1/1     Running     0          13m
aptos-node-53-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-53-validator-0               1/1     Running     0          13m
aptos-node-54-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-54-validator-0               1/1     Running     0          13m
aptos-node-55-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-55-validator-0               1/1     Running     0          13m
aptos-node-56-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-56-validator-0               1/1     Running     0          13m
aptos-node-57-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-57-validator-0               1/1     Running     0          13m
aptos-node-58-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-58-validator-0               1/1     Running     0          13m
aptos-node-59-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-59-validator-0               1/1     Running     0          13m
aptos-node-6-fullnode-eforge229-0       1/1     Running     0          13m
aptos-node-6-validator-0                1/1     Running     0          13m
aptos-node-60-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-60-validator-0               1/1     Running     0          13m
aptos-node-61-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-61-validator-0               1/1     Running     0          13m
aptos-node-62-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-62-validator-0               1/1     Running     0          13m
aptos-node-63-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-63-validator-0               1/1     Running     0          13m
aptos-node-64-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-64-validator-0               1/1     Running     0          13m
aptos-node-65-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-65-validator-0               1/1     Running     0          13m
aptos-node-66-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-66-validator-0               1/1     Running     0          13m
aptos-node-67-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-67-validator-0               1/1     Running     0          13m
aptos-node-68-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-68-validator-0               1/1     Running     0          13m
aptos-node-69-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-69-validator-0               1/1     Running     0          13m
aptos-node-7-fullnode-eforge229-0       1/1     Running     0          13m
aptos-node-7-validator-0                1/1     Running     0          13m
aptos-node-70-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-70-validator-0               1/1     Running     0          13m
aptos-node-71-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-71-validator-0               1/1     Running     0          13m
aptos-node-72-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-72-validator-0               1/1     Running     0          13m
aptos-node-73-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-73-validator-0               1/1     Running     0          13m
aptos-node-74-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-74-validator-0               1/1     Running     0          13m
aptos-node-75-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-75-validator-0               1/1     Running     0          13m
aptos-node-76-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-76-validator-0               1/1     Running     0          13m
aptos-node-77-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-77-validator-0               1/1     Running     0          13m
aptos-node-78-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-78-validator-0               1/1     Running     0          13m
aptos-node-79-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-79-validator-0               1/1     Running     0          13m
aptos-node-8-fullnode-eforge229-0       1/1     Running     0          13m
aptos-node-8-validator-0                1/1     Running     0          13m
aptos-node-80-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-80-validator-0               1/1     Running     0          13m
aptos-node-81-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-81-validator-0               1/1     Running     0          13m
aptos-node-82-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-82-validator-0               1/1     Running     0          13m
aptos-node-83-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-83-validator-0               1/1     Running     0          13m
aptos-node-84-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-84-validator-0               1/1     Running     0          13m
aptos-node-85-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-85-validator-0               1/1     Running     0          13m
aptos-node-86-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-86-validator-0               1/1     Running     0          13m
aptos-node-87-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-87-validator-0               1/1     Running     0          13m
aptos-node-88-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-88-validator-0               1/1     Running     0          13m
aptos-node-89-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-89-validator-0               1/1     Running     0          13m
aptos-node-9-fullnode-eforge229-0       1/1     Running     0          13m
aptos-node-9-validator-0                1/1     Running     0          13m
aptos-node-90-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-90-validator-0               1/1     Running     0          13m
aptos-node-91-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-91-validator-0               1/1     Running     0          13m
aptos-node-92-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-92-validator-0               1/1     Running     0          13m
aptos-node-93-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-93-validator-0               1/1     Running     0          13m
aptos-node-94-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-94-validator-0               1/1     Running     0          13m
aptos-node-95-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-95-validator-0               1/1     Running     0          13m
aptos-node-96-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-96-validator-0               1/1     Running     0          13m
aptos-node-97-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-97-validator-0               1/1     Running     0          13m
aptos-node-98-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-98-validator-0               1/1     Running     0          13m
aptos-node-99-fullnode-eforge229-0      1/1     Running     0          13m
aptos-node-99-validator-0               1/1     Running     0          13m
genesis-aptos-genesis-eforge229-rwh8t   0/1     Completed   0          16m

Grafana dashboard
Humio Logs
Test runner output
Test run is land-blocking

github-actions[bot] commented 10 months ago

Forge is running suite `framework_upgrade` on `aptos-node-v1.4.4` ==> `3edaf810ccf5b6068ba1622b491422f8c4feedec`

github-actions[bot] commented 10 months ago

Forge is running suite `realistic_env_max_load` on `3edaf810ccf5b6068ba1622b491422f8c4feedec`

github-actions[bot] commented 10 months ago

Forge is running suite `compat` on `aptos-node-v1.4.4` ==> `3edaf810ccf5b6068ba1622b491422f8c4feedec`

github-actions[bot] commented 10 months ago

:white_check_mark: Forge suite `realistic_env_max_load` success on `3edaf810ccf5b6068ba1622b491422f8c4feedec`

Forge test runner terminated:
Trailing Log Lines:
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/mod.rs:61"},"thread_name":"main","hostname":"forge-e2e-pr-8994-1689020131-3edaf810ccf5b6068ba1622b491422f8c4","timestamp":"2023-07-10T20:15:57.150401Z","message":"Using forge namespace: forge-e2e-pr-8994"}

running 0 tests

test result: ok. 0 passed; 0 failed; 0 filtered out

Debugging output:

Grafana dashboard
Humio Logs
Test runner output
Test run is land-blocking

github-actions[bot] commented 10 months ago

Forge is running suite `framework_upgrade` on `aptos-node-v1.4.4` ==> `8293da64fe390bc5002b510534212c75ac5f802f`

github-actions[bot] commented 10 months ago

Forge is running suite `realistic_env_max_load` on `8293da64fe390bc5002b510534212c75ac5f802f`

github-actions[bot] commented 10 months ago

Forge is running suite `compat` on `aptos-node-v1.4.4` ==> `8293da64fe390bc5002b510534212c75ac5f802f`

github-actions[bot] commented 10 months ago

:x: Forge suite `framework_upgrade` failure on `aptos-node-v1.4.4` ==> `8293da64fe390bc5002b510534212c75ac5f802f`

Compatibility test results for aptos-node-v1.4.4 ==> 8293da64fe390bc5002b510534212c75ac5f802f (PR)
Upgrade the nodes to version: 8293da64fe390bc5002b510534212c75ac5f802f
Test Failed: Waiting for nodes to catch up to target version and epoch (Some(1098), None) timed out after 10 seconds, current status: [("validator-0", 1078, 9), ("validator-1", 1140, 9), ("validator-2", 1140, 9), ("validator-3", 1140, 9), ("validator-4", 1098, 9)]

Stack backtrace:
   0: aptos_forge::interface::swarm::wait_for_all_nodes_to_catchup_to_target_version_or_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:451:24
   1: aptos_forge::interface::swarm::wait_for_all_nodes_to_catchup_to_version::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:368:5
   2: aptos_forge::interface::swarm::SwarmExt::wait_for_all_nodes_to_catchup_to_version::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:254:9
   3: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/future/future.rs:125:9
      tokio::park::thread::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/park/thread.rs:267:54
      tokio::coop::with_budget::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:102:9
      std::thread::local::LocalKey<T>::try_with
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/thread/local.rs:252:16
      std::thread::local::LocalKey<T>::with
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/thread/local.rs:228:9
      tokio::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:95:5
      tokio::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:72:5
      tokio::park::thread::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/park/thread.rs:267:31
   4: tokio::runtime::enter::Enter::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/enter.rs:152:13
      tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/scheduler/multi_thread/mod.rs:79:9
   5: tokio::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/mod.rs:492:44
   6: aptos_forge::interface::swarm::SwarmExt::fork_check
             at ./testsuite/forge/src/interface/swarm.rs:232:9
   7: <aptos_testcases::framework_upgrade::FrameworkUpgrade as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/framework_upgrade.rs:58:9
   8: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:482:42
      aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:550:11
      aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:482:30
   9: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:380:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:306:21
  10: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
  11: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
  12: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  13: main
  14: __libc_start_main
  15: _start
Trailing Log Lines:
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  13: main
  14: __libc_start_main
  15: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:291"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-8994-1689021206-aptos-node-v1-4-4","timestamp":"2023-07-10T20:43:46.496407Z","message":"Deleting namespace forge-framework-upgrade-pr-8994: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:399"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-8994-1689021206-aptos-node-v1-4-4","timestamp":"2023-07-10T20:43:46.496436Z","message":"aptos-node resources for Forge removed in namespace: forge-framework-upgrade-pr-8994"}
Failed to run tests:
Tests Failed

failures:
    framework_upgrade::framework-upgrade

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Error: Tests Failed

Stack backtrace:
   0: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:502:13
   1: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:380:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:306:21
   2: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
   3: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
   4: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
   5: main
   6: __libc_start_main
   7: _start
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-validator-0                1/1     Running     0          5m54s
aptos-node-1-validator-0                1/1     Running     0          3m54s
aptos-node-2-validator-0                1/1     Running     0          3m4s
aptos-node-3-validator-0                1/1     Running     0          2m4s
aptos-node-4-validator-0                1/1     Running     0          63s
genesis-aptos-genesis-eforge157-9kppp   0/1     Completed   0          9m46s

Grafana dashboard
Humio Logs
Test runner output
Test run is land-blocking

github-actions[bot] commented 10 months ago

:x: Forge suite `compat` failure on `aptos-node-v1.4.4` ==> `8293da64fe390bc5002b510534212c75ac5f802f`

Compatibility test results for aptos-node-v1.4.4 ==> 8293da64fe390bc5002b510534212c75ac5f802f (PR)
1. Check liveness of validators at old version: aptos-node-v1.4.4
compatibility::simple-validator-upgrade::liveness-check : committed: 7083 txn/s, latency: 4714 ms, (p50: 4600 ms, p90: 6600 ms, p99: 8700 ms), latency samples: 240840
2. Upgrading first Validator to new version: 8293da64fe390bc5002b510534212c75ac5f802f
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1282 txn/s, latency: 20369 ms, (p50: 20400 ms, p90: 26700 ms, p99: 31500 ms), latency samples: 78220
3. Upgrading rest of first batch to new version: 8293da64fe390bc5002b510534212c75ac5f802f
Test Failed: Tried executing 10 txns, request counters: "success 0, failed submit [0], failed wait [10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10], by client: [(0, 0, 180): http://aptos-node-0-validator.forge-compat-pr-8994.svc:8080/v1/]"

Caused by:
    Unknown error Ledger on endpoint (http://aptos-node-0-validator.forge-compat-pr-8994.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (d8d5363c) might still succeed.

Stack backtrace:
   0: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/result.rs:2002:27
      aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiReliableTransactionSubmitter::submit_check_and_retry::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:119:28
   1: <futures_util::future::maybe_done::MaybeDone<Fut> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.24/src/future/maybe_done.rs:95:38
      <futures_util::future::join_all::JoinAll<F> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.24/src/future/join_all.rs:142:24
   2: <aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiReliableTransactionSubmitter as aptos_transaction_generator_lib::ReliableTransactionSubmitter>::execute_transactions_with_counter::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:273:9
   3: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/future/future.rs:125:9
      aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_and_fund_seed_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:314:17
      aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:187:13
   4: aptos_transaction_emitter_lib::emitter::TxnEmitter::start_job::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:608:13
   5: aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for_impl::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:712:13
      aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:744:13
      tokio::park::thread::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/park/thread.rs:267:54
      tokio::coop::with_budget::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:102:9
      std::thread::local::LocalKey<T>::try_with
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/thread/local.rs:252:16
      std::thread::local::LocalKey<T>::with
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/thread/local.rs:228:9
   6: tokio::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:95:5
      tokio::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/coop.rs:72:5
      tokio::park::thread::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/park/thread.rs:267:31
   7: tokio::runtime::enter::Enter::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/enter.rs:152:13
      tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/scheduler/multi_thread/mod.rs:79:9
   8: tokio::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.21.2/src/runtime/mod.rs:492:44
   9: aptos_testcases::generate_traffic
             at ./testsuite/testcases/src/lib.rs:99:17
  10: <aptos_testcases::compatibility_test::SimpleValidatorUpgrade as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/compatibility_test.rs:93:24
  11: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:482:42
      aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:550:11
      aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:482:30
  12: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:380:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:306:21
  13: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
  14: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
  15: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  16: main
  17: __libc_start_main
  18: _start
Trailing Log Lines:
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  16: main
  17: __libc_start_main
  18: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:291"},"thread_name":"main","hostname":"forge-compat-pr-8994-1689021251-aptos-node-v1-4-4","timestamp":"2023-07-10T20:49:15.498100Z","message":"Deleting namespace forge-compat-pr-8994: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:399"},"thread_name":"main","hostname":"forge-compat-pr-8994-1689021251-aptos-node-v1-4-4","timestamp":"2023-07-10T20:49:15.498140Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-8994"}

failures:
    compatibility::simple-validator-upgrade
Failed to run tests:
Tests Failed

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Error: Tests Failed

Stack backtrace:
   0: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:502:13
   1: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:380:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:306:21
   2: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
   3: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
   4: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
   5: main
   6: __libc_start_main
   7: _start
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-validator-0                1/1     Running     0          4m18s
aptos-node-1-validator-0                1/1     Running     0          8m33s
aptos-node-2-validator-0                1/1     Running     0          12m
aptos-node-3-validator-0                1/1     Running     0          12m
aptos-node-4-validator-0                1/1     Running     0          12m
genesis-aptos-genesis-eforge148-x7c26   0/1     Completed   0          14m

Grafana dashboard
Humio Logs
Test runner output
Test run is land-blocking

Next