[Consensus Observer] Test v1.23 build.

JoshLind commented 6 days ago

Description

How Has This Been Tested?

Key Areas to Review

Type of Change

[ ] New feature
[ ] Bug fix
[ ] Breaking change
[ ] Performance improvement
[ ] Refactoring
[ ] Dependency update
[ ] Documentation update
[ ] Tests

Which Components or Systems Does This Change Impact?

[ ] Validator Node
[ ] Full Node (API, Indexer, etc.)
[ ] Move/Aptos Virtual Machine
[ ] Aptos Framework
[ ] Aptos CLI/SDK
[ ] Developer Infrastructure
[ ] Move Compiler
[ ] Other (specify)

Checklist

[ ] I have read and followed the CONTRIBUTING doc
[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I identified and added all stakeholders and component owners affected by this change as reviewers
[ ] I tested both happy and unhappy path of the functionality
[ ] I have made corresponding changes to the documentation

trunk-io[bot] commented 6 days ago

⏱️ 3h 31m total CI duration on this PR

| Slowest 15 Jobs | Cumulative Duration | Recent Runs | |---|---|---| | [execution-performance / single-node-performance](https://github.com/aptos-labs/aptos-core/actions/runs/11936328362/job/33270047880) | 49m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11924688060/job/33235606289) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11936328362/job/33270047880) | | [rust-unit-tests](https://github.com/aptos-labs/aptos-core/actions/runs/11924688016/job/33235470987) | 23m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11924688016/job/33235470987) | | [rust-unit-tests](https://github.com/aptos-labs/aptos-core/actions/runs/11936328322/job/33269724638) | 22m | [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/11936328322/job/33269724638) | | [rust-unit-tests](https://github.com/aptos-labs/aptos-core/actions/runs/11924350266/job/33234519060) | 22m | [🟥](https://github.com/aptos-labs/aptos-core/actions/runs/11924350266/job/33234519060) | | [rust-images / rust-all](https://github.com/aptos-labs/aptos-core/actions/runs/11924688133/job/33236856247) | 15m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11924688133/job/33236856247) | | [execution-performance / test-target-determinator](https://github.com/aptos-labs/aptos-core/actions/runs/11936328362/job/33269723611) | 10m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11924688060/job/33235468476) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11936328362/job/33269723611) | | [rust-move-tests](https://github.com/aptos-labs/aptos-core/actions/runs/11936328302/job/33269723094) | 9m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11936328302/job/33269723094) | | [rust-move-tests](https://github.com/aptos-labs/aptos-core/actions/runs/11924350259/job/33234518607) | 9m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11924350259/job/33234518607) | | [rust-move-tests](https://github.com/aptos-labs/aptos-core/actions/runs/11924688011/job/33235469444) | 9m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11924688011/job/33235469444) | | [test-target-determinator](https://github.com/aptos-labs/aptos-core/actions/runs/11936328212/job/33269736865) | 9m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11924688133/job/33235474943) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11936328212/job/33269736865) | | [check](https://github.com/aptos-labs/aptos-core/actions/runs/11936328304/job/33269723122) | 7m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11924688008/job/33235468032) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11936328304/job/33269723122) | | [rust-cargo-deny](https://github.com/aptos-labs/aptos-core/actions/runs/11936328322/job/33269746866) | 6m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11924350266/job/33234527313) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11924688016/job/33235479479) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11936328322/job/33269746866) | | [check-dynamic-deps](https://github.com/aptos-labs/aptos-core/actions/runs/11936328328/job/33269722533) | 5m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11924350245/job/33234517829) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11924688018/job/33235467668) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11924685987/job/33235462470) [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11936328328/job/33269722533) | | [rust-doc-tests](https://github.com/aptos-labs/aptos-core/actions/runs/11924688016/job/33235470814) | 5m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11924688016/job/33235470814) | | [rust-doc-tests](https://github.com/aptos-labs/aptos-core/actions/runs/11936328322/job/33269724252) | 5m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/11936328322/job/33269724252) |

🚨 1 job on the last run was significantly faster/slower than expected

Job	Duration	vs 7d avg	Delta
execution-performance / single-node-performance	26m	16m

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

github-actions[bot] commented 6 days ago

Forge is running suite `realistic_env_max_load` on `1628ce17c6d2a29e2f544414ee5d031f1b54d0cc`

github-actions[bot] commented 6 days ago

Forge is running suite `compat` on `afa8d75c467820b7d96bbe13ab0d011c12f8b3a3` ==> `1628ce17c6d2a29e2f544414ee5d031f1b54d0cc`

github-actions[bot] commented 6 days ago

:white_check_mark: Forge suite `realistic_env_max_load` success on `1628ce17c6d2a29e2f544414ee5d031f1b54d0cc`

two traffics test: inner traffic : committed: 14271.88 txn/s, latency: 2787.44 ms, (p50: 2700 ms, p70: 2700, p90: 3000 ms, p99: 3200 ms), latency samples: 5426540
two traffics test : committed: 100.01 txn/s, latency: 1395.92 ms, (p50: 1400 ms, p70: 1400, p90: 1600 ms, p99: 1900 ms), latency samples: 1740
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 2.057, avg: 1.604", "ConsensusProposalToOrdered: max: 0.318, avg: 0.294", "ConsensusOrderedToCommit: max: 0.367, avg: 0.356", "ConsensusProposalToCommit: max: 0.661, avg: 0.650"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.01s no progress at version 1968990 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.66s no progress at version 1968988 (avg 7.92s) [limit 15].
Test Ok

github-actions[bot] commented 6 days ago

:white_check_mark: Forge suite `compat` success on `afa8d75c467820b7d96bbe13ab0d011c12f8b3a3` ==> `1628ce17c6d2a29e2f544414ee5d031f1b54d0cc`

Compatibility test results for afa8d75c467820b7d96bbe13ab0d011c12f8b3a3 ==> 1628ce17c6d2a29e2f544414ee5d031f1b54d0cc (PR)
1. Check liveness of validators at old version: afa8d75c467820b7d96bbe13ab0d011c12f8b3a3
compatibility::simple-validator-upgrade::liveness-check : committed: 14800.22 txn/s, latency: 2335.16 ms, (p50: 2100 ms, p70: 2100, p90: 2300 ms, p99: 7500 ms), latency samples: 532300
2. Upgrading first Validator to new version: 1628ce17c6d2a29e2f544414ee5d031f1b54d0cc
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 7925.73 txn/s, latency: 3511.86 ms, (p50: 4000 ms, p70: 4200, p90: 4500 ms, p99: 4800 ms), latency samples: 140500
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 7863.36 txn/s, latency: 4102.79 ms, (p50: 4300 ms, p70: 4400, p90: 5800 ms, p99: 6200 ms), latency samples: 256980
3. Upgrading rest of first batch to new version: 1628ce17c6d2a29e2f544414ee5d031f1b54d0cc
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 7765.26 txn/s, latency: 3563.40 ms, (p50: 3600 ms, p70: 4100, p90: 5000 ms, p99: 5400 ms), latency samples: 143680
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 7576.67 txn/s, latency: 4226.61 ms, (p50: 4300 ms, p70: 4500, p90: 6300 ms, p99: 6500 ms), latency samples: 251840
4. upgrading second batch to new version: 1628ce17c6d2a29e2f544414ee5d031f1b54d0cc
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 12244.45 txn/s, latency: 2247.34 ms, (p50: 2500 ms, p70: 2600, p90: 2700 ms, p99: 2800 ms), latency samples: 213480
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 10323.13 txn/s, latency: 2990.30 ms, (p50: 2600 ms, p70: 2700, p90: 6100 ms, p99: 8000 ms), latency samples: 338300
5. check swarm health
Compatibility test for afa8d75c467820b7d96bbe13ab0d011c12f8b3a3 ==> 1628ce17c6d2a29e2f544414ee5d031f1b54d0cc passed
Test Ok

github-actions[bot] commented 5 days ago

Forge is running suite `realistic_env_max_load` on `710286633e64f56dcd85ab235a19294e301f4ac7`

github-actions[bot] commented 5 days ago

Forge is running suite `compat` on `afa8d75c467820b7d96bbe13ab0d011c12f8b3a3` ==> `710286633e64f56dcd85ab235a19294e301f4ac7`

github-actions[bot] commented 5 days ago

:x: Forge suite `realistic_env_max_load` failure on `710286633e64f56dcd85ab235a19294e301f4ac7`

two traffics test: inner traffic : committed: 14293.43 txn/s, latency: 2781.19 ms, (p50: 2700 ms, p70: 2700, p90: 3000 ms, p99: 8300 ms), latency samples: 5434720
Test Failed: Failed to query prometheus HTTP status server error (500 Internal Server Error) for url (https://ds.groundcover.com/datasources/prometheus/api/v1/query_range?query=quantile%280.67%2C+rate%28aptos_consensus_block_tracing_sum%7Bnamespace%3D%22forge-e2e-pr-15324%22%2Crole%3D%7E%22validator%22%2C+stage%3D%22ordered%22%7D%5B1m%5D%29+%2F+rate%28aptos_consensus_block_tracing_count%7Bnamespace%3D%22forge-e2e-pr-15324%22%2Crole%3D%7E%22validator%22%2C+stage%3D%22ordered%22%7D%5B1m%5D%29%29&start=1732117275&end=1732117658&step=30). start=1732117275, end=1732117658, query=quantile(0.67, rate(aptos_consensus_block_tracing_sum{namespace="forge-e2e-pr-15324",role=~"validator", stage="ordered"}[1m]) / rate(aptos_consensus_block_tracing_count{namespace="forge-e2e-pr-15324",role=~"validator", stage="ordered"}[1m]))

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.89/src/error.rs:85:36
   1: aptos_forge::backend::k8s::prometheus::query_range_with_metadata::{{closure}}::{{closure}}
             at ./testsuite/forge/src/backend/k8s/prometheus.rs:174:13
   2: core::result::Result<T,E>::map_err
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/result.rs:829:27
   3: aptos_forge::backend::k8s::prometheus::query_range_with_metadata::{{closure}}
             at ./testsuite/forge/src/backend/k8s/prometheus.rs:168:13
   4: <aptos_forge::backend::k8s::swarm::K8sSwarm as aptos_forge::interface::swarm::Swarm>::query_range_metrics::{{closure}}
             at ./testsuite/forge/src/backend/k8s/swarm.rs:433:14
   5: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   6: aptos_forge::interface::prometheus_metrics::fetch_latency_breakdown::{{closure}}
             at ./testsuite/forge/src/interface/prometheus_metrics.rs:177:10
   7: aptos_testcases::create_buffered_load::{{closure}}
             at ./testsuite/testcases/src/lib.rs:465:10
   8: <dyn aptos_testcases::NetworkLoadTest>::network_load_test::{{closure}}
             at ./testsuite/testcases/src/lib.rs:510:10
   9: <dyn aptos_testcases::NetworkLoadTest as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/lib.rs:268:14
  10: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
  11: <aptos_testcases::two_traffics_test::TwoTrafficsTest as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/two_traffics_test.rs:77:47
  12: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
  13: <aptos_testcases::CompositeNetworkTest as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/lib.rs:632:37
  14: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
  15: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:63
  16: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:107:5
  17: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:73:5
  18: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:31
  19: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/blocking.rs:66:9
  20: tokio::runtime::handle::Handle::block_on_inner::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:324:22
  21: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16
  22: tokio::runtime::handle::Handle::block_on_inner
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:323:9
  23: tokio::runtime::handle::Handle::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:302:18
  24: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:332:50
  25: forge::run_forge_with_changelog
             at ./testsuite/forge-cli/src/main.rs:426:24
  26: forge::main
             at ./testsuite/forge-cli/src/main.rs:329:21
  27: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
  28: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
  29: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
  30: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
  31: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  32: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  33: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  34: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  35: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  36: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  37: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  38: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  39: main
  40: __libc_start_main
  41: _start
Trailing Log Lines:
  36: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  37: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  38: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  39: main
  40: __libc_start_main
  41: _start

=== BEGIN JUNIT ===
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="forge" tests="1" failures="1" errors="0" uuid="c5c89308-586b-492e-8ed2-3cf2dbbf4f11">
    <testsuite name="local" tests="1" disabled="0" errors="0" failures="1">
        <testcase name="CompositeNetworkTest(CpuChaosWrapper(network:multi-region-network-emulation(two traffics test))) with ">
            <failure message="Failed to query prometheus HTTP status server error (500 Internal Server Error) for url (https://ds.groundcover.com/datasources/prometheus/api/v1/query_range?query=quantile%280.67%2C+rate%28aptos_consensus_block_tracing_sum%7Bnamespace%3D%22forge-e2e-pr-15324%22%2Crole%3D%7E%22validator%22%2C+stage%3D%22ordered%22%7D%5B1m%5D%29+%2F+rate%28aptos_consensus_block_tracing_count%7Bnamespace%3D%22forge-e2e-pr-15324%22%2Crole%3D%7E%22validator%22%2C+stage%3D%22ordered%22%7D%5B1m%5D%29%29&amp;start=1732117275&amp;end=1732117658&amp;step=30). start=1732117275, end=1732117658, query=quantile(0.67, rate(aptos_consensus_block_tracing_sum{namespace=&quot;forge-e2e-pr-15324&quot;,role=~&quot;validator&quot;, stage=&quot;ordered&quot;}[1m]) / rate(aptos_consensus_block_tracing_count{namespace=&quot;forge-e2e-pr-15324&quot;,role=~&quot;validator&quot;, stage=&quot;ordered&quot;}[1m]))

Stack backtrace:
   0: anyhow::error::&lt;impl anyhow::Error&gt;::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.89/src/error.rs:85:36
   1: aptos_forge::backend::k8s::prometheus::query_range_with_metadata::{{closure}}::{{closure}}
             at ./testsuite/forge/src/backend/k8s/prometheus.rs:174:13
   2: core::result::Result&lt;T,E&gt;::map_err
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/result.rs:829:27
   3: aptos_forge::backend::k8s::prometheus::query_range_with_metadata::{{closure}}
             at ./testsuite/forge/src/backend/k8s/prometheus.rs:168:13
   4: &lt;aptos_forge::backend::k8s::swarm::K8sSwarm as aptos_forge::interface::swarm::Swarm&gt;::query_range_metrics::{{closure}}
             at ./testsuite/forge/src/backend/k8s/swarm.rs:433:14
   5: &lt;core::pin::Pin&lt;P&gt; as core::future::future::Future&gt;::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   6: aptos_forge::interface::prometheus_metrics::fetch_latency_breakdown::{{closure}}
             at ./testsuite/forge/src/interface/prometheus_metrics.rs:177:10
   7: aptos_testcases::create_buffered_load::{{closure}}
             at ./testsuite/testcases/src/lib.rs:465:10
   8: &lt;dyn aptos_testcases::NetworkLoadTest&gt;::network_load_test::{{closure}}
             at ./testsuite/testcases/src/lib.rs:510:10
   9: &lt;dyn aptos_testcases::NetworkLoadTest as aptos_forge::interface::network::NetworkTest&gt;::run::{{closure}}
             at ./testsuite/testcases/src/lib.rs:268:14
  10: &lt;core::pin::Pin&lt;P&gt; as core::future::future::Future&gt;::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
  11: &lt;aptos_testcases::two_traffics_test::TwoTrafficsTest as aptos_forge::interface::network::NetworkTest&gt;::run::{{closure}}
             at ./testsuite/testcases/src/two_traffics_test.rs:77:47
  12: &lt;core::pin::Pin&lt;P&gt; as core::future::future::Future&gt;::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
  13: &lt;aptos_testcases::CompositeNetworkTest as aptos_forge::interface::network::NetworkTest&gt;::run::{{closure}}
             at ./testsuite/testcases/src/lib.rs:632:37
  14: &lt;core::pin::Pin&lt;P&gt; as core::future::future::Future&gt;::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
  15: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:63
  16: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:107:5
  17: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:73:5
  18: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:31
  19: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/blocking.rs:66:9
  20: tokio::runtime::handle::Handle::block_on_inner::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:324:22
  21: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16
  22: tokio::runtime::handle::Handle::block_on_inner
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:323:9
  23: tokio::runtime::handle::Handle::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:302:18
  24: aptos_forge::runner::Forge&lt;F&gt;::run
             at ./testsuite/forge/src/runner.rs:332:50
  25: forge::run_forge_with_changelog
             at ./testsuite/forge-cli/src/main.rs:426:24
  26: forge::main
             at ./testsuite/forge-cli/src/main.rs:329:21
  27: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
  28: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
  29: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
  30: core::ops::function::impls::&lt;impl core::ops::function::FnOnce&lt;A&gt; for &amp;F&gt;::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
  31: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  32: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  33: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  34: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  35: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  36: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  37: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  38: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  39: main
  40: __libc_start_main
  41: _start"/>
        </testcase>
    </testsuite>
</testsuites>
=== END JUNIT ===

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:381"},"thread_name":"main","hostname":"forge-e2e-pr-15324-1732116982-710286633e64f56dcd85ab235a19294e3","timestamp":"2024-11-20T15:48:01.150602Z","message":"Deleting namespace forge-e2e-pr-15324: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:398"},"thread_name":"main","hostname":"forge-e2e-pr-15324-1732116982-710286633e64f56dcd85ab235a19294e3","timestamp":"2024-11-20T15:48:01.150628Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-15324"}

failures:
    CompositeNetworkTest

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.89/src/error.rs:85:36
   1: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:358:13
   2: forge::run_forge_with_changelog
             at ./testsuite/forge-cli/src/main.rs:426:24
   3: forge::main
             at ./testsuite/forge-cli/src/main.rs:329:21
   4: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
   5: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
   6: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
   7: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
   8: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
   9: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  10: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  11: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  12: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  13: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  14: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  15: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  16: main
  17: __libc_start_main
  18: _start
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-fullnode-eforge229-0       1/1     Running     0          10m
aptos-node-0-validator-0                1/1     Running     0          10m
aptos-node-1-fullnode-eforge229-0       1/1     Running     0          10m
aptos-node-1-validator-0                1/1     Running     0          10m
aptos-node-2-fullnode-eforge229-0       1/1     Running     0          10m
aptos-node-2-validator-0                1/1     Running     0          10m
aptos-node-3-fullnode-eforge229-0       1/1     Running     0          10m
aptos-node-3-validator-0                1/1     Running     0          10m
aptos-node-4-fullnode-eforge229-0       1/1     Running     0          10m
aptos-node-4-validator-0                1/1     Running     0          10m
aptos-node-5-validator-0                1/1     Running     0          10m
aptos-node-6-validator-0                1/1     Running     0          10m
forge-testnet-deployer-qtgf5            0/1     Completed   0          11m
genesis-aptos-genesis-eforge229-ckdzt   0/1     Completed   0          11m

github-actions[bot] commented 5 days ago

:white_check_mark: Forge suite `compat` success on `afa8d75c467820b7d96bbe13ab0d011c12f8b3a3` ==> `710286633e64f56dcd85ab235a19294e301f4ac7`

Compatibility test results for afa8d75c467820b7d96bbe13ab0d011c12f8b3a3 ==> 710286633e64f56dcd85ab235a19294e301f4ac7 (PR)
1. Check liveness of validators at old version: afa8d75c467820b7d96bbe13ab0d011c12f8b3a3
compatibility::simple-validator-upgrade::liveness-check : committed: 16498.75 txn/s, latency: 2025.46 ms, (p50: 1900 ms, p70: 2000, p90: 2600 ms, p99: 4500 ms), latency samples: 542140
2. Upgrading first Validator to new version: 710286633e64f56dcd85ab235a19294e301f4ac7
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 6817.03 txn/s, latency: 4116.36 ms, (p50: 4500 ms, p70: 4600, p90: 5200 ms, p99: 5500 ms), latency samples: 142200
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 7311.36 txn/s, latency: 4407.81 ms, (p50: 4600 ms, p70: 4700, p90: 6400 ms, p99: 6500 ms), latency samples: 245200
3. Upgrading rest of first batch to new version: 710286633e64f56dcd85ab235a19294e301f4ac7
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 7171.88 txn/s, latency: 3884.78 ms, (p50: 4200 ms, p70: 4400, p90: 5200 ms, p99: 5300 ms), latency samples: 145580
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 7613.49 txn/s, latency: 4165.92 ms, (p50: 4200 ms, p70: 4300, p90: 6600 ms, p99: 6900 ms), latency samples: 253900
4. upgrading second batch to new version: 710286633e64f56dcd85ab235a19294e301f4ac7
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 11444.80 txn/s, latency: 2403.34 ms, (p50: 2400 ms, p70: 2700, p90: 3400 ms, p99: 3700 ms), latency samples: 200000
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 10981.00 txn/s, latency: 2794.90 ms, (p50: 2500 ms, p70: 2700, p90: 4800 ms, p99: 6100 ms), latency samples: 356260
5. check swarm health
Compatibility test for afa8d75c467820b7d96bbe13ab0d011c12f8b3a3 ==> 710286633e64f56dcd85ab235a19294e301f4ac7 passed
Test Ok

aptos-labs / aptos-core