[Quorum Store] fairness and priority in proposal via per-author queue - Githubissues

aptos-labs / aptos-core

Aptos is a layer 1 blockchain built to support the widespread use of blockchain through better technology and user experience.

https://aptosfoundation.org

Other

5.86k stars 3.54k forks source link

[Quorum Store] fairness and priority in proposal via per-author queue #7315

Closed bchocho closed 1 year ago

bchocho commented 1 year ago

Description

Add a per-author queue in proof manager. On proposal, proofs are taken in a (shuffled) round-robin of authors. This helps maintain some amount of fairness. For example, a single author that creates a huge backlog of proofs can't monopolize blocks if other proofs are also being received.

The per-author queue is prioritized. The tail is taken of the queue, which is sorted by gas bucket (ascending) and batch id (descending).

As a safeguard to spamming attacks, only a single proof can be stored at a time for a given batch.

Test Plan

Added unit tests, run three region overload tests.

github-actions[bot] commented 1 year ago

Forge is running suite `compat` on `testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b` ==> `2916215e9ed496df0b955ba9c152d6d3d44dc920`

github-actions[bot] commented 1 year ago

Forge is running suite `land_blocking` on `2916215e9ed496df0b955ba9c152d6d3d44dc920`

github-actions[bot] commented 1 year ago

Forge is running suite `framework_upgrade` on `cb4ba0a57c998c60cbab65af31a64875d2588ca5` ==> `2916215e9ed496df0b955ba9c152d6d3d44dc920`

github-actions[bot] commented 1 year ago

:x: Forge suite `framework_upgrade` failure on `cb4ba0a57c998c60cbab65af31a64875d2588ca5` ==> `2916215e9ed496df0b955ba9c152d6d3d44dc920`

Compatibility test results for cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> 2916215e9ed496df0b955ba9c152d6d3d44dc920 (PR)
Upgrade the nodes to version: 2916215e9ed496df0b955ba9c152d6d3d44dc920
Test Failed: CrashLoopBackOff
Trailing Log Lines:
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/stateful_set.rs:107"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-7315-1679436097-cb4ba0a57c998c60cbab","timestamp":"2023-03-21T22:08:54.520393Z","message":"StatefulSet aptos-node-1-validator has 0/1 replicas"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/stateful_set.rs:133"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-7315-1679436097-cb4ba0a57c998c60cbab","timestamp":"2023-03-21T22:08:54.524355Z","message":"Pod aptos-node-1-validator-0 has CrashLoopBackOff"}
::error::CrashLoopBackOff
test framework_upgrade::framework-upgrade ... FAILED
Error: CrashLoopBackOff
Test Statistics: 
Compatibility test results for cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> 2916215e9ed496df0b955ba9c152d6d3d44dc920 (PR)
Upgrade the nodes to version: 2916215e9ed496df0b955ba9c152d6d3d44dc920
Test Failed: CrashLoopBackOff

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:281"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-7315-1679436097-cb4ba0a57c998c60cbab","timestamp":"2023-03-21T22:08:54.553313Z","message":"Deleting namespace forge-framework-upgrade-pr-7315: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:389"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-7315-1679436097-cb4ba0a57c998c60cbab","timestamp":"2023-03-21T22:08:54.553339Z","message":"aptos-node resources for Forge removed in namespace: forge-framework-upgrade-pr-7315"}

failures:
    framework_upgrade::framework-upgrade

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
genesis-aptos-genesis-eforge118-4djzc   0/1     Completed   0          6m27s

github-actions[bot] commented 1 year ago

:x: Forge suite `compat` failure on `testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b` ==> `2916215e9ed496df0b955ba9c152d6d3d44dc920`

Compatibility test results for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 2916215e9ed496df0b955ba9c152d6d3d44dc920 (PR)
1. Check liveness of validators at old version: testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b
compatibility::simple-validator-upgrade::liveness-check : 8102 TPS, 4730 ms latency, 7500 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 2916215e9ed496df0b955ba9c152d6d3d44dc920
Test Failed: Tried executing 10 txns, failed by call by retry: [0, 10, 0, 10, 0, 10, 3, 10, 1, 10, 0, 10, 8, 10, 10, 10]

Caused by:
    Unknown error Transaction expired, without being seen in mempool. It is guaranteed it will not be committed on chain.
Trailing Log Lines:
Test Statistics: 
Compatibility test results for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 2916215e9ed496df0b955ba9c152d6d3d44dc920 (PR)
1. Check liveness of validators at old version: testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b
compatibility::simple-validator-upgrade::liveness-check : 8102 TPS, 4730 ms latency, 7500 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 2916215e9ed496df0b955ba9c152d6d3d44dc920
Test Failed: Tried executing 10 txns, failed by call by retry: [0, 10, 0, 10, 0, 10, 3, 10, 1, 10, 0, 10, 8, 10, 10, 10]

Caused by:
    Unknown error Transaction expired, without being seen in mempool. It is guaranteed it will not be committed on chain.

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:281"},"thread_name":"main","hostname":"forge-compat-pr-7315-1679436090-testnet-2d8b1b57553d869190f61df","timestamp":"2023-03-21T22:11:44.878517Z","message":"Deleting namespace forge-compat-pr-7315: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:389"},"thread_name":"main","hostname":"forge-compat-pr-7315-1679436090-testnet-2d8b1b57553d869190f61df","timestamp":"2023-03-21T22:11:44.878544Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-7315"}

failures:
    compatibility::simple-validator-upgrade

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
NAME                                    READY   STATUS        RESTARTS      AGE
aptos-node-0-validator-0                1/1     Terminating   0             9m11s
aptos-node-1-validator-0                0/1     Error         3 (36s ago)   4m5s
aptos-node-2-validator-0                1/1     Running       0             9m11s
aptos-node-3-validator-0                1/1     Running       0             9m11s
aptos-node-4-validator-0                1/1     Running       0             9m11s
genesis-aptos-genesis-eforge228-5jg78   0/1     Completed     0             9m20s

github-actions[bot] commented 1 year ago

:white_check_mark: Forge suite `land_blocking` success on `2916215e9ed496df0b955ba9c152d6d3d44dc920`

performance benchmark with full nodes : 5837 TPS, 6788 ms latency, 25800 ms p99 latency,no expired txns
Test Ok

github-actions[bot] commented 1 year ago

Forge is running suite `compat` on `testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b` ==> `a045b9be19cd90a33f5279e33301047086f3ee4a`

github-actions[bot] commented 1 year ago

Forge is running suite `land_blocking` on `a045b9be19cd90a33f5279e33301047086f3ee4a`

github-actions[bot] commented 1 year ago

Forge is running suite `framework_upgrade` on `cb4ba0a57c998c60cbab65af31a64875d2588ca5` ==> `a045b9be19cd90a33f5279e33301047086f3ee4a`

github-actions[bot] commented 1 year ago

:x: Forge suite `compat` failure on `testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b` ==> `a045b9be19cd90a33f5279e33301047086f3ee4a`

Compatibility test results for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> a045b9be19cd90a33f5279e33301047086f3ee4a (PR)
1. Check liveness of validators at old version: testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b
compatibility::simple-validator-upgrade::liveness-check : 7808 TPS, 4933 ms latency, 6900 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: a045b9be19cd90a33f5279e33301047086f3ee4a
Test Failed: Tried executing 10 txns, failed by call by retry: [0, 10, 0, 10, 0, 10, 2, 10, 0, 10, 0, 10, 3, 10, 3, 10]

Caused by:
    Unknown error Transaction expired, without being seen in mempool. It is guaranteed it will not be committed on chain.
Trailing Log Lines:
Test Statistics: 
Compatibility test results for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> a045b9be19cd90a33f5279e33301047086f3ee4a (PR)
1. Check liveness of validators at old version: testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b
compatibility::simple-validator-upgrade::liveness-check : 7808 TPS, 4933 ms latency, 6900 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: a045b9be19cd90a33f5279e33301047086f3ee4a
Test Failed: Tried executing 10 txns, failed by call by retry: [0, 10, 0, 10, 0, 10, 2, 10, 0, 10, 0, 10, 3, 10, 3, 10]

Caused by:
    Unknown error Transaction expired, without being seen in mempool. It is guaranteed it will not be committed on chain.

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:281"},"thread_name":"main","hostname":"forge-compat-pr-7315-1679472295-testnet-2d8b1b57553d869190f61df","timestamp":"2023-03-22T08:14:00.283950Z","message":"Deleting namespace forge-compat-pr-7315: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:389"},"thread_name":"main","hostname":"forge-compat-pr-7315-1679472295-testnet-2d8b1b57553d869190f61df","timestamp":"2023-03-22T08:14:00.283974Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-7315"}

failures:
    compatibility::simple-validator-upgrade

test result: FAILED. 0 passed; 1 failed; 0 filtered out
Failed to run tests:
Tests Failed

Error: Tests Failed
Debugging output:

github-actions[bot] commented 1 year ago

:white_check_mark: Forge suite `land_blocking` success on `a045b9be19cd90a33f5279e33301047086f3ee4a`

performance benchmark with full nodes : 6244 TPS, 6351 ms latency, 14400 ms p99 latency,no expired txns
Test Ok

github-actions[bot] commented 1 year ago

:x: Forge suite `framework_upgrade` failure on `cb4ba0a57c998c60cbab65af31a64875d2588ca5` ==> `a045b9be19cd90a33f5279e33301047086f3ee4a`

Compatibility test results for cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> a045b9be19cd90a33f5279e33301047086f3ee4a (PR)
Upgrade the nodes to version: a045b9be19cd90a33f5279e33301047086f3ee4a
Test Failed: Timed out waiting for Node validator-0:ecf76a8d9d768cefdafba5a7241814e495adfda1820b181fe31dd4e3c30acbb1 to be healthy
Trailing Log Lines:
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:529"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-7315-1679472302-cb4ba0a57c998c60cbab","timestamp":"2023-03-22T08:38:31.941524Z","message":"Node validator-3 healthy @ version 644 > 100"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:529"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-7315-1679472302-cb4ba0a57c998c60cbab","timestamp":"2023-03-22T08:38:31.948843Z","message":"Node validator-4 healthy @ version 644 > 100"}
::error::Timed out waiting for Node validator-0:ecf76a8d9d768cefdafba5a7241814e495adfda1820b181fe31dd4e3c30acbb1 to be healthy
test framework_upgrade::framework-upgrade ... FAILED
Error: Timed out waiting for Node validator-0:ecf76a8d9d768cefdafba5a7241814e495adfda1820b181fe31dd4e3c30acbb1 to be healthy
Test Statistics: 
Compatibility test results for cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> a045b9be19cd90a33f5279e33301047086f3ee4a (PR)
Upgrade the nodes to version: a045b9be19cd90a33f5279e33301047086f3ee4a
Test Failed: Timed out waiting for Node validator-0:ecf76a8d9d768cefdafba5a7241814e495adfda1820b181fe31dd4e3c30acbb1 to be healthy

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:281"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-7315-1679472302-cb4ba0a57c998c60cbab","timestamp":"2023-03-22T08:39:33.404479Z","message":"Deleting namespace forge-framework-upgrade-pr-7315: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:389"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-7315-1679472302-cb4ba0a57c998c60cbab","timestamp":"2023-03-22T08:39:33.404502Z","message":"aptos-node resources for Forge removed in namespace: forge-framework-upgrade-pr-7315"}

failures:
    framework_upgrade::framework-upgrade

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
NAME                                  READY   STATUS             RESTARTS        AGE
aptos-node-0-validator-0              0/1     CrashLoopBackOff   10 (102s ago)   30m
aptos-node-1-validator-0              0/1     CrashLoopBackOff   9 (102s ago)    29m
aptos-node-2-validator-0              0/1     CrashLoopBackOff   9 (62s ago)     28m
aptos-node-3-validator-0              0/1     CrashLoopBackOff   9 (63s ago)     26m
aptos-node-4-validator-0              0/1     CrashLoopBackOff   9 (62s ago)     25m
genesis-aptos-genesis-eforge9-c9qjk   0/1     Completed          0               33m

bchocho commented 1 year ago

The land_blocking test with quorum store enabled is: https://github.com/aptos-labs/aptos-core/pull/7315#issuecomment-1479090035

github-actions[bot] commented 1 year ago

Forge is running suite `compat` on `testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b` ==> `9a123a09e21ea2c50545d8ae25c1f02ba932209b`

github-actions[bot] commented 1 year ago

Forge is running suite `land_blocking` on `9a123a09e21ea2c50545d8ae25c1f02ba932209b`

github-actions[bot] commented 1 year ago

Forge is running suite `framework_upgrade` on `cb4ba0a57c998c60cbab65af31a64875d2588ca5` ==> `9a123a09e21ea2c50545d8ae25c1f02ba932209b`

github-actions[bot] commented 1 year ago

:white_check_mark: Forge suite `framework_upgrade` success on `cb4ba0a57c998c60cbab65af31a64875d2588ca5` ==> `9a123a09e21ea2c50545d8ae25c1f02ba932209b`

Compatibility test results for cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> 9a123a09e21ea2c50545d8ae25c1f02ba932209b (PR)
Upgrade the nodes to version: 9a123a09e21ea2c50545d8ae25c1f02ba932209b
framework_upgrade::framework-upgrade::full-framework-upgrade : 6511 TPS, 5910 ms latency, 8600 ms p99 latency,no expired txns
5. check swarm health
Compatibility test for cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> 9a123a09e21ea2c50545d8ae25c1f02ba932209b passed
Test Ok

github-actions[bot] commented 1 year ago

:white_check_mark: Forge suite `land_blocking` success on `9a123a09e21ea2c50545d8ae25c1f02ba932209b`

performance benchmark with full nodes : 5980 TPS, 6593 ms latency, 9600 ms p99 latency,no expired txns
Test Ok

github-actions[bot] commented 1 year ago

:white_check_mark: Forge suite `compat` success on `testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b` ==> `9a123a09e21ea2c50545d8ae25c1f02ba932209b`

Compatibility test results for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 9a123a09e21ea2c50545d8ae25c1f02ba932209b (PR)
1. Check liveness of validators at old version: testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b
compatibility::simple-validator-upgrade::liveness-check : 8100 TPS, 4745 ms latency, 7200 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 9a123a09e21ea2c50545d8ae25c1f02ba932209b
compatibility::simple-validator-upgrade::single-validator-upgrade : 3597 TPS, 10583 ms latency, 13400 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: 9a123a09e21ea2c50545d8ae25c1f02ba932209b
compatibility::simple-validator-upgrade::half-validator-upgrade : 4231 TPS, 9121 ms latency, 11700 ms p99 latency,no expired txns
4. upgrading second batch to new version: 9a123a09e21ea2c50545d8ae25c1f02ba932209b
compatibility::simple-validator-upgrade::rest-validator-upgrade : 6490 TPS, 5675 ms latency, 8400 ms p99 latency,no expired txns
5. check swarm health
Compatibility test for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 9a123a09e21ea2c50545d8ae25c1f02ba932209b passed
Test Ok

bchocho commented 1 year ago

I'm going to work on bucketing on this PR, it should also address some of the comments above.

github-actions[bot] commented 1 year ago

Forge is running suite `framework_upgrade` on `aptos-node-v1.3.0_3fc3d42b6cfe27460004f9a0326451bcda840a60` ==> `c8188d1e4ccc085db2df74172f081239ec6aa6e4`

github-actions[bot] commented 1 year ago

Forge is running suite `land_blocking` on `c8188d1e4ccc085db2df74172f081239ec6aa6e4`

github-actions[bot] commented 1 year ago

Forge is running suite `compat` on `testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b` ==> `c8188d1e4ccc085db2df74172f081239ec6aa6e4`

github-actions[bot] commented 1 year ago

:white_check_mark: Forge suite `land_blocking` success on `c8188d1e4ccc085db2df74172f081239ec6aa6e4`

performance benchmark with full nodes : 5569 TPS, 7115 ms latency, 11700 ms p99 latency,no expired txns
Test Ok

github-actions[bot] commented 1 year ago

:white_check_mark: Forge suite `compat` success on `testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b` ==> `c8188d1e4ccc085db2df74172f081239ec6aa6e4`

Compatibility test results for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> c8188d1e4ccc085db2df74172f081239ec6aa6e4 (PR)
1. Check liveness of validators at old version: testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b
compatibility::simple-validator-upgrade::liveness-check : 8145 TPS, 4678 ms latency, 6700 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: c8188d1e4ccc085db2df74172f081239ec6aa6e4
compatibility::simple-validator-upgrade::single-validator-upgrade : 4643 TPS, 8869 ms latency, 12200 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: c8188d1e4ccc085db2df74172f081239ec6aa6e4
compatibility::simple-validator-upgrade::half-validator-upgrade : 4992 TPS, 7774 ms latency, 10700 ms p99 latency,no expired txns
4. upgrading second batch to new version: c8188d1e4ccc085db2df74172f081239ec6aa6e4
compatibility::simple-validator-upgrade::rest-validator-upgrade : 6838 TPS, 5791 ms latency, 9400 ms p99 latency,no expired txns
5. check swarm health
Compatibility test for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> c8188d1e4ccc085db2df74172f081239ec6aa6e4 passed
Test Ok

github-actions[bot] commented 1 year ago

:white_check_mark: Forge suite `framework_upgrade` success on `aptos-node-v1.3.0_3fc3d42b6cfe27460004f9a0326451bcda840a60` ==> `c8188d1e4ccc085db2df74172f081239ec6aa6e4`

Compatibility test results for aptos-node-v1.3.0_3fc3d42b6cfe27460004f9a0326451bcda840a60 ==> c8188d1e4ccc085db2df74172f081239ec6aa6e4 (PR)
Upgrade the nodes to version: c8188d1e4ccc085db2df74172f081239ec6aa6e4
framework_upgrade::framework-upgrade::full-framework-upgrade : 6442 TPS, 6132 ms latency, 9300 ms p99 latency,no expired txns
5. check swarm health
Compatibility test for aptos-node-v1.3.0_3fc3d42b6cfe27460004f9a0326451bcda840a60 ==> c8188d1e4ccc085db2df74172f081239ec6aa6e4 passed
Test Ok

github-actions[bot] commented 1 year ago

Forge is running suite `framework_upgrade` on `aptos-node-v1.3.0_3fc3d42b6cfe27460004f9a0326451bcda840a60` ==> `9aa54866e864f01768ff29fea542ee88737abee1`

github-actions[bot] commented 1 year ago

Forge is running suite `land_blocking` on `9aa54866e864f01768ff29fea542ee88737abee1`

github-actions[bot] commented 1 year ago

Forge is running suite `compat` on `testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b` ==> `9aa54866e864f01768ff29fea542ee88737abee1`

github-actions[bot] commented 1 year ago

:x: Forge suite `compat` failure on `testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b` ==> `9aa54866e864f01768ff29fea542ee88737abee1`

Compatibility test results for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 9aa54866e864f01768ff29fea542ee88737abee1 (PR)
1. Check liveness of validators at old version: testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b
compatibility::simple-validator-upgrade::liveness-check : 8074 TPS, 4723 ms latency, 7100 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 9aa54866e864f01768ff29fea542ee88737abee1
Test Failed: Tried executing 10 txns, request counters: "success 0, failed submit [0, 1, 2, 4, 2, 6, 8, 6, 7, 6, 9, 9, 10, 10, 10, 10, 10, 10], failed wait [10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10], by client: [(0, 120, 180): http://aptos-node-1-validator.forge-compat-pr-7315.svc:8080/v1/]"

Caused by:
    Unknown error Transaction expired, without being seen in mempool. It is guaranteed it will not be committed on chain.

Stack backtrace:
   0: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/result.rs:2108:27
      aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiTransactionExecutor::submit_check_and_retry::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:120:9
   1: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
      <futures_util::future::maybe_done::MaybeDone<Fut> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.24/src/future/maybe_done.rs:95:38
      <futures_util::future::join_all::JoinAll<F> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.24/src/future/join_all.rs:142:24
   2: <aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiTransactionExecutor as aptos_transaction_generator_lib::TransactionExecutor>::execute_transactions_with_counter::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:216:9
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
   3: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/future.rs:124:9
      aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_and_fund_seed_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:304:17
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
      aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:182:13
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
   4: aptos_transaction_emitter_lib::emitter::TxnEmitter::start_job::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:568:13
   5: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
      aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for_impl::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:685:13
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
      aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:718:13
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
   6: tokio::park::thread::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/park/thread.rs:267:54
      tokio::coop::with_budget::{{closure}}
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:102:9
      std::thread::local::LocalKey<T>::try_with
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/thread/local.rs:446:16
      std::thread::local::LocalKey<T>::with
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/thread/local.rs:422:9
   7: tokio::coop::with_budget
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:95:5
      tokio::coop::budget
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:72:5
      tokio::park::thread::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/park/thread.rs:267:31
   8: tokio::runtime::enter::Enter::block_on
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/enter.rs:152:13
      tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/multi_thread/mod.rs:79:9
   9: tokio::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/mod.rs:492:44
  10: aptos_testcases::generate_traffic
             at ./testsuite/testcases/src/lib.rs:98:17
  11: <aptos_testcases::compatibility_test::SimpleValidatorUpgrade as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/compatibility_test.rs:80:24
  12: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:397:42
      aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:464:11
      aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:397:30
  13: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:334:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:261:21
  14: core::ops::function::FnOnce::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:251:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/sys_common/backtrace.rs:121:18
  15: std::rt::lang_start::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:166:18
  16: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:286:13
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
  17: main
  18: __libc_start_main
  19: _start
Trailing Log Lines:
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
  17: main
  18: __libc_start_main
  19: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:281"},"thread_name":"main","hostname":"forge-compat-pr-7315-1681174964-testnet-2d8b1b57553d869190f61df","timestamp":"2023-04-11T01:13:57.494551Z","message":"Deleting namespace forge-compat-pr-7315: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:389"},"thread_name":"main","hostname":"forge-compat-pr-7315-1681174964-testnet-2d8b1b57553d869190f61df","timestamp":"2023-04-11T01:13:57.494578Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-7315"}

failures:
    compatibility::simple-validator-upgrade

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed

Stack backtrace:
   0: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:417:17
   1: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:334:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:261:21
   2: core::ops::function::FnOnce::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:251:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/sys_common/backtrace.rs:121:18
   3: std::rt::lang_start::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:166:18
   4: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:286:13
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
   5: main
   6: __libc_start_main
   7: _start
Debugging output:

github-actions[bot] commented 1 year ago

:x: Forge suite `framework_upgrade` failure on `aptos-node-v1.3.0_3fc3d42b6cfe27460004f9a0326451bcda840a60` ==> `9aa54866e864f01768ff29fea542ee88737abee1`

Compatibility test results for aptos-node-v1.3.0_3fc3d42b6cfe27460004f9a0326451bcda840a60 ==> 9aa54866e864f01768ff29fea542ee88737abee1 (PR)
Upgrade the nodes to version: 9aa54866e864f01768ff29fea542ee88737abee1
Test Failed: Timed out waiting for Node validator-1:b2503d0ee606462cc28eecfe0f5bde94e4a4f45576237a702d18193502a456b6 to be healthy

Stack backtrace:
   0: aptos_forge::interface::node::NodeExt::wait_until_healthy::{{closure}}
             at ./testsuite/forge/src/interface/node.rs:235:13
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
   1: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/future.rs:124:9
      aptos_testcases::batch_update::{{closure}}
             at ./testsuite/testcases/src/lib.rs:58:13
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
   2: tokio::park::thread::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/park/thread.rs:267:54
      tokio::coop::with_budget::{{closure}}
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:102:9
      std::thread::local::LocalKey<T>::try_with
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/thread/local.rs:446:16
      std::thread::local::LocalKey<T>::with
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/thread/local.rs:422:9
   3: tokio::coop::with_budget
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:95:5
      tokio::coop::budget
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:72:5
      tokio::park::thread::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/park/thread.rs:267:31
   4: tokio::runtime::enter::Enter::block_on
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/enter.rs:152:13
      tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/multi_thread/mod.rs:79:9
   5: tokio::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/mod.rs:492:44
   6: <aptos_testcases::framework_upgrade::FrameworkUpgrade as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/framework_upgrade.rs:56:9
   7: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:397:42
      aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:464:11
      aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:397:30
   8: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:334:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:261:21
   9: core::ops::function::FnOnce::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:251:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/sys_common/backtrace.rs:121:18
  10: std::rt::lang_start::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:166:18
  11: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:286:13
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
  12: main
  13: __libc_start_main
  14: _start
Trailing Log Lines:
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
  12: main
  13: __libc_start_main
  14: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:281"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-7315-1681174955-aptos-node-v1-3-0-3f","timestamp":"2023-04-11T01:25:31.817378Z","message":"Deleting namespace forge-framework-upgrade-pr-7315: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:389"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-7315-1681174955-aptos-node-v1-3-0-3f","timestamp":"2023-04-11T01:25:31.817402Z","message":"aptos-node resources for Forge removed in namespace: forge-framework-upgrade-pr-7315"}

failures:
    framework_upgrade::framework-upgrade

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed

Stack backtrace:
   0: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:417:17
   1: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:334:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:261:21
   2: core::ops::function::FnOnce::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:251:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/sys_common/backtrace.rs:121:18
   3: std::rt::lang_start::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:166:18
   4: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:286:13
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
   5: main
   6: __libc_start_main
   7: _start
Debugging output:
NAME                                   READY   STATUS             RESTARTS        AGE
aptos-node-0-validator-0               0/1     CrashLoopBackOff   8 (65s ago)     19m
aptos-node-1-validator-0               0/1     CrashLoopBackOff   7 (4m48s ago)   17m
aptos-node-2-validator-0               0/1     CrashLoopBackOff   7 (65s ago)     16m
aptos-node-3-validator-0               0/1     CrashLoopBackOff   7 (64s ago)     14m
aptos-node-4-validator-0               0/1     CrashLoopBackOff   6 (4m48s ago)   13m
genesis-aptos-genesis-eforge68-4q8mb   0/1     Completed          0               22m

github-actions[bot] commented 1 year ago

:x: Forge suite `land_blocking` failure on `9aa54866e864f01768ff29fea542ee88737abee1`

three region simulation two traffics test : 540 TPS, 47 ms latency, 15900 ms p99 latency,(!) expired 397824 out of 864060 txns
Test Failed: check for success

Caused by:
    TPS requirement failed. Average TPS 540, minimum TPS requirement 900

Stack backtrace:
   0: aptos_forge::success_criteria::SuccessCriteriaChecker::check_for_success::{{closure}}
             at ./testsuite/forge/src/success_criteria.rs:97:13
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
   1: tokio::park::thread::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/park/thread.rs:267:54
      tokio::coop::with_budget::{{closure}}
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:102:9
      std::thread::local::LocalKey<T>::try_with
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/thread/local.rs:446:16
      std::thread::local::LocalKey<T>::with
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/thread/local.rs:422:9
   2: tokio::coop::with_budget
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:95:5
      tokio::coop::budget
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:72:5
      tokio::park::thread::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/park/thread.rs:267:31
   3: tokio::runtime::enter::Enter::block_on
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/enter.rs:152:13
      tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/multi_thread/mod.rs:79:9
   4: tokio::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/mod.rs:492:44
   5: aptos_forge::interface::network::NetworkContext::check_for_success
             at ./testsuite/forge/src/interface/network.rs:69:9
   6: <dyn aptos_testcases::NetworkLoadTest as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/lib.rs:190:9
   7: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:397:42
      aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:464:11
      aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:397:30
   8: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:334:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:261:21
   9: core::ops::function::FnOnce::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:251:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/sys_common/backtrace.rs:121:18
  10: std::rt::lang_start::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:166:18
  11: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:286:13
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
  12: main
  13: __libc_start_main
  14: _start
Trailing Log Lines:
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
  12: main
  13: __libc_start_main
  14: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:281"},"thread_name":"main","hostname":"forge-e2e-pr-7315-1681174957-9aa54866e864f01768ff29fea542ee8873","timestamp":"2023-04-11T01:26:38.547897Z","message":"Deleting namespace forge-e2e-pr-7315: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:389"},"thread_name":"main","hostname":"forge-e2e-pr-7315-1681174957-9aa54866e864f01768ff29fea542ee8873","timestamp":"2023-04-11T01:26:38.547932Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-7315"}

failures:
    three region simulation two traffics test

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed

Stack backtrace:
   0: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:417:17
   1: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:334:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:261:21
   2: core::ops::function::FnOnce::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:251:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/sys_common/backtrace.rs:121:18
   3: std::rt::lang_start::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:166:18
   4: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:286:13
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
   5: main
   6: __libc_start_main
   7: _start
Debugging output:
NAME                                   READY   STATUS      RESTARTS   AGE
aptos-node-14-validator-0              1/1     Running     0          20m
aptos-node-15-fullnode-eforge26-0      1/1     Running     0          20m
aptos-node-15-validator-0              1/1     Running     0          20m
aptos-node-16-fullnode-eforge26-0      1/1     Running     0          20m
aptos-node-16-validator-0              1/1     Running     0          20m
aptos-node-17-fullnode-eforge26-0      1/1     Running     0          20m
aptos-node-17-validator-0              1/1     Running     0          20m
aptos-node-18-fullnode-eforge26-0      1/1     Running     0          20m
aptos-node-18-validator-0              1/1     Running     0          20m
aptos-node-19-fullnode-eforge26-0      1/1     Running     0          20m
aptos-node-19-validator-0              1/1     Running     0          20m
aptos-node-2-fullnode-eforge26-0       1/1     Running     0          20m
aptos-node-2-validator-0               1/1     Running     0          20m
aptos-node-3-fullnode-eforge26-0       1/1     Running     0          20m
aptos-node-3-validator-0               1/1     Running     0          20m
aptos-node-4-fullnode-eforge26-0       1/1     Running     0          20m
aptos-node-4-validator-0               1/1     Running     0          20m
aptos-node-5-fullnode-eforge26-0       1/1     Running     0          20m
aptos-node-5-validator-0               1/1     Running     0          20m
aptos-node-6-fullnode-eforge26-0       1/1     Running     0          20m
aptos-node-6-validator-0               1/1     Running     0          20m
aptos-node-7-fullnode-eforge26-0       1/1     Running     0          20m
aptos-node-7-validator-0               1/1     Running     0          20m
aptos-node-8-fullnode-eforge26-0       1/1     Running     0          20m
aptos-node-8-validator-0               1/1     Running     0          20m
aptos-node-9-fullnode-eforge26-0       1/1     Running     0          20m
aptos-node-9-validator-0               1/1     Running     0          20m
genesis-aptos-genesis-eforge26-pc48l   0/1     Completed   0          23m

github-actions[bot] commented 1 year ago

Forge is running suite `compat` on `testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b` ==> `0170fabdcc43ef74c0b51ca98eefbe59de6afbef`

github-actions[bot] commented 1 year ago

Forge is running suite `land_blocking` on `0170fabdcc43ef74c0b51ca98eefbe59de6afbef`

github-actions[bot] commented 1 year ago

Forge is running suite `framework_upgrade` on `aptos-node-v1.3.0_3fc3d42b6cfe27460004f9a0326451bcda840a60` ==> `0170fabdcc43ef74c0b51ca98eefbe59de6afbef`

github-actions[bot] commented 1 year ago

:x: Forge suite `compat` failure on `testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b` ==> `0170fabdcc43ef74c0b51ca98eefbe59de6afbef`

Compatibility test results for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 0170fabdcc43ef74c0b51ca98eefbe59de6afbef (PR)
1. Check liveness of validators at old version: testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b
compatibility::simple-validator-upgrade::liveness-check : 7530 TPS, 5057 ms latency, 8400 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 0170fabdcc43ef74c0b51ca98eefbe59de6afbef
Test Failed: Tried executing 10 txns, request counters: "success 0, failed submit [0, 0, 1, 1, 6, 4, 5, 8, 6, 4, 9, 10, 9, 10, 9, 10, 10, 10], failed wait [10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10], by client: [(0, 112, 180): http://aptos-node-1-validator.forge-compat-pr-7315.svc:8080/v1/]"

Caused by:
    Unknown error Transaction expired, without being seen in mempool. It is guaranteed it will not be committed on chain.

Stack backtrace:
   0: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/result.rs:2108:27
      aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiTransactionExecutor::submit_check_and_retry::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:120:9
   1: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
      <futures_util::future::maybe_done::MaybeDone<Fut> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.24/src/future/maybe_done.rs:95:38
      <futures_util::future::join_all::JoinAll<F> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.24/src/future/join_all.rs:142:24
   2: <aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiTransactionExecutor as aptos_transaction_generator_lib::TransactionExecutor>::execute_transactions_with_counter::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:216:9
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
   3: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/future.rs:124:9
      aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_and_fund_seed_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:304:17
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
      aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:182:13
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
   4: aptos_transaction_emitter_lib::emitter::TxnEmitter::start_job::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:568:13
   5: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
      aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for_impl::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:685:13
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
      aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:718:13
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
   6: tokio::park::thread::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/park/thread.rs:267:54
      tokio::coop::with_budget::{{closure}}
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:102:9
      std::thread::local::LocalKey<T>::try_with
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/thread/local.rs:446:16
      std::thread::local::LocalKey<T>::with
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/thread/local.rs:422:9
   7: tokio::coop::with_budget
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:95:5
      tokio::coop::budget
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:72:5
      tokio::park::thread::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/park/thread.rs:267:31
   8: tokio::runtime::enter::Enter::block_on
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/enter.rs:152:13
      tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/multi_thread/mod.rs:79:9
   9: tokio::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/mod.rs:492:44
  10: aptos_testcases::generate_traffic
             at ./testsuite/testcases/src/lib.rs:98:17
  11: <aptos_testcases::compatibility_test::SimpleValidatorUpgrade as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/compatibility_test.rs:80:24
  12: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:397:42
      aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:464:11
      aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:397:30
  13: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:334:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:261:21
  14: core::ops::function::FnOnce::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:251:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/sys_common/backtrace.rs:121:18
  15: std::rt::lang_start::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:166:18
  16: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:286:13
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
  17: main
  18: __libc_start_main
  19: _start
Trailing Log Lines:
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
  17: main
  18: __libc_start_main
  19: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:282"},"thread_name":"main","hostname":"forge-compat-pr-7315-1681234515-testnet-2d8b1b57553d869190f61df","timestamp":"2023-04-11T17:48:32.638863Z","message":"Deleting namespace forge-compat-pr-7315: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:390"},"thread_name":"main","hostname":"forge-compat-pr-7315-1681234515-testnet-2d8b1b57553d869190f61df","timestamp":"2023-04-11T17:48:32.638885Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-7315"}

failures:
    compatibility::simple-validator-upgrade

Failed to run tests:
Tests Failed
test result: FAILED. 0 passed; 1 failed; 0 filtered out

Error: Tests Failed

Stack backtrace:
   0: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:417:17
   1: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:334:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:261:21
   2: core::ops::function::FnOnce::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:251:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/sys_common/backtrace.rs:121:18
   3: std::rt::lang_start::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:166:18
   4: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:286:13
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
   5: main
   6: __libc_start_main
   7: _start
Debugging output:
NAME                                   READY   STATUS      RESTARTS       AGE
aptos-node-0-validator-0               1/1     Running     0              9m46s
aptos-node-1-validator-0               0/1     Error       5 (109s ago)   5m17s
aptos-node-2-validator-0               1/1     Running     0              9m46s
aptos-node-3-validator-0               1/1     Running     0              9m46s
aptos-node-4-validator-0               1/1     Running     0              9m46s
genesis-aptos-genesis-eforge32-xv9hp   0/1     Completed   0              12m

github-actions[bot] commented 1 year ago

:x: Forge suite `land_blocking` failure on `0170fabdcc43ef74c0b51ca98eefbe59de6afbef`

Test Failed: test NetworkLoadTest

Caused by:
    TPS requirement for inner traffic failed. Average TPS 3234, minimum TPS requirement 3400. Full inner stats: TxnStatsRate { submitted: 12709, committed: 3234, expired: 9474, failed_submission: 0, latency: 59369, latency_samples: 35190, p50_latency: 64300, p90_latency: 66800, p99_latency: 69800 }

Stack backtrace:
   0: <aptos_testcases::two_traffics_test::TwoTrafficsTest as aptos_testcases::NetworkLoadTest>::test
             at ./testsuite/testcases/src/two_traffics_test.rs:77:13
   1: <aptos_testcases::two_traffics_test::ThreeRegionSimulationTwoTrafficsTest as aptos_testcases::NetworkLoadTest>::test
             at ./testsuite/testcases/src/two_traffics_test.rs:128:9
   2: <dyn aptos_testcases::NetworkLoadTest>::network_load_test
             at ./testsuite/testcases/src/lib.rs:275:13
   3: <dyn aptos_testcases::NetworkLoadTest as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/lib.rs:167:87
   4: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:397:42
      aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:464:11
      aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:397:30
   5: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:334:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:261:21
   6: core::ops::function::FnOnce::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:251:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/sys_common/backtrace.rs:121:18
   7: std::rt::lang_start::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:166:18
   8: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:286:13
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
   9: main
  10: __libc_start_main
  11: _start
Trailing Log Lines:
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
   9: main
  10: __libc_start_main
  11: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:282"},"thread_name":"main","hostname":"forge-e2e-pr-7315-1681234515-0170fabdcc43ef74c0b51ca98eefbe59de","timestamp":"2023-04-11T17:55:02.231170Z","message":"Deleting namespace forge-e2e-pr-7315: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:390"},"thread_name":"main","hostname":"forge-e2e-pr-7315-1681234515-0170fabdcc43ef74c0b51ca98eefbe59de","timestamp":"2023-04-11T17:55:02.231194Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-7315"}

failures:
    three region simulation two traffics test

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed

Stack backtrace:
   0: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:417:17
   1: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:334:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:261:21
   2: core::ops::function::FnOnce::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:251:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/sys_common/backtrace.rs:121:18
   3: std::rt::lang_start::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:166:18
   4: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:286:13
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
   5: main
   6: __libc_start_main
   7: _start
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-fullnode-eforge121-0       1/1     Running     0          16m
aptos-node-0-validator-0                1/1     Running     0          16m
aptos-node-1-fullnode-eforge121-0       1/1     Running     0          16m
aptos-node-1-validator-0                1/1     Running     0          16m
aptos-node-10-fullnode-eforge121-0      1/1     Running     0          16m
aptos-node-10-validator-0               1/1     Running     0          16m
aptos-node-11-fullnode-eforge121-0      1/1     Running     0          16m
aptos-node-11-validator-0               1/1     Running     0          16m
aptos-node-12-fullnode-eforge121-0      1/1     Running     0          16m
aptos-node-12-validator-0               1/1     Running     0          16m
aptos-node-13-fullnode-eforge121-0      1/1     Running     0          16m
aptos-node-13-validator-0               1/1     Running     0          16m
aptos-node-14-fullnode-eforge121-0      1/1     Running     0          16m
aptos-node-14-validator-0               1/1     Running     0          16m
aptos-node-15-fullnode-eforge121-0      1/1     Running     0          16m
aptos-node-15-validator-0               1/1     Running     0          16m
aptos-node-16-fullnode-eforge121-0      1/1     Running     0          16m
aptos-node-16-validator-0               1/1     Running     0          16m
aptos-node-17-fullnode-eforge121-0      1/1     Running     0          16m
aptos-node-17-validator-0               1/1     Running     0          16m
aptos-node-18-fullnode-eforge121-0      1/1     Running     0          16m
aptos-node-18-validator-0               1/1     Running     0          16m
aptos-node-19-fullnode-eforge121-0      1/1     Running     0          16m
aptos-node-19-validator-0               1/1     Running     0          16m
aptos-node-2-fullnode-eforge121-0       1/1     Running     0          16m
aptos-node-2-validator-0                1/1     Running     0          16m
aptos-node-3-fullnode-eforge121-0       1/1     Running     0          16m
aptos-node-3-validator-0                1/1     Running     0          16m
aptos-node-4-fullnode-eforge121-0       1/1     Running     0          16m
aptos-node-4-validator-0                1/1     Running     0          16m
aptos-node-5-fullnode-eforge121-0       1/1     Running     0          16m
aptos-node-5-validator-0                1/1     Running     0          16m
aptos-node-6-fullnode-eforge121-0       1/1     Running     0          16m
aptos-node-6-validator-0                1/1     Running     0          16m
aptos-node-7-fullnode-eforge121-0       1/1     Running     0          16m
aptos-node-7-validator-0                1/1     Running     0          16m
aptos-node-8-fullnode-eforge121-0       1/1     Running     0          16m
aptos-node-8-validator-0                1/1     Running     0          16m
aptos-node-9-fullnode-eforge121-0       1/1     Running     0          16m
aptos-node-9-validator-0                1/1     Running     0          16m
genesis-aptos-genesis-eforge121-x6bnl   0/1     Completed   0          19m

github-actions[bot] commented 1 year ago

:x: Forge suite `framework_upgrade` failure on `aptos-node-v1.3.0_3fc3d42b6cfe27460004f9a0326451bcda840a60` ==> `0170fabdcc43ef74c0b51ca98eefbe59de6afbef`

Compatibility test results for aptos-node-v1.3.0_3fc3d42b6cfe27460004f9a0326451bcda840a60 ==> 0170fabdcc43ef74c0b51ca98eefbe59de6afbef (PR)
Upgrade the nodes to version: 0170fabdcc43ef74c0b51ca98eefbe59de6afbef
Test Failed: Timed out waiting for Node validator-2:0ba542611668e470163f9b320b66cf7e4d06b87f9eb8f29a1826b94a3155f918 to be healthy

Stack backtrace:
   0: aptos_forge::interface::node::NodeExt::wait_until_healthy::{{closure}}
             at ./testsuite/forge/src/interface/node.rs:235:13
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
   1: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/future.rs:124:9
      aptos_testcases::batch_update::{{closure}}
             at ./testsuite/testcases/src/lib.rs:58:13
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
   2: tokio::park::thread::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/park/thread.rs:267:54
      tokio::coop::with_budget::{{closure}}
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:102:9
      std::thread::local::LocalKey<T>::try_with
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/thread/local.rs:446:16
      std::thread::local::LocalKey<T>::with
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/thread/local.rs:422:9
   3: tokio::coop::with_budget
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:95:5
      tokio::coop::budget
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:72:5
      tokio::park::thread::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/park/thread.rs:267:31
   4: tokio::runtime::enter::Enter::block_on
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/enter.rs:152:13
      tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/multi_thread/mod.rs:79:9
   5: tokio::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/mod.rs:492:44
   6: <aptos_testcases::framework_upgrade::FrameworkUpgrade as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/framework_upgrade.rs:56:9
   7: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:397:42
      aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:464:11
      aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:397:30
   8: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:334:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:261:21
   9: core::ops::function::FnOnce::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:251:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/sys_common/backtrace.rs:121:18
  10: std::rt::lang_start::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:166:18
  11: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:286:13
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
  12: main
  13: __libc_start_main
  14: _start
Trailing Log Lines:
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
  12: main
  13: __libc_start_main
  14: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:282"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-7315-1681234517-aptos-node-v1-3-0-3f","timestamp":"2023-04-11T18:08:38.883794Z","message":"Deleting namespace forge-framework-upgrade-pr-7315: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:390"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-7315-1681234517-aptos-node-v1-3-0-3f","timestamp":"2023-04-11T18:08:38.883817Z","message":"aptos-node resources for Forge removed in namespace: forge-framework-upgrade-pr-7315"}

failures:
    framework_upgrade::framework-upgrade

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed

Stack backtrace:
   0: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:417:17
   1: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:334:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:261:21
   2: core::ops::function::FnOnce::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:251:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/sys_common/backtrace.rs:121:18
   3: std::rt::lang_start::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:166:18
   4: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:286:13
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
   5: main
   6: __libc_start_main
   7: _start
Debugging output:
NAME                                   READY   STATUS             RESTARTS        AGE
aptos-node-0-validator-0               0/1     CrashLoopBackOff   9 (63s ago)     25m
aptos-node-1-validator-0               0/1     CrashLoopBackOff   9 (63s ago)     24m
aptos-node-2-validator-0               0/1     CrashLoopBackOff   8 (4m40s ago)   23m
aptos-node-3-validator-0               0/1     CrashLoopBackOff   8 (63s ago)     21m
aptos-node-4-validator-0               0/1     CrashLoopBackOff   7 (4m40s ago)   20m
genesis-aptos-genesis-eforge28-ksdvk   0/1     Completed          0               32m

github-actions[bot] commented 1 year ago

Forge is running suite `compat` on `testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b` ==> `b9e5f29761349a665636206465907d9181d05f30`

github-actions[bot] commented 1 year ago

Forge is running suite `land_blocking` on `b9e5f29761349a665636206465907d9181d05f30`

github-actions[bot] commented 1 year ago

Forge is running suite `framework_upgrade` on `aptos-node-v1.3.0_3fc3d42b6cfe27460004f9a0326451bcda840a60` ==> `b9e5f29761349a665636206465907d9181d05f30`

github-actions[bot] commented 1 year ago

:x: Forge suite `framework_upgrade` failure on `aptos-node-v1.3.0_3fc3d42b6cfe27460004f9a0326451bcda840a60` ==> `b9e5f29761349a665636206465907d9181d05f30`

Compatibility test results for aptos-node-v1.3.0_3fc3d42b6cfe27460004f9a0326451bcda840a60 ==> b9e5f29761349a665636206465907d9181d05f30 (PR)
Upgrade the nodes to version: b9e5f29761349a665636206465907d9181d05f30
Test Failed: CrashLoopBackOff

Stack backtrace:
   0: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/result.rs:2108:27
      aptos_forge::backend::k8s::stateful_set::wait_stateful_set::{{closure}}
             at ./testsuite/forge/src/backend/k8s/stateful_set.rs:69:5
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
   1: aptos_forge::backend::k8s::stateful_set::scale_stateful_set_replicas::{{closure}}
             at ./testsuite/forge/src/backend/k8s/stateful_set.rs:243:5
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
   2: <aptos_forge::backend::k8s::node::K8sNode as aptos_forge::interface::node::Node>::start::{{closure}}
             at ./testsuite/forge/src/backend/k8s/node.rs:138:83
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
   3: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/future.rs:124:9
      <aptos_forge::backend::k8s::swarm::K8sSwarm as aptos_forge::interface::swarm::Swarm>::upgrade_validator::{{closure}}
             at ./testsuite/forge/src/backend/k8s/swarm.rs:198:26
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
   4: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/future.rs:124:9
      aptos_testcases::batch_update::{{closure}}
             at ./testsuite/testcases/src/lib.rs:48:59
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
   5: tokio::park::thread::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/park/thread.rs:267:54
      tokio::coop::with_budget::{{closure}}
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:102:9
      std::thread::local::LocalKey<T>::try_with
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/thread/local.rs:446:16
      std::thread::local::LocalKey<T>::with
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/thread/local.rs:422:9
   6: tokio::coop::with_budget
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:95:5
      tokio::coop::budget
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:72:5
      tokio::park::thread::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/park/thread.rs:267:31
   7: tokio::runtime::enter::Enter::block_on
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/enter.rs:152:13
      tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/multi_thread/mod.rs:79:9
   8: tokio::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/mod.rs:492:44
   9: <aptos_testcases::framework_upgrade::FrameworkUpgrade as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/framework_upgrade.rs:56:9
  10: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:397:42
      aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:464:11
      aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:397:30
  11: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:334:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:261:21
  12: core::ops::function::FnOnce::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:251:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/sys_common/backtrace.rs:121:18
  13: std::rt::lang_start::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:166:18
  14: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:286:13
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
  15: main
  16: __libc_start_main
  17: _start
Trailing Log Lines:
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
  15: main
  16: __libc_start_main
  17: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:282"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-7315-1681246853-aptos-node-v1-3-0-3f","timestamp":"2023-04-11T21:12:39.265686Z","message":"Deleting namespace forge-framework-upgrade-pr-7315: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:390"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-7315-1681246853-aptos-node-v1-3-0-3f","timestamp":"2023-04-11T21:12:39.265712Z","message":"aptos-node resources for Forge removed in namespace: forge-framework-upgrade-pr-7315"}

failures:
    framework_upgrade::framework-upgrade

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed

Stack backtrace:
   0: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:417:17
   1: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:334:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:261:21
   2: core::ops::function::FnOnce::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:251:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/sys_common/backtrace.rs:121:18
   3: std::rt::lang_start::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:166:18
   4: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:286:13
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
   5: main
   6: __libc_start_main
   7: _start
Debugging output:
NAME                                   READY   STATUS             RESTARTS      AGE
aptos-node-0-validator-0               0/1     CrashLoopBackOff   4 (29s ago)   3m24s
aptos-node-1-validator-0               0/1     CrashLoopBackOff   2 (25s ago)   2m4s
aptos-node-2-validator-0               1/1     Running            0             6m49s
aptos-node-3-validator-0               1/1     Running            0             6m49s
aptos-node-4-validator-0               1/1     Running            0             6m49s
genesis-aptos-genesis-eforge59-89qrn   0/1     Completed          0             9m17s

github-actions[bot] commented 1 year ago

:x: Forge suite `compat` failure on `testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b` ==> `b9e5f29761349a665636206465907d9181d05f30`

Compatibility test results for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> b9e5f29761349a665636206465907d9181d05f30 (PR)
1. Check liveness of validators at old version: testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b
compatibility::simple-validator-upgrade::liveness-check : 7759 TPS, 4937 ms latency, 7300 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: b9e5f29761349a665636206465907d9181d05f30
Test Failed: Tried executing 10 txns, request counters: "success 0, failed submit [0, 0, 1, 0, 3, 2, 9, 6, 9, 6, 8, 10, 10, 10, 9, 10, 10, 10], failed wait [10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10], by client: [(0, 113, 180): http://aptos-node-1-validator.forge-compat-pr-7315.svc:8080/v1/]"

Caused by:
    Unknown error Transaction expired, without being seen in mempool. It is guaranteed it will not be committed on chain.

Stack backtrace:
   0: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/result.rs:2108:27
      aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiTransactionExecutor::submit_check_and_retry::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:120:9
   1: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
      <futures_util::future::maybe_done::MaybeDone<Fut> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.24/src/future/maybe_done.rs:95:38
      <futures_util::future::join_all::JoinAll<F> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.24/src/future/join_all.rs:142:24
   2: <aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiTransactionExecutor as aptos_transaction_generator_lib::TransactionExecutor>::execute_transactions_with_counter::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:216:9
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
   3: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/future.rs:124:9
      aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_and_fund_seed_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:304:17
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
      aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:182:13
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
   4: aptos_transaction_emitter_lib::emitter::TxnEmitter::start_job::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:568:13
   5: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
      aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for_impl::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:685:13
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
      aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:718:13
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
   6: tokio::park::thread::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/park/thread.rs:267:54
      tokio::coop::with_budget::{{closure}}
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:102:9
      std::thread::local::LocalKey<T>::try_with
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/thread/local.rs:446:16
      std::thread::local::LocalKey<T>::with
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/thread/local.rs:422:9
   7: tokio::coop::with_budget
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:95:5
      tokio::coop::budget
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:72:5
      tokio::park::thread::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/park/thread.rs:267:31
   8: tokio::runtime::enter::Enter::block_on
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/enter.rs:152:13
      tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/multi_thread/mod.rs:79:9
   9: tokio::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/mod.rs:492:44
  10: aptos_testcases::generate_traffic
             at ./testsuite/testcases/src/lib.rs:98:17
  11: <aptos_testcases::compatibility_test::SimpleValidatorUpgrade as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/compatibility_test.rs:80:24
  12: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:397:42
      aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:464:11
      aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:397:30
  13: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:334:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:261:21
  14: core::ops::function::FnOnce::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:251:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/sys_common/backtrace.rs:121:18
  15: std::rt::lang_start::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:166:18
  16: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:286:13
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
  17: main
  18: __libc_start_main
  19: _start
Trailing Log Lines:
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
  17: main
  18: __libc_start_main
  19: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:282"},"thread_name":"main","hostname":"forge-compat-pr-7315-1681246842-testnet-2d8b1b57553d869190f61df","timestamp":"2023-04-11T21:13:08.334504Z","message":"Deleting namespace forge-compat-pr-7315: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:390"},"thread_name":"main","hostname":"forge-compat-pr-7315-1681246842-testnet-2d8b1b57553d869190f61df","timestamp":"2023-04-11T21:13:08.334526Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-7315"}

failures:
    compatibility::simple-validator-upgrade

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed

Stack backtrace:
   0: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:417:17
   1: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:334:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:261:21
   2: core::ops::function::FnOnce::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:251:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/sys_common/backtrace.rs:121:18
   3: std::rt::lang_start::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:166:18
   4: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:286:13
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
   5: main
   6: __libc_start_main
   7: _start
Debugging output:

github-actions[bot] commented 1 year ago

:x: Forge suite `land_blocking` failure on `b9e5f29761349a665636206465907d9181d05f30`

Test Failed: test NetworkLoadTest

Caused by:
    TPS requirement for inner traffic failed. Average TPS 3252, minimum TPS requirement 3400. Full inner stats: TxnStatsRate { submitted: 12783, committed: 3252, expired: 9531, failed_submission: 0, latency: 59998, latency_samples: 39053, p50_latency: 64300, p90_latency: 66200, p99_latency: 68500 }

Stack backtrace:
   0: <aptos_testcases::two_traffics_test::TwoTrafficsTest as aptos_testcases::NetworkLoadTest>::test
             at ./testsuite/testcases/src/two_traffics_test.rs:77:13
   1: <aptos_testcases::two_traffics_test::ThreeRegionSimulationTwoTrafficsTest as aptos_testcases::NetworkLoadTest>::test
             at ./testsuite/testcases/src/two_traffics_test.rs:128:9
   2: <dyn aptos_testcases::NetworkLoadTest>::network_load_test
             at ./testsuite/testcases/src/lib.rs:275:13
   3: <dyn aptos_testcases::NetworkLoadTest as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/lib.rs:167:87
   4: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:397:42
      aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:464:11
      aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:397:30
   5: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:334:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:261:21
   6: core::ops::function::FnOnce::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:251:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/sys_common/backtrace.rs:121:18
   7: std::rt::lang_start::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:166:18
   8: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:286:13
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
   9: main
  10: __libc_start_main
  11: _start
Trailing Log Lines:
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
   9: main
  10: __libc_start_main
  11: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:282"},"thread_name":"main","hostname":"forge-e2e-pr-7315-1681246842-b9e5f29761349a665636206465907d9181","timestamp":"2023-04-11T21:22:11.248820Z","message":"Deleting namespace forge-e2e-pr-7315: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:390"},"thread_name":"main","hostname":"forge-e2e-pr-7315-1681246842-b9e5f29761349a665636206465907d9181","timestamp":"2023-04-11T21:22:11.248853Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-7315"}

failures:
    three region simulation two traffics test

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed

Stack backtrace:
   0: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:417:17
   1: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:334:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:261:21
   2: core::ops::function::FnOnce::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:251:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/sys_common/backtrace.rs:121:18
   3: std::rt::lang_start::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:166:18
   4: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:286:13
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
   5: main
   6: __libc_start_main
   7: _start
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-13-fullnode-eforge212-0      1/1     Running     0          16m
aptos-node-13-validator-0               1/1     Running     0          16m
aptos-node-14-fullnode-eforge212-0      1/1     Running     0          16m
aptos-node-14-validator-0               1/1     Running     0          16m
aptos-node-15-fullnode-eforge212-0      1/1     Running     0          16m
aptos-node-15-validator-0               1/1     Running     0          16m
aptos-node-16-fullnode-eforge212-0      1/1     Running     0          16m
aptos-node-16-validator-0               1/1     Running     0          16m
aptos-node-17-fullnode-eforge212-0      1/1     Running     0          16m
aptos-node-17-validator-0               1/1     Running     0          16m
aptos-node-18-fullnode-eforge212-0      1/1     Running     0          16m
aptos-node-18-validator-0               1/1     Running     0          16m
aptos-node-19-fullnode-eforge212-0      1/1     Running     0          16m
aptos-node-19-validator-0               1/1     Running     0          16m
aptos-node-2-fullnode-eforge212-0       1/1     Running     0          16m
aptos-node-2-validator-0                1/1     Running     0          16m
aptos-node-3-fullnode-eforge212-0       1/1     Running     0          16m
aptos-node-3-validator-0                1/1     Running     0          16m
aptos-node-4-fullnode-eforge212-0       1/1     Running     0          16m
aptos-node-4-validator-0                1/1     Running     0          16m
aptos-node-5-fullnode-eforge212-0       1/1     Running     0          16m
aptos-node-5-validator-0                1/1     Running     0          16m
aptos-node-6-fullnode-eforge212-0       1/1     Running     0          16m
aptos-node-6-validator-0                1/1     Running     0          16m
aptos-node-7-fullnode-eforge212-0       1/1     Running     0          16m
aptos-node-7-validator-0                1/1     Running     0          16m
aptos-node-8-fullnode-eforge212-0       1/1     Running     0          16m
aptos-node-8-validator-0                1/1     Running     0          16m
aptos-node-9-fullnode-eforge212-0       1/1     Running     0          16m
aptos-node-9-validator-0                1/1     Running     0          16m
genesis-aptos-genesis-eforge212-nm9x6   0/1     Completed   0          19m

github-actions[bot] commented 1 year ago

Forge is running suite `land_blocking` on `cee6fd049ee4a0fd8b3e8f87873719487fee769b`

github-actions[bot] commented 1 year ago

Forge is running suite `compat` on `testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b` ==> `cee6fd049ee4a0fd8b3e8f87873719487fee769b`

github-actions[bot] commented 1 year ago

Forge is running suite `framework_upgrade` on `aptos-node-v1.3.0_3fc3d42b6cfe27460004f9a0326451bcda840a60` ==> `cee6fd049ee4a0fd8b3e8f87873719487fee769b`

github-actions[bot] commented 1 year ago

:x: Forge suite `compat` failure on `testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b` ==> `cee6fd049ee4a0fd8b3e8f87873719487fee769b`

Compatibility test results for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> cee6fd049ee4a0fd8b3e8f87873719487fee769b (PR)
1. Check liveness of validators at old version: testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b
compatibility::simple-validator-upgrade::liveness-check : 7860 TPS, 4972 ms latency, 6800 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: cee6fd049ee4a0fd8b3e8f87873719487fee769b
Test Failed: Tried executing 10 txns, request counters: "success 0, failed submit [0, 0, 1, 1, 2, 2, 8, 9, 8, 5, 6, 10, 10, 9, 10, 10, 10, 10], failed wait [10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10], by client: [(0, 111, 180): http://aptos-node-1-validator.forge-compat-pr-7315.svc:8080/v1/]"

Caused by:
    Unknown error Transaction expired, without being seen in mempool. It is guaranteed it will not be committed on chain.

Stack backtrace:
   0: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/result.rs:2108:27
      aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiTransactionExecutor::submit_check_and_retry::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:120:9
   1: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
      <futures_util::future::maybe_done::MaybeDone<Fut> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.24/src/future/maybe_done.rs:95:38
      <futures_util::future::join_all::JoinAll<F> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.24/src/future/join_all.rs:142:24
   2: <aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiTransactionExecutor as aptos_transaction_generator_lib::TransactionExecutor>::execute_transactions_with_counter::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:216:9
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
   3: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/future.rs:124:9
      aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_and_fund_seed_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:304:17
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
      aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:182:13
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
   4: aptos_transaction_emitter_lib::emitter::TxnEmitter::start_job::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:568:13
   5: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
      aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for_impl::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:685:13
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
      aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:718:13
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
   6: tokio::park::thread::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/park/thread.rs:267:54
      tokio::coop::with_budget::{{closure}}
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:102:9
      std::thread::local::LocalKey<T>::try_with
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/thread/local.rs:446:16
      std::thread::local::LocalKey<T>::with
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/thread/local.rs:422:9
   7: tokio::coop::with_budget
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:95:5
      tokio::coop::budget
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:72:5
      tokio::park::thread::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/park/thread.rs:267:31
   8: tokio::runtime::enter::Enter::block_on
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/enter.rs:152:13
      tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/multi_thread/mod.rs:79:9
   9: tokio::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/mod.rs:492:44
  10: aptos_testcases::generate_traffic
             at ./testsuite/testcases/src/lib.rs:98:17
  11: <aptos_testcases::compatibility_test::SimpleValidatorUpgrade as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/compatibility_test.rs:80:24
  12: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:397:42
      aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:464:11
      aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:397:30
  13: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:334:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:261:21
  14: core::ops::function::FnOnce::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:251:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/sys_common/backtrace.rs:121:18
  15: std::rt::lang_start::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:166:18
  16: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:286:13
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
  17: main
  18: __libc_start_main
  19: _start
Trailing Log Lines:
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
  17: main
  18: __libc_start_main
  19: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:282"},"thread_name":"main","hostname":"forge-compat-pr-7315-1681252331-testnet-2d8b1b57553d869190f61df","timestamp":"2023-04-11T22:45:25.925131Z","message":"Deleting namespace forge-compat-pr-7315: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:390"},"thread_name":"main","hostname":"forge-compat-pr-7315-1681252331-testnet-2d8b1b57553d869190f61df","timestamp":"2023-04-11T22:45:25.925155Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-7315"}

failures:
    compatibility::simple-validator-upgrade

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed

Stack backtrace:
   0: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:417:17
   1: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:334:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:261:21
   2: core::ops::function::FnOnce::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:251:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/sys_common/backtrace.rs:121:18
   3: std::rt::lang_start::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:166:18
   4: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:286:13
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
   5: main
   6: __libc_start_main
   7: _start
Debugging output:
NAME                                   READY   STATUS      RESTARTS       AGE
aptos-node-0-validator-0               1/1     Running     0              10m
aptos-node-1-validator-0               0/1     Error       5 (104s ago)   5m8s
aptos-node-2-validator-0               1/1     Running     0              10m
aptos-node-3-validator-0               1/1     Running     0              10m
aptos-node-4-validator-0               1/1     Running     0              10m
genesis-aptos-genesis-eforge78-z4bg2   0/1     Completed   0              10m

github-actions[bot] commented 1 year ago

:x: Forge suite `land_blocking` failure on `cee6fd049ee4a0fd8b3e8f87873719487fee769b`

Test Failed: test NetworkLoadTest

Caused by:
    TPS requirement for inner traffic failed. Average TPS 3248, minimum TPS requirement 3400. Full inner stats: TxnStatsRate { submitted: 12863, committed: 3248, expired: 9615, failed_submission: 0, latency: 58857, latency_samples: 34107, p50_latency: 64300, p90_latency: 66600, p99_latency: 70000 }

Stack backtrace:
   0: <aptos_testcases::two_traffics_test::TwoTrafficsTest as aptos_testcases::NetworkLoadTest>::test
             at ./testsuite/testcases/src/two_traffics_test.rs:77:13
   1: <aptos_testcases::two_traffics_test::ThreeRegionSimulationTwoTrafficsTest as aptos_testcases::NetworkLoadTest>::test
             at ./testsuite/testcases/src/two_traffics_test.rs:128:9
   2: <dyn aptos_testcases::NetworkLoadTest>::network_load_test
             at ./testsuite/testcases/src/lib.rs:275:13
   3: <dyn aptos_testcases::NetworkLoadTest as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/lib.rs:167:87
   4: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:397:42
      aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:464:11
      aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:397:30
   5: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:334:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:261:21
   6: core::ops::function::FnOnce::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:251:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/sys_common/backtrace.rs:121:18
   7: std::rt::lang_start::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:166:18
   8: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:286:13
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
   9: main
  10: __libc_start_main
  11: _start
Trailing Log Lines:
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
   9: main
  10: __libc_start_main
  11: _start

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:282"},"thread_name":"main","hostname":"forge-e2e-pr-7315-1681252325-cee6fd049ee4a0fd8b3e8f87873719487f","timestamp":"2023-04-11T22:51:30.344832Z","message":"Deleting namespace forge-e2e-pr-7315: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:390"},"thread_name":"main","hostname":"forge-e2e-pr-7315-1681252325-cee6fd049ee4a0fd8b3e8f87873719487f","timestamp":"2023-04-11T22:51:30.344854Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-7315"}

failures:
    three region simulation two traffics test

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed

Stack backtrace:
   0: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:417:17
   1: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:334:11
      forge::main
             at ./testsuite/forge-cli/src/main.rs:261:21
   2: core::ops::function::FnOnce::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:251:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/sys_common/backtrace.rs:121:18
   3: std::rt::lang_start::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:166:18
   4: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:286:13
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
   5: main
   6: __libc_start_main
   7: _start
Debugging output:
NAME                                   READY   STATUS      RESTARTS   AGE
aptos-node-0-fullnode-eforge34-0       1/1     Running     0          15m
aptos-node-0-validator-0               1/1     Running     0          15m
aptos-node-1-fullnode-eforge34-0       1/1     Running     0          15m
aptos-node-1-validator-0               1/1     Running     0          15m
aptos-node-10-fullnode-eforge34-0      1/1     Running     0          15m
aptos-node-10-validator-0              1/1     Running     0          15m
aptos-node-11-fullnode-eforge34-0      1/1     Running     0          15m
aptos-node-11-validator-0              1/1     Running     0          15m
aptos-node-12-fullnode-eforge34-0      1/1     Running     0          15m
aptos-node-12-validator-0              1/1     Running     0          15m
aptos-node-13-fullnode-eforge34-0      1/1     Running     0          15m
aptos-node-13-validator-0              1/1     Running     0          15m
aptos-node-14-fullnode-eforge34-0      1/1     Running     0          15m
aptos-node-14-validator-0              1/1     Running     0          15m
aptos-node-15-fullnode-eforge34-0      1/1     Running     0          15m
aptos-node-15-validator-0              1/1     Running     0          15m
aptos-node-16-fullnode-eforge34-0      1/1     Running     0          15m
aptos-node-16-validator-0              1/1     Running     0          15m
aptos-node-17-fullnode-eforge34-0      1/1     Running     0          15m
aptos-node-17-validator-0              1/1     Running     0          15m
aptos-node-18-fullnode-eforge34-0      1/1     Running     0          15m
aptos-node-18-validator-0              1/1     Running     0          15m
aptos-node-19-fullnode-eforge34-0      1/1     Running     0          15m
aptos-node-19-validator-0              1/1     Running     0          15m
aptos-node-2-fullnode-eforge34-0       1/1     Running     0          15m
aptos-node-2-validator-0               1/1     Running     0          15m
aptos-node-3-fullnode-eforge34-0       1/1     Running     0          15m
aptos-node-3-validator-0               1/1     Running     0          15m
aptos-node-4-fullnode-eforge34-0       1/1     Running     0          15m
aptos-node-4-validator-0               1/1     Running     0          15m
aptos-node-5-fullnode-eforge34-0       1/1     Running     0          15m
aptos-node-5-validator-0               1/1     Running     0          15m
aptos-node-6-fullnode-eforge34-0       1/1     Running     0          15m
aptos-node-6-validator-0               1/1     Running     0          15m
aptos-node-7-fullnode-eforge34-0       1/1     Running     0          15m
aptos-node-7-validator-0               1/1     Running     0          15m
aptos-node-8-fullnode-eforge34-0       1/1     Running     0          15m
aptos-node-8-validator-0               1/1     Running     0          15m
aptos-node-9-fullnode-eforge34-0       1/1     Running     0          15m
aptos-node-9-validator-0               1/1     Running     0          15m
genesis-aptos-genesis-eforge34-z48vt   0/1     Completed   0          18m

github-actions[bot] commented 1 year ago

Forge is running suite `framework_upgrade` on `aptos-node-v1.3.0_3fc3d42b6cfe27460004f9a0326451bcda840a60` ==> `01eaa0f51e1b22e5625632d6bea67edc8abdcfa5`

Next