search

aptos-labs / aptos-core

Aptos is a layer 1 blockchain built to support the widespread use of blockchain through better technology and user experience.
https://aptosfoundation.org
Other
5.86k stars 3.54k forks source link

[forge] Add a test for validation set changes #4279

Closed zcchahaha closed 1 year ago

zcchahaha commented 1 year ago

Description

This change is Reviewable

sitalkedia commented 1 year ago

@zcchahaha - Is this ready for review?

zcchahaha commented 1 year ago

@sitalkedia Not yet. Still try to figure out some dependencies issues. https://aptos-org.slack.com/archives/C036SHTFHGR/p1663364319592519

github-actions[bot] commented 1 year ago

Forge is running suite compat on 843b204dce971d98449b82624f4f684c7a18b991 ==> 7a4cb71965e35c676b95e3b5a9c192d348343b54

github-actions[bot] commented 1 year ago

Forge is running suite land_blocking on 7a4cb71965e35c676b95e3b5a9c192d348343b54

github-actions[bot] commented 1 year ago

:white_check_mark: Forge suite land_blocking success on 7a4cb71965e35c676b95e3b5a9c192d348343b54

performance benchmark with full nodes : 7493 TPS, 5294 ms latency, 9300 ms p99 latency,(!) expired 180 out of 3237360 txns
Test Ok
github-actions[bot] commented 1 year ago

:white_check_mark: Forge suite compat success on 843b204dce971d98449b82624f4f684c7a18b991 ==> 7a4cb71965e35c676b95e3b5a9c192d348343b54

Compatibility test results for 843b204dce971d98449b82624f4f684c7a18b991 ==> 7a4cb71965e35c676b95e3b5a9c192d348343b54 (PR)
1. Check liveness of validators at old version: 843b204dce971d98449b82624f4f684c7a18b991
compatibility::simple-validator-upgrade::liveness-check : 7994 TPS, 4751 ms latency, 6600 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 7a4cb71965e35c676b95e3b5a9c192d348343b54
compatibility::simple-validator-upgrade::single-validator-upgrade : 5676 TPS, 6828 ms latency, 8500 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: 7a4cb71965e35c676b95e3b5a9c192d348343b54
compatibility::simple-validator-upgrade::half-validator-upgrade : 5114 TPS, 7440 ms latency, 9600 ms p99 latency,no expired txns
4. upgrading second batch to new version: 7a4cb71965e35c676b95e3b5a9c192d348343b54
compatibility::simple-validator-upgrade::rest-validator-upgrade : 7898 TPS, 4431 ms latency, 7600 ms p99 latency,no expired txns
5. check swarm health
Compatibility test for 843b204dce971d98449b82624f4f684c7a18b991 ==> 7a4cb71965e35c676b95e3b5a9c192d348343b54 passed
Test Ok
github-actions[bot] commented 1 year ago

Forge is running suite compat on 843b204dce971d98449b82624f4f684c7a18b991 ==> 023620c51293dd9d027d24e179ac4aa34ebbeeb0

github-actions[bot] commented 1 year ago

Forge is running suite land_blocking on 023620c51293dd9d027d24e179ac4aa34ebbeeb0

github-actions[bot] commented 1 year ago

:white_check_mark: Forge suite land_blocking success on 023620c51293dd9d027d24e179ac4aa34ebbeeb0

performance benchmark with full nodes : 7425 TPS, 5341 ms latency, 7200 ms p99 latency,(!) expired 20 out of 3208000 txns
Test Ok
github-actions[bot] commented 1 year ago

:white_check_mark: Forge suite compat success on 843b204dce971d98449b82624f4f684c7a18b991 ==> 023620c51293dd9d027d24e179ac4aa34ebbeeb0

Compatibility test results for 843b204dce971d98449b82624f4f684c7a18b991 ==> 023620c51293dd9d027d24e179ac4aa34ebbeeb0 (PR)
1. Check liveness of validators at old version: 843b204dce971d98449b82624f4f684c7a18b991
compatibility::simple-validator-upgrade::liveness-check : 7334 TPS, 5011 ms latency, 7500 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 023620c51293dd9d027d24e179ac4aa34ebbeeb0
compatibility::simple-validator-upgrade::single-validator-upgrade : 5250 TPS, 7288 ms latency, 8800 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: 023620c51293dd9d027d24e179ac4aa34ebbeeb0
compatibility::simple-validator-upgrade::half-validator-upgrade : 5696 TPS, 6517 ms latency, 8700 ms p99 latency,no expired txns
4. upgrading second batch to new version: 023620c51293dd9d027d24e179ac4aa34ebbeeb0
compatibility::simple-validator-upgrade::rest-validator-upgrade : 7402 TPS, 4913 ms latency, 8500 ms p99 latency,no expired txns
5. check swarm health
Compatibility test for 843b204dce971d98449b82624f4f684c7a18b991 ==> 023620c51293dd9d027d24e179ac4aa34ebbeeb0 passed
Test Ok
github-actions[bot] commented 1 year ago

Forge is running suite compat on 843b204dce971d98449b82624f4f684c7a18b991 ==> 137eb93230419b38bce9436f5ab799d8e529c4b7

github-actions[bot] commented 1 year ago

Forge is running suite land_blocking on 137eb93230419b38bce9436f5ab799d8e529c4b7

github-actions[bot] commented 1 year ago

:white_check_mark: Forge suite land_blocking success on 137eb93230419b38bce9436f5ab799d8e529c4b7

performance benchmark with full nodes : 7805 TPS, 5091 ms latency, 7200 ms p99 latency,no expired txns
Test Ok
github-actions[bot] commented 1 year ago

:white_check_mark: Forge suite compat success on 843b204dce971d98449b82624f4f684c7a18b991 ==> 137eb93230419b38bce9436f5ab799d8e529c4b7

Compatibility test results for 843b204dce971d98449b82624f4f684c7a18b991 ==> 137eb93230419b38bce9436f5ab799d8e529c4b7 (PR)
1. Check liveness of validators at old version: 843b204dce971d98449b82624f4f684c7a18b991
compatibility::simple-validator-upgrade::liveness-check : 7798 TPS, 4759 ms latency, 6600 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 137eb93230419b38bce9436f5ab799d8e529c4b7
compatibility::simple-validator-upgrade::single-validator-upgrade : 5626 TPS, 6878 ms latency, 9600 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: 137eb93230419b38bce9436f5ab799d8e529c4b7
compatibility::simple-validator-upgrade::half-validator-upgrade : 5794 TPS, 6575 ms latency, 10200 ms p99 latency,no expired txns
4. upgrading second batch to new version: 137eb93230419b38bce9436f5ab799d8e529c4b7
compatibility::simple-validator-upgrade::rest-validator-upgrade : 7791 TPS, 4601 ms latency, 7900 ms p99 latency,no expired txns
5. check swarm health
Compatibility test for 843b204dce971d98449b82624f4f684c7a18b991 ==> 137eb93230419b38bce9436f5ab799d8e529c4b7 passed
Test Ok
zcchahaha commented 1 year ago

How are you testing this? I think if you run this, it will fail.

one way to run non-land-blocking forge test on a PR is this (hacky way):

aebd094#diff-72030b71c7c579ba040371a1574c390f94b62f40581abb29a3cd26cb4a5018db

hardcoded part here:

let suite_name = if suite_name == "compat" {
    panic!();
} else {
    "slow_processing_catching_up"
};

let duration = Duration::from_secs(1800);

I thought the test would be run automatically when the forge run is triggered on this PR. So I need to make it land blocking in order to run it?

igor-aptos commented 1 year ago

cc @rustielin , @perryjrandall if there is a better way to run a test.

but by default, land_blocking and compat are run on any PR, irrespective if it is a forge or non-forge PR. above code tricks forge to run your test when it wants to run land_blocking forge test

sitalkedia commented 1 year ago

How are you testing this? I think if you run this, it will fail. one way to run non-land-blocking forge test on a PR is this (hacky way): aebd094#diff-72030b71c7c579ba040371a1574c390f94b62f40581abb29a3cd26cb4a5018db hardcoded part here:

let suite_name = if suite_name == "compat" {
    panic!();
} else {
    "slow_processing_catching_up"
};

let duration = Duration::from_secs(1800);

I thought the test would be run automatically when the forge run is triggered on this PR. So I need to make it land blocking in order to run it?

In order to test it before landing, you can follow @igor-aptos's approach. You also need to add a github yaml file for your test to be running in continuous mode after landing.

github-actions[bot] commented 1 year ago

Forge is running suite validators_join_and_leave on b2ec738a46712a51e9159f119c5de8500d5f3107

github-actions[bot] commented 1 year ago

:x: Forge suite validators_join_and_leave failure on b2ec738a46712a51e9159f119c5de8500d5f3107

Forge test runner terminated:
Trailing Log Lines:
Error: Invalid --suite given: "validators_join_and_leave"
Debugging output:
github-actions[bot] commented 1 year ago

Forge is running suite validators_join_and_leave on bb2ab09cb2e56d1ecd097093742a75c8fe3b8b6a

github-actions[bot] commented 1 year ago

:x: Forge suite validators_join_and_leave failure on bb2ab09cb2e56d1ecd097093742a75c8fe3b8b6a

Forge test runner terminated:
Trailing Log Lines:
Error: Invalid --suite given: "validators_join_and_leave"
Debugging output:
github-actions[bot] commented 1 year ago

Forge is running suite validators_join_and_leave on e56dd85bacc2090f2906bf6e1db7169f9d8aaad4

github-actions[bot] commented 1 year ago

Forge is running suite validators_join_and_leave on 843b204dce971d98449b82624f4f684c7a18b991 ==> e56dd85bacc2090f2906bf6e1db7169f9d8aaad4

github-actions[bot] commented 1 year ago

:x: Forge suite validators_join_and_leave failure on 843b204dce971d98449b82624f4f684c7a18b991 ==> e56dd85bacc2090f2906bf6e1db7169f9d8aaad4

Test Failed: Odd number of digits
Trailing Log Lines:
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:518"},"thread_name":"main","hostname":"forge-compat-pr-4279-1663791276-843b204dce971d98449b82624f4f684","timestamp":"2022-09-21T20:19:08.079491Z","message":"Node validator-0 healthy @ version 1507 > 100"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:518"},"thread_name":"main","hostname":"forge-compat-pr-4279-1663791276-843b204dce971d98449b82624f4f684","timestamp":"2022-09-21T20:19:08.087418Z","message":"Node validator-6 healthy @ version 1507 > 100"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:518"},"thread_name":"main","hostname":"forge-compat-pr-4279-1663791276-843b204dce971d98449b82624f4f684","timestamp":"2022-09-21T20:19:08.094873Z","message":"Node validator-2 healthy @ version 1507 > 100"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/prometheus.rs:14"},"thread_name":"main","hostname":"forge-compat-pr-4279-1663791276-843b204dce971d98449b82624f4f684","timestamp":"2022-09-21T20:19:08.108923Z","message":"Attempting to create prometheus client with: http://aptos-node-mon-aptos-monitoring-prometheus.default.svc:9090 "}
::error::Odd number of digits
test validator join and leave sets ... FAILED
Error: Odd number of digits
Test Statistics: 
Test Failed: Odd number of digits

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:280"},"thread_name":"main","hostname":"forge-compat-pr-4279-1663791276-843b204dce971d98449b82624f4f684","timestamp":"2022-09-21T20:19:08.151521Z","message":"Deleting namespace forge-compat-pr-4279: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:388"},"thread_name":"main","hostname":"forge-compat-pr-4279-1663791276-843b204dce971d98449b82624f4f684","timestamp":"2022-09-21T20:19:08.151542Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-4279"}

failures:
    validator join and leave sets

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
genesis-aptos-genesis-eforge140-78d25   0/1     Completed   0          3m48s
github-actions[bot] commented 1 year ago

:x: Forge suite validators_join_and_leave failure on e56dd85bacc2090f2906bf6e1db7169f9d8aaad4

Test Failed: Odd number of digits
Trailing Log Lines:
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:518"},"thread_name":"main","hostname":"forge-e2e-pr-4279-1663791270-e56dd85bacc2090f2906bf6e1db7169f9d","timestamp":"2022-09-21T20:20:34.004977Z","message":"Node validator-6 healthy @ version 201 > 100"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:518"},"thread_name":"main","hostname":"forge-e2e-pr-4279-1663791270-e56dd85bacc2090f2906bf6e1db7169f9d","timestamp":"2022-09-21T20:20:34.013384Z","message":"Node validator-5 healthy @ version 201 > 100"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:518"},"thread_name":"main","hostname":"forge-e2e-pr-4279-1663791270-e56dd85bacc2090f2906bf6e1db7169f9d","timestamp":"2022-09-21T20:20:34.021881Z","message":"Node validator-2 healthy @ version 201 > 100"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/prometheus.rs:14"},"thread_name":"main","hostname":"forge-e2e-pr-4279-1663791270-e56dd85bacc2090f2906bf6e1db7169f9d","timestamp":"2022-09-21T20:20:34.037930Z","message":"Attempting to create prometheus client with: http://aptos-node-mon-aptos-monitoring-prometheus.default.svc:9090 "}
::error::Odd number of digits
test validator join and leave sets ... FAILED
Error: Odd number of digits
Test Statistics: 
Test Failed: Odd number of digits

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:280"},"thread_name":"main","hostname":"forge-e2e-pr-4279-1663791270-e56dd85bacc2090f2906bf6e1db7169f9d","timestamp":"2022-09-21T20:20:34.070971Z","message":"Deleting namespace forge-e2e-pr-4279: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:388"},"thread_name":"main","hostname":"forge-e2e-pr-4279-1663791270-e56dd85bacc2090f2906bf6e1db7169f9d","timestamp":"2022-09-21T20:20:34.070999Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-4279"}

failures:
    validator join and leave sets

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
github-actions[bot] commented 1 year ago

Forge is running suite validators_join_and_leave on bb2ab09cb2e56d1ecd097093742a75c8fe3b8b6a

github-actions[bot] commented 1 year ago

:x: Forge suite validators_join_and_leave failure on bb2ab09cb2e56d1ecd097093742a75c8fe3b8b6a

Forge test runner terminated:
Trailing Log Lines:
Error: Invalid --suite given: "validators_join_and_leave"
Debugging output:
github-actions[bot] commented 1 year ago

Forge is running suite validators_join_and_leave on bb2ab09cb2e56d1ecd097093742a75c8fe3b8b6a

github-actions[bot] commented 1 year ago

:x: Forge suite validators_join_and_leave failure on bb2ab09cb2e56d1ecd097093742a75c8fe3b8b6a

Forge test runner terminated:
Trailing Log Lines:
Error: Invalid --suite given: "validators_join_and_leave"
Debugging output:
github-actions[bot] commented 1 year ago

Forge is running suite validators_join_and_leave on f4a65b44e7d6addf4ba0e676d319b205d979c2a0

github-actions[bot] commented 1 year ago

Forge is running suite validators_join_and_leave on 843b204dce971d98449b82624f4f684c7a18b991 ==> f4a65b44e7d6addf4ba0e676d319b205d979c2a0

github-actions[bot] commented 1 year ago

:x: Forge suite validators_join_and_leave failure on 843b204dce971d98449b82624f4f684c7a18b991 ==> f4a65b44e7d6addf4ba0e676d319b205d979c2a0

Test Failed: Odd number of digits
Trailing Log Lines:
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:518"},"thread_name":"main","hostname":"forge-compat-pr-4279-1663803114-843b204dce971d98449b82624f4f684","timestamp":"2022-09-21T23:35:08.938981Z","message":"Node validator-3 healthy @ version 651 > 100"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:518"},"thread_name":"main","hostname":"forge-compat-pr-4279-1663803114-843b204dce971d98449b82624f4f684","timestamp":"2022-09-21T23:35:08.947833Z","message":"Node validator-6 healthy @ version 651 > 100"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/prometheus.rs:14"},"thread_name":"main","hostname":"forge-compat-pr-4279-1663803114-843b204dce971d98449b82624f4f684","timestamp":"2022-09-21T23:35:08.965088Z","message":"Attempting to create prometheus client with: http://aptos-node-mon-aptos-monitoring-prometheus.default.svc:9090 "}
Seed in hex: 80000
::error::Odd number of digits
test validator join and leave sets ... FAILED
Error: Odd number of digits
Test Statistics: 
Test Failed: Odd number of digits

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:280"},"thread_name":"main","hostname":"forge-compat-pr-4279-1663803114-843b204dce971d98449b82624f4f684","timestamp":"2022-09-21T23:35:09.001601Z","message":"Deleting namespace forge-compat-pr-4279: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:388"},"thread_name":"main","hostname":"forge-compat-pr-4279-1663803114-843b204dce971d98449b82624f4f684","timestamp":"2022-09-21T23:35:09.001628Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-4279"}

failures:
    validator join and leave sets
Failed to run tests:
Tests Failed

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Error: Tests Failed
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-validator-0                1/1     Running     0          2m21s
aptos-node-1-validator-0                1/1     Running     0          2m20s
aptos-node-2-validator-0                1/1     Running     0          2m20s
aptos-node-3-validator-0                1/1     Running     0          2m21s
aptos-node-4-validator-0                1/1     Running     0          2m21s
aptos-node-5-validator-0                1/1     Running     0          2m20s
aptos-node-6-validator-0                1/1     Running     0          2m20s
genesis-aptos-genesis-eforge188-fd4h5   0/1     Completed   0          2m32s
github-actions[bot] commented 1 year ago

:x: Forge suite validators_join_and_leave failure on f4a65b44e7d6addf4ba0e676d319b205d979c2a0

Test Failed: Odd number of digits
Trailing Log Lines:
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:518"},"thread_name":"main","hostname":"forge-e2e-pr-4279-1663803102-f4a65b44e7d6addf4ba0e676d319b205d9","timestamp":"2022-09-21T23:37:33.993879Z","message":"Node validator-1 healthy @ version 569 > 100"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:518"},"thread_name":"main","hostname":"forge-e2e-pr-4279-1663803102-f4a65b44e7d6addf4ba0e676d319b205d9","timestamp":"2022-09-21T23:37:34.005588Z","message":"Node validator-5 healthy @ version 569 > 100"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/prometheus.rs:14"},"thread_name":"main","hostname":"forge-e2e-pr-4279-1663803102-f4a65b44e7d6addf4ba0e676d319b205d9","timestamp":"2022-09-21T23:37:34.020814Z","message":"Attempting to create prometheus client with: http://aptos-node-mon-aptos-monitoring-prometheus.default.svc:9090 "}
Seed in hex: 80000
::error::Odd number of digits
test validator join and leave sets ... FAILED
Error: Odd number of digits
Test Statistics: 
Test Failed: Odd number of digits

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:280"},"thread_name":"main","hostname":"forge-e2e-pr-4279-1663803102-f4a65b44e7d6addf4ba0e676d319b205d9","timestamp":"2022-09-21T23:37:34.063728Z","message":"Deleting namespace forge-e2e-pr-4279: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:388"},"thread_name":"main","hostname":"forge-e2e-pr-4279-1663803102-f4a65b44e7d6addf4ba0e676d319b205d9","timestamp":"2022-09-21T23:37:34.063750Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-4279"}

failures:
    validator join and leave sets

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-validator-0                1/1     Running     0          2m20s
aptos-node-1-validator-0                1/1     Running     0          2m20s
aptos-node-2-validator-0                1/1     Running     0          2m20s
aptos-node-3-validator-0                1/1     Running     0          2m20s
aptos-node-4-validator-0                1/1     Running     0          2m20s
aptos-node-5-validator-0                1/1     Running     0          2m20s
aptos-node-6-validator-0                1/1     Running     0          2m20s
genesis-aptos-genesis-eforge187-nxx69   0/1     Completed   0          4m56s
github-actions[bot] commented 1 year ago

Forge is running suite validators_join_and_leave on c913872f8bbfef39eaeb5773b53655eb8675ce19

github-actions[bot] commented 1 year ago

Forge is running suite validators_join_and_leave on 843b204dce971d98449b82624f4f684c7a18b991 ==> c913872f8bbfef39eaeb5773b53655eb8675ce19

github-actions[bot] commented 1 year ago

:x: Forge suite validators_join_and_leave failure on c913872f8bbfef39eaeb5773b53655eb8675ce19

Forge test runner terminated:
Trailing Log Lines:
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:518"},"thread_name":"main","hostname":"forge-e2e-pr-4279-1663882005-c913872f8bbfef39eaeb5773b53655eb86","timestamp":"2022-09-22T21:32:48.656515Z","message":"Node validator-3 healthy @ version 563 > 100"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:518"},"thread_name":"main","hostname":"forge-e2e-pr-4279-1663882005-c913872f8bbfef39eaeb5773b53655eb86","timestamp":"2022-09-22T21:32:48.665247Z","message":"Node validator-1 healthy @ version 563 > 100"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:518"},"thread_name":"main","hostname":"forge-e2e-pr-4279-1663882005-c913872f8bbfef39eaeb5773b53655eb86","timestamp":"2022-09-22T21:32:48.673543Z","message":"Node validator-5 healthy @ version 563 > 100"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:518"},"thread_name":"main","hostname":"forge-e2e-pr-4279-1663882005-c913872f8bbfef39eaeb5773b53655eb86","timestamp":"2022-09-22T21:32:48.682114Z","message":"Node validator-0 healthy @ version 565 > 100"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:518"},"thread_name":"main","hostname":"forge-e2e-pr-4279-1663882005-c913872f8bbfef39eaeb5773b53655eb86","timestamp":"2022-09-22T21:32:48.690804Z","message":"Node validator-2 healthy @ version 565 > 100"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/prometheus.rs:14"},"thread_name":"main","hostname":"forge-e2e-pr-4279-1663882005-c913872f8bbfef39eaeb5773b53655eb86","timestamp":"2022-09-22T21:32:48.706256Z","message":"Attempting to create prometheus client with: http://aptos-node-mon-aptos-monitoring-prometheus.default.svc:9090 "}
thread 'main' panicked at 'attempt to divide by zero', crates/transaction-emitter-lib/src/emitter/mod.rs:210:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:280"},"thread_name":"main","hostname":"forge-e2e-pr-4279-1663882005-c913872f8bbfef39eaeb5773b53655eb86","timestamp":"2022-09-22T21:32:48.747823Z","message":"Deleting namespace forge-e2e-pr-4279: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:388"},"thread_name":"main","hostname":"forge-e2e-pr-4279-1663882005-c913872f8bbfef39eaeb5773b53655eb86","timestamp":"2022-09-22T21:32:48.747849Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-4279"}
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-validator-0                1/1     Running     0          2m19s
aptos-node-1-validator-0                1/1     Running     0          2m19s
aptos-node-2-validator-0                1/1     Running     0          2m19s
aptos-node-3-validator-0                1/1     Running     0          2m19s
aptos-node-4-validator-0                1/1     Running     0          2m19s
aptos-node-5-validator-0                1/1     Running     0          2m19s
aptos-node-6-validator-0                1/1     Running     0          2m19s
genesis-aptos-genesis-eforge221-hdbkk   0/1     Completed   0          5m13s
github-actions[bot] commented 1 year ago

:x: Forge suite validators_join_and_leave failure on 843b204dce971d98449b82624f4f684c7a18b991 ==> c913872f8bbfef39eaeb5773b53655eb8675ce19

Forge test runner terminated:
Trailing Log Lines:
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:518"},"thread_name":"main","hostname":"forge-compat-pr-4279-1663882034-843b204dce971d98449b82624f4f684","timestamp":"2022-09-22T21:41:39.722756Z","message":"Node validator-2 healthy @ version 2259 > 100"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:518"},"thread_name":"main","hostname":"forge-compat-pr-4279-1663882034-843b204dce971d98449b82624f4f684","timestamp":"2022-09-22T21:41:39.730344Z","message":"Node validator-1 healthy @ version 2259 > 100"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:518"},"thread_name":"main","hostname":"forge-compat-pr-4279-1663882034-843b204dce971d98449b82624f4f684","timestamp":"2022-09-22T21:41:39.738321Z","message":"Node validator-6 healthy @ version 2259 > 100"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:518"},"thread_name":"main","hostname":"forge-compat-pr-4279-1663882034-843b204dce971d98449b82624f4f684","timestamp":"2022-09-22T21:41:39.746288Z","message":"Node validator-3 healthy @ version 2259 > 100"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:518"},"thread_name":"main","hostname":"forge-compat-pr-4279-1663882034-843b204dce971d98449b82624f4f684","timestamp":"2022-09-22T21:41:39.754860Z","message":"Node validator-4 healthy @ version 2259 > 100"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/prometheus.rs:14"},"thread_name":"main","hostname":"forge-compat-pr-4279-1663882034-843b204dce971d98449b82624f4f684","timestamp":"2022-09-22T21:41:39.770620Z","message":"Attempting to create prometheus client with: http://aptos-node-mon-aptos-monitoring-prometheus.default.svc:9090 "}
thread 'main' panicked at 'attempt to divide by zero', crates/transaction-emitter-lib/src/emitter/mod.rs:210:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:280"},"thread_name":"main","hostname":"forge-compat-pr-4279-1663882034-843b204dce971d98449b82624f4f684","timestamp":"2022-09-22T21:41:39.809412Z","message":"Deleting namespace forge-compat-pr-4279: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:388"},"thread_name":"main","hostname":"forge-compat-pr-4279-1663882034-843b204dce971d98449b82624f4f684","timestamp":"2022-09-22T21:41:39.809441Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-4279"}
Debugging output:
NAME                       READY   STATUS    RESTARTS   AGE
aptos-node-0-validator-0   1/1     Running   0          13m
aptos-node-1-validator-0   1/1     Running   0          13m
aptos-node-2-validator-0   1/1     Running   0          13m
aptos-node-3-validator-0   1/1     Running   0          13m
aptos-node-4-validator-0   1/1     Running   0          13m
aptos-node-5-validator-0   1/1     Running   0          13m
aptos-node-6-validator-0   1/1     Running   0          13m
github-actions[bot] commented 1 year ago

Forge is running suite validators_join_and_leave on 843b204dce971d98449b82624f4f684c7a18b991 ==> a71c4768fd21ecfe7adaa514bad7d7106c80efc7

github-actions[bot] commented 1 year ago

Forge is running suite validators_join_and_leave on a71c4768fd21ecfe7adaa514bad7d7106c80efc7

github-actions[bot] commented 1 year ago

:x: Forge suite validators_join_and_leave failure on 843b204dce971d98449b82624f4f684c7a18b991 ==> a71c4768fd21ecfe7adaa514bad7d7106c80efc7

Forge test runner terminated:
Trailing Log Lines:
 * [95777] Executed successfully: sender=0x78075e0f53189479431612a563ffc095f15596a8335f58d8fd93539812267997, payload=EntryFunctionPayload(EntryFunctionPayload { function: EntryFunctionId { module: MoveModuleId { address: Address(0000000000000000000000000000000000000000000000000000000000000001), name: IdentifierWrapper(Identifier("coin")) }, name: IdentifierWrapper(Identifier("transfer")) }, type_arguments: [Struct(MoveStructTag { address: Address(0000000000000000000000000000000000000000000000000000000000000001), module: IdentifierWrapper(Identifier("aptos_coin")), name: IdentifierWrapper(Identifier("AptosCoin")), generic_type_params: [] })], arguments: [String("0xfabc8619420154211075c70e2c005fe036548cc3d1ecc32fc9bdbd9f331a5b43"), String("1")] })
 * [95778] Executed successfully: sender=0xe6f3182991d0c7daaa51c58954603a803c8666bd4c22dc489e18bc600de89822, payload=EntryFunctionPayload(EntryFunctionPayload { function: EntryFunctionId { module: MoveModuleId { address: Address(0000000000000000000000000000000000000000000000000000000000000001), name: IdentifierWrapper(Identifier("coin")) }, name: IdentifierWrapper(Identifier("transfer")) }, type_arguments: [Struct(MoveStructTag { address: Address(0000000000000000000000000000000000000000000000000000000000000001), module: IdentifierWrapper(Identifier("aptos_coin")), name: IdentifierWrapper(Identifier("AptosCoin")), generic_type_params: [] })], arguments: [String("0xfabc8619420154211075c70e2c005fe036548cc3d1ecc32fc9bdbd9f331a5b43"), String("1")] })
 * [95779] Executed successfully: sender=0xef0f9712573e1b61b8856814f7da3f65c47337280e4c9828e0062a3ecf485337, payload=EntryFunctionPayload(EntryFunctionPayload { function: EntryFunctionId { module: MoveModuleId { address: Address(0000000000000000000000000000000000000000000000000000000000000001), name: IdentifierWrapper(Identifier("coin")) }, name: IdentifierWrapper(Identifier("transfer")) }, type_arguments: [Struct(MoveStructTag { address: Address(0000000000000000000000000000000000000000000000000000000000000001), module: IdentifierWrapper(Identifier("aptos_coin")), name: IdentifierWrapper(Identifier("AptosCoin")), generic_type_params: [] })], arguments: [String("0xfabc8619420154211075c70e2c005fe036548cc3d1ecc32fc9bdbd9f331a5b43"), String("1")] })
 * [95780] Executed successfully: state_checkpoint_transaction
 * [95781] Executed successfully: block_metadata_transaction
 * [95782] Executed successfully: state_checkpoint_transaction
', /aptos/crates/aptos/src/test/mod.rs:579:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:280"},"thread_name":"main","hostname":"forge-compat-pr-4279-1663885399-843b204dce971d98449b82624f4f684","timestamp":"2022-09-22T22:28:14.584230Z","message":"Deleting namespace forge-compat-pr-4279: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:388"},"thread_name":"main","hostname":"forge-compat-pr-4279-1663885399-843b204dce971d98449b82624f4f684","timestamp":"2022-09-22T22:28:14.584256Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-4279"}
Debugging output:
github-actions[bot] commented 1 year ago

:x: Forge suite validators_join_and_leave failure on a71c4768fd21ecfe7adaa514bad7d7106c80efc7

Forge test runner terminated:
Trailing Log Lines:
 * [168816] Executed successfully: sender=0x57a42af7d466bd602074d52a4807a039f087c8c47321e9ec2fbe6ad7807d8835, payload=EntryFunctionPayload(EntryFunctionPayload { function: EntryFunctionId { module: MoveModuleId { address: Address(0000000000000000000000000000000000000000000000000000000000000001), name: IdentifierWrapper(Identifier("coin")) }, name: IdentifierWrapper(Identifier("transfer")) }, type_arguments: [Struct(MoveStructTag { address: Address(0000000000000000000000000000000000000000000000000000000000000001), module: IdentifierWrapper(Identifier("aptos_coin")), name: IdentifierWrapper(Identifier("AptosCoin")), generic_type_params: [] })], arguments: [String("0x43bd30431c2d73da07a7564969eacf9908a5d9e1fb378f06b2bc84657b15186d"), String("1")] })
 * [168817] Executed successfully: sender=0xebfcdd8709fa7491bedef039a93e5e518d4a1d211d805bc8cd681a17bd1a0577, payload=EntryFunctionPayload(EntryFunctionPayload { function: EntryFunctionId { module: MoveModuleId { address: Address(0000000000000000000000000000000000000000000000000000000000000001), name: IdentifierWrapper(Identifier("coin")) }, name: IdentifierWrapper(Identifier("transfer")) }, type_arguments: [Struct(MoveStructTag { address: Address(0000000000000000000000000000000000000000000000000000000000000001), module: IdentifierWrapper(Identifier("aptos_coin")), name: IdentifierWrapper(Identifier("AptosCoin")), generic_type_params: [] })], arguments: [String("0x61bd099f1c3c3d868567b83f7fbc5b5e75524216cc563cd63fe10c9a05f97df"), String("1")] })
 * [168818] Executed successfully: sender=0xecdf24bb527eb08abbf50f08c10ba083e0e3eec038d461c074cb98b36ee3ea90, payload=EntryFunctionPayload(EntryFunctionPayload { function: EntryFunctionId { module: MoveModuleId { address: Address(0000000000000000000000000000000000000000000000000000000000000001), name: IdentifierWrapper(Identifier("coin")) }, name: IdentifierWrapper(Identifier("transfer")) }, type_arguments: [Struct(MoveStructTag { address: Address(0000000000000000000000000000000000000000000000000000000000000001), module: IdentifierWrapper(Identifier("aptos_coin")), name: IdentifierWrapper(Identifier("AptosCoin")), generic_type_params: [] })], arguments: [String("0x43bd30431c2d73da07a7564969eacf9908a5d9e1fb378f06b2bc84657b15186d"), String("1")] })
 * [168819] Executed successfully: sender=0xd0c7b37d39b2cc3b8149541b0f9ce3ac4ee5851571110a5cf7904cad5d60fef5, payload=EntryFunctionPayload(EntryFunctionPayload { function: EntryFunctionId { module: MoveModuleId { address: Address(0000000000000000000000000000000000000000000000000000000000000001), name: IdentifierWrapper(Identifier("coin")) }, name: IdentifierWrapper(Identifier("transfer")) }, type_arguments: [Struct(MoveStructTag { address: Address(0000000000000000000000000000000000000000000000000000000000000001), module: IdentifierWrapper(Identifier("aptos_coin")), name: IdentifierWrapper(Identifier("AptosCoin")), generic_type_params: [] })], arguments: [String("0x43bd30431c2d73da07a7564969eacf9908a5d9e1fb378f06b2bc84657b15186d"), String("1")] })
 * [168820] Executed successfully: sender=0x39a7780d1363dded66d3253351db844ca8d9d1f69df4e62b378010efb71b6763, payload=EntryFunctionPayload(EntryFunctionPayload { function: EntryFunctionId { module: MoveModuleId { address: Address(0000000000000000000000000000000000000000000000000000000000000001), name: IdentifierWrapper(Identifier("coin")) }, name: IdentifierWrapper(Identifier("transfer")) }, type_arguments: [Struct(MoveStructTag { address: Address(0000000000000000000000000000000000000000000000000000000000000001), module: IdentifierWrapper(Identifier("aptos_coin")), name: IdentifierWrapper(Identifier("AptosCoin")), generic_type_params: [] })], arguments: [String("0x37adbf34fe32f9e401cf15e3536c2849e67eb99e4d2be3321ea86e27cdf03068"), String("1")] })
 * [168821] Executed successfully: state_checkpoint_transaction
', /aptos/crates/aptos/src/test/mod.rs:579:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:280"},"thread_name":"main","hostname":"forge-e2e-pr-4279-1663885403-a71c4768fd21ecfe7adaa514bad7d7106c","timestamp":"2022-09-22T22:29:31.070731Z","message":"Deleting namespace forge-e2e-pr-4279: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:388"},"thread_name":"main","hostname":"forge-e2e-pr-4279-1663885403-a71c4768fd21ecfe7adaa514bad7d7106c","timestamp":"2022-09-22T22:29:31.070759Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-4279"}
Debugging output:
github-actions[bot] commented 1 year ago

Forge is running suite validators_join_and_leave on 843b204dce971d98449b82624f4f684c7a18b991 ==> ca7e10ec24bb22615cf9f13b573bd23dac5e5da9

github-actions[bot] commented 1 year ago

Forge is running suite validators_join_and_leave on ca7e10ec24bb22615cf9f13b573bd23dac5e5da9

github-actions[bot] commented 1 year ago

:x: Forge suite validators_join_and_leave failure on 843b204dce971d98449b82624f4f684c7a18b991 ==> ca7e10ec24bb22615cf9f13b573bd23dac5e5da9

Forge test runner terminated:
Trailing Log Lines:
{"level":"INFO","source":{"package":"transaction_emitter_lib","file":"crates/transaction-emitter-lib/src/emitter/account_minter.rs:137"},"thread_name":"main","hostname":"forge-compat-pr-4279-1663970400-843b204dce971d98449b82624f4f684","timestamp":"2022-09-23T22:05:02.745709Z","message":"Minting additional 1995 accounts with 5012581425313 coins each"}
{"level":"INFO","source":{"package":"transaction_emitter_lib","file":"crates/transaction-emitter-lib/src/emitter/account_minter.rs:189"},"thread_name":"main","hostname":"forge-compat-pr-4279-1663970400-843b204dce971d98449b82624f4f684","timestamp":"2022-09-23T22:05:06.038516Z","message":"Successfully completed creating accounts, had to retry 0 transactions"}
{"level":"INFO","source":{"package":"transaction_emitter_lib","file":"crates/transaction-emitter-lib/src/emitter/mod.rs:452"},"thread_name":"main","hostname":"forge-compat-pr-4279-1663970400-843b204dce971d98449b82624f4f684","timestamp":"2022-09-23T22:05:06.039155Z","message":"Checking account sequence and counting latency for 1995 out of 1995 total_workers"}
{"level":"INFO","source":{"package":"transaction_emitter_lib","file":"crates/transaction-emitter-lib/src/emitter/mod.rs:482"},"thread_name":"main","hostname":"forge-compat-pr-4279-1663970400-843b204dce971d98449b82624f4f684","timestamp":"2022-09-23T22:05:06.045214Z","message":"Tx emitter workers started"}
{"level":"INFO","source":{"package":"testcases","file":"testsuite/testcases/src/lib.rs:256"},"thread_name":"main","hostname":"forge-compat-pr-4279-1663970400-843b204dce971d98449b82624f4f684","timestamp":"2022-09-23T22:05:06.045278Z","message":"Starting emitting txns for 300s"}
{"level":"INFO","source":{"package":"testcases","file":"testsuite/testcases/src/lib.rs:259"},"thread_name":"main","hostname":"forge-compat-pr-4279-1663970400-843b204dce971d98449b82624f4f684","timestamp":"2022-09-23T22:05:21.045363Z","message":"15s warmup finished"}
thread 'main' panicked at 'This is a serious bug', crates/aptos-keygen/src/lib.rs:22:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:280"},"thread_name":"main","hostname":"forge-compat-pr-4279-1663970400-843b204dce971d98449b82624f4f684","timestamp":"2022-09-23T22:05:21.108916Z","message":"Deleting namespace forge-compat-pr-4279: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:388"},"thread_name":"main","hostname":"forge-compat-pr-4279-1663970400-843b204dce971d98449b82624f4f684","timestamp":"2022-09-23T22:05:21.108946Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-4279"}
Debugging output:
NAME                       READY   STATUS    RESTARTS   AGE
aptos-node-4-validator-0   1/1     Running   0          4m10s
aptos-node-5-validator-0   1/1     Running   0          4m10s
aptos-node-6-validator-0   1/1     Running   0          4m10s
github-actions[bot] commented 1 year ago

:x: Forge suite validators_join_and_leave failure on ca7e10ec24bb22615cf9f13b573bd23dac5e5da9

Forge test runner terminated:
Trailing Log Lines:
             at /aptos/crates/aptos/src/main.rs:20:5
  27: core::ops::function::FnOnce::call_once
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/ops/function.rs:248:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:81"},"thread_name":"main","hostname":"forge-e2e-pr-4279-1663970401-ca7e10ec24bb22615cf9f13b573bd23dac","timestamp":"2022-09-23T22:10:49.220299Z","message":"Genesis status: JobStatus { active: Some(1), completion_time: None, conditions: None, failed: Some(6), start_time: Some(Time(2022-09-23T22:00:52Z)), succeeded: None }"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:280"},"thread_name":"main","hostname":"forge-e2e-pr-4279-1663970401-ca7e10ec24bb22615cf9f13b573bd23dac","timestamp":"2022-09-23T22:10:49.243251Z","message":"Deleting namespace forge-e2e-pr-4279: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:388"},"thread_name":"main","hostname":"forge-e2e-pr-4279-1663970401-ca7e10ec24bb22615cf9f13b573bd23dac","timestamp":"2022-09-23T22:10:49.243278Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-4279"}
Failed to run tests:
Genesis did not succeed
Error: Genesis did not succeed
Debugging output:
github-actions[bot] commented 1 year ago

Forge is running suite validators_join_and_leave on 846e3906245f2d915b20ef6db3fe1b32a14f1257

github-actions[bot] commented 1 year ago

Forge is running suite validators_join_and_leave on 843b204dce971d98449b82624f4f684c7a18b991 ==> 846e3906245f2d915b20ef6db3fe1b32a14f1257

github-actions[bot] commented 1 year ago

:x: Forge suite validators_join_and_leave failure on 843b204dce971d98449b82624f4f684c7a18b991 ==> 846e3906245f2d915b20ef6db3fe1b32a14f1257

Forge test runner terminated:
Trailing Log Lines:
.............8
.............0
.............0
{"level":"INFO","source":{"package":"aptos","file":"crates/aptos/src/test/mod.rs:148"},"thread_name":"main","hostname":"forge-compat-pr-4279-1664217840-843b204dce971d98449b82624f4f684","timestamp":"2022-09-26T18:48:40.090584Z","message":"Account Address: c15f7032d7f6c534ee8c1110a836484859fb4a0e02f1e50b4408fca646d888f8"}
.......... cli test framework: 40021637eaad12143038bda2ad1750d32c1e1fcc8393905bfed001cf35f5282c
.......... cli test framework: c15f7032d7f6c534ee8c1110a836484859fb4a0e02f1e50b4408fca646d888f8
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ApiError("API error Error(ResourceNotFound): Resource not found by Address(0x40021637eaad12143038bda2ad1750d32c1e1fcc8393905bfed001cf35f5282c), Struct tag(0x1::account::Account) and Ledger version(153916)")', testsuite/testcases/src/validator_join_leave_test.rs:106:18
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:280"},"thread_name":"main","hostname":"forge-compat-pr-4279-1664217840-843b204dce971d98449b82624f4f684","timestamp":"2022-09-26T18:48:40.188414Z","message":"Deleting namespace forge-compat-pr-4279: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:388"},"thread_name":"main","hostname":"forge-compat-pr-4279-1664217840-843b204dce971d98449b82624f4f684","timestamp":"2022-09-26T18:48:40.188449Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-4279"}
Debugging output:
github-actions[bot] commented 1 year ago

:x: Forge suite validators_join_and_leave failure on 846e3906245f2d915b20ef6db3fe1b32a14f1257

Forge test runner terminated:
Trailing Log Lines:
.............8
.............0
.............0
{"level":"INFO","source":{"package":"aptos","file":"crates/aptos/src/test/mod.rs:148"},"thread_name":"main","hostname":"forge-e2e-pr-4279-1664217832-846e3906245f2d915b20ef6db3fe1b32a1","timestamp":"2022-09-26T18:49:41.986936Z","message":"Account Address: c15f7032d7f6c534ee8c1110a836484859fb4a0e02f1e50b4408fca646d888f8"}
.......... cli test framework: 40021637eaad12143038bda2ad1750d32c1e1fcc8393905bfed001cf35f5282c
.......... cli test framework: c15f7032d7f6c534ee8c1110a836484859fb4a0e02f1e50b4408fca646d888f8
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ApiError("API error Error(ResourceNotFound): Resource not found by Address(0x40021637eaad12143038bda2ad1750d32c1e1fcc8393905bfed001cf35f5282c), Struct tag(0x1::account::Account) and Ledger version(237301)")', testsuite/testcases/src/validator_join_leave_test.rs:106:18
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:280"},"thread_name":"main","hostname":"forge-e2e-pr-4279-1664217832-846e3906245f2d915b20ef6db3fe1b32a1","timestamp":"2022-09-26T18:49:42.054106Z","message":"Deleting namespace forge-e2e-pr-4279: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:388"},"thread_name":"main","hostname":"forge-e2e-pr-4279-1664217832-846e3906245f2d915b20ef6db3fe1b32a1","timestamp":"2022-09-26T18:49:42.054132Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-4279"}
Debugging output:
NAME                                   READY   STATUS        RESTARTS   AGE
aptos-node-5-validator-0               1/1     Terminating   0          3m19s
aptos-node-6-validator-0               1/1     Running       0          3m19s
genesis-aptos-genesis-eforge69-scp8j   0/1     Completed     0          5m2s
github-actions[bot] commented 1 year ago

Forge is running suite validators_join_and_leave on 843b204dce971d98449b82624f4f684c7a18b991 ==> d3930e5c5a21d18835a4281906cb29a04cdd5b75