Closed danielxiangzl closed 1 year ago
github-actions[bot]
commented
1 year ago compat on testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 7af4ee991610a285118a7468f7845326a231ba9dgithub-actions[bot]
commented
1 year ago land_blocking on 7af4ee991610a285118a7468f7845326a231ba9dgithub-actions[bot]
commented
1 year ago framework_upgrade on cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> 7af4ee991610a285118a7468f7845326a231ba9dgithub-actions[bot]
commented
1 year ago framework_upgrade failure on cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> 7af4ee991610a285118a7468f7845326a231ba9dForge test runner terminated:
Trailing Log Lines:
Error: Invalid --suite given: "framework_upgrade"
Debugging output:
compat failure on testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 7af4ee991610a285118a7468f7845326a231ba9dCompatibility test results for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 7af4ee991610a285118a7468f7845326a231ba9d (PR)
1. Check liveness of validators at old version: testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b
compatibility::simple-validator-upgrade::liveness-check : 8138 TPS, 4674 ms latency, 6900 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 7af4ee991610a285118a7468f7845326a231ba9d
compatibility::simple-validator-upgrade::single-validator-upgrade : 3376 TPS, 14122 ms latency, 20800 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: 7af4ee991610a285118a7468f7845326a231ba9d
Test Failed: Unknown error Ledger on endpoint (http://aptos-node-0-validator.forge-compat-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (1c7d197f) might still succeed.
Trailing Log Lines:
Error: Unknown error Ledger on endpoint (http://aptos-node-0-validator.forge-compat-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (1c7d197f) might still succeed.
Test Statistics:
Compatibility test results for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 7af4ee991610a285118a7468f7845326a231ba9d (PR)
1. Check liveness of validators at old version: testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b
compatibility::simple-validator-upgrade::liveness-check : 8138 TPS, 4674 ms latency, 6900 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 7af4ee991610a285118a7468f7845326a231ba9d
compatibility::simple-validator-upgrade::single-validator-upgrade : 3376 TPS, 14122 ms latency, 20800 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: 7af4ee991610a285118a7468f7845326a231ba9d
Test Failed: Unknown error Ledger on endpoint (http://aptos-node-0-validator.forge-compat-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (1c7d197f) might still succeed.
Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:280"},"thread_name":"main","hostname":"forge-compat-pr-6914-1677888984-testnet-2d8b1b57553d869190f61df","timestamp":"2023-03-04T00:28:03.428741Z","message":"Deleting namespace forge-compat-pr-6914: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:388"},"thread_name":"main","hostname":"forge-compat-pr-6914-1677888984-testnet-2d8b1b57553d869190f61df","timestamp":"2023-03-04T00:28:03.428764Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-6914"}
failures:
compatibility::simple-validator-upgrade
test result: FAILED. 0 passed; 1 failed; 0 filtered out
Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
land_blocking on 7af4ee991610a285118a7468f7845326a231ba9dgithub-actions[bot]
commented
1 year ago land_blocking on 7af4ee991610a285118a7468f7845326a231ba9dgithub-actions[bot]
commented
1 year ago land_blocking on 7af4ee991610a285118a7468f7845326a231ba9dgithub-actions[bot]
commented
1 year ago framework_upgrade on cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> 50fdc31064330efd94ef973254ce970547502a69github-actions[bot]
commented
1 year ago land_blocking on 50fdc31064330efd94ef973254ce970547502a69github-actions[bot]
commented
1 year ago compat on testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 50fdc31064330efd94ef973254ce970547502a69github-actions[bot]
commented
1 year ago consensus_only_perf_benchmark on consensus_only_perf_test_50fdc31064330efd94ef973254ce970547502a69 ==> 50fdc31064330efd94ef973254ce970547502a69github-actions[bot]
commented
1 year ago land_blocking failure on 50fdc31064330efd94ef973254ce970547502a69Test Failed: start emitter job
Caused by:
Failed to create accounts: Account 83856da95d1ef469d269835ea763d7eccd6ab6788189208c82e688ef932ca205 couldn't mint
Caused by:
Unknown error Ledger on endpoint (http://aptos-node-4-fullnode.forge-e2e-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (d8a1d788) might still succeed.
Trailing Log Lines:
Unknown error Ledger on endpoint (http://aptos-node-4-fullnode.forge-e2e-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (d8a1d788) might still succeed.
Test Statistics:
Test Failed: start emitter job
Caused by:
Failed to create accounts: Account 83856da95d1ef469d269835ea763d7eccd6ab6788189208c82e688ef932ca205 couldn't mint
Caused by:
Unknown error Ledger on endpoint (http://aptos-node-4-fullnode.forge-e2e-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (d8a1d788) might still succeed.
Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:281"},"thread_name":"main","hostname":"forge-e2e-pr-6914-1678268686-50fdc31064330efd94ef973254ce970547","timestamp":"2023-03-08T09:54:54.116997Z","message":"Deleting namespace forge-e2e-pr-6914: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:389"},"thread_name":"main","hostname":"forge-e2e-pr-6914-1678268686-50fdc31064330efd94ef973254ce970547","timestamp":"2023-03-08T09:54:54.117019Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-6914"}
failures:
performance benchmark with full nodes
test result: FAILED. 0 passed; 1 failed; 0 filtered out
Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
NAME READY STATUS RESTARTS AGE
aptos-node-6-validator-0 1/1 Running 0 5m4s
aptos-node-7-fullnode-eforge73-0 1/1 Running 0 5m5s
aptos-node-7-validator-0 1/1 Running 0 5m4s
aptos-node-8-fullnode-eforge73-0 1/1 Running 0 5m5s
aptos-node-8-validator-0 1/1 Running 0 5m5s
aptos-node-9-fullnode-eforge73-0 1/1 Running 0 5m4s
aptos-node-9-validator-0 1/1 Running 0 5m5s
genesis-aptos-genesis-eforge73-jbndw 0/1 Completed 0 9m23s
compat failure on testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 50fdc31064330efd94ef973254ce970547502a69Compatibility test results for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 50fdc31064330efd94ef973254ce970547502a69 (PR)
1. Check liveness of validators at old version: testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b
compatibility::simple-validator-upgrade::liveness-check : 8032 TPS, 4747 ms latency, 7300 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 50fdc31064330efd94ef973254ce970547502a69
compatibility::simple-validator-upgrade::single-validator-upgrade : 3084 TPS, 14691 ms latency, 21100 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: 50fdc31064330efd94ef973254ce970547502a69
Test Failed: Unknown error Ledger on endpoint (http://aptos-node-0-validator.forge-compat-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (307a8b00) might still succeed.
Trailing Log Lines:
Error: Unknown error Ledger on endpoint (http://aptos-node-0-validator.forge-compat-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (307a8b00) might still succeed.
Test Statistics:
Compatibility test results for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 50fdc31064330efd94ef973254ce970547502a69 (PR)
1. Check liveness of validators at old version: testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b
compatibility::simple-validator-upgrade::liveness-check : 8032 TPS, 4747 ms latency, 7300 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 50fdc31064330efd94ef973254ce970547502a69
compatibility::simple-validator-upgrade::single-validator-upgrade : 3084 TPS, 14691 ms latency, 21100 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: 50fdc31064330efd94ef973254ce970547502a69
Test Failed: Unknown error Ledger on endpoint (http://aptos-node-0-validator.forge-compat-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (307a8b00) might still succeed.
Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:281"},"thread_name":"main","hostname":"forge-compat-pr-6914-1678268688-testnet-2d8b1b57553d869190f61df","timestamp":"2023-03-08T09:58:47.073516Z","message":"Deleting namespace forge-compat-pr-6914: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:389"},"thread_name":"main","hostname":"forge-compat-pr-6914-1678268688-testnet-2d8b1b57553d869190f61df","timestamp":"2023-03-08T09:58:47.073540Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-6914"}
failures:
compatibility::simple-validator-upgrade
test result: FAILED. 0 passed; 1 failed; 0 filtered out
Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
NAME READY STATUS RESTARTS AGE
aptos-node-0-validator-0 1/1 Running 0 3m48s
aptos-node-1-validator-0 1/1 Running 0 7m32s
aptos-node-2-validator-0 1/1 Running 0 10m
aptos-node-3-validator-0 1/1 Running 0 10m
aptos-node-4-validator-0 1/1 Running 0 10m
genesis-aptos-genesis-eforge59-sdf66 0/1 Completed 0 13m
framework_upgrade failure on cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> 50fdc31064330efd94ef973254ce970547502a69Compatibility test results for cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> 50fdc31064330efd94ef973254ce970547502a69 (PR)
Upgrade the nodes to version: 50fdc31064330efd94ef973254ce970547502a69
Test Failed: Failed to create accounts: Account 2eb7fe6e64b184d7b9a80ed948f83675afe13fa01ceb484a1f5d30c01b70ed23 couldn't mint
Caused by:
Unknown error Ledger on endpoint (http://aptos-node-0-validator.forge-framework-upgrade-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (1c9203e0) might still succeed.
Trailing Log Lines:
Caused by:
Unknown error Ledger on endpoint (http://aptos-node-0-validator.forge-framework-upgrade-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (1c9203e0) might still succeed.
Test Statistics:
Compatibility test results for cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> 50fdc31064330efd94ef973254ce970547502a69 (PR)
Upgrade the nodes to version: 50fdc31064330efd94ef973254ce970547502a69
Test Failed: Failed to create accounts: Account 2eb7fe6e64b184d7b9a80ed948f83675afe13fa01ceb484a1f5d30c01b70ed23 couldn't mint
Caused by:
Unknown error Ledger on endpoint (http://aptos-node-0-validator.forge-framework-upgrade-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (1c9203e0) might still succeed.
Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:281"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-6914-1678268679-cb4ba0a57c998c60cbab","timestamp":"2023-03-08T10:01:49.045721Z","message":"Deleting namespace forge-framework-upgrade-pr-6914: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:389"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-6914-1678268679-cb4ba0a57c998c60cbab","timestamp":"2023-03-08T10:01:49.045744Z","message":"aptos-node resources for Forge removed in namespace: forge-framework-upgrade-pr-6914"}
failures:
framework_upgrade::framework-upgrade
test result: FAILED. 0 passed; 1 failed; 0 filtered out
Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
NAME READY STATUS RESTARTS AGE
aptos-node-0-validator-0 1/1 Running 0 11m
aptos-node-1-validator-0 1/1 Running 0 9m52s
aptos-node-2-validator-0 1/1 Running 0 8m32s
aptos-node-3-validator-0 1/1 Running 0 7m12s
aptos-node-4-validator-0 1/1 Running 0 5m52s
consensus_only_perf_benchmark failure on consensus_only_perf_test_50fdc31064330efd94ef973254ce970547502a69 ==> 50fdc31064330efd94ef973254ce970547502a69Test Failed: start emitter job
Caused by:
Failed to create accounts: Account ad5e95d69c55e319c6468e5d111e01f66b338f02414479da82c5df2e1345515c couldn't mint
Caused by:
Unknown error Ledger on endpoint (http://aptos-node-7-validator.forge-consensus-only-perf-test-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (a11e0fa3) might still succeed.
Trailing Log Lines:
Unknown error Ledger on endpoint (http://aptos-node-7-validator.forge-consensus-only-perf-test-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (a11e0fa3) might still succeed.
Test Statistics:
Test Failed: start emitter job
Caused by:
Failed to create accounts: Account ad5e95d69c55e319c6468e5d111e01f66b338f02414479da82c5df2e1345515c couldn't mint
Caused by:
Unknown error Ledger on endpoint (http://aptos-node-7-validator.forge-consensus-only-perf-test-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (a11e0fa3) might still succeed.
Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:281"},"thread_name":"main","hostname":"forge-consensus-only-perf-test-pr-6914-1678268721-consensus-onl","timestamp":"2023-03-08T10:07:00.542641Z","message":"Deleting namespace forge-consensus-only-perf-test-pr-6914: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:389"},"thread_name":"main","hostname":"forge-consensus-only-perf-test-pr-6914-1678268721-consensus-onl","timestamp":"2023-03-08T10:07:00.542667Z","message":"aptos-node resources for Forge removed in namespace: forge-consensus-only-perf-test-pr-6914"}
failures:
continuous progress test
test result: FAILED. 0 passed; 1 failed; 0 filtered out
Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
NAME READY STATUS RESTARTS AGE
aptos-node-0-validator-0 1/1 Running 0 16m
aptos-node-1-validator-0 1/1 Running 0 16m
aptos-node-10-validator-0 1/1 Running 0 16m
aptos-node-11-validator-0 1/1 Running 0 16m
aptos-node-12-validator-0 1/1 Running 0 16m
aptos-node-13-validator-0 1/1 Running 0 16m
aptos-node-14-validator-0 1/1 Running 0 16m
aptos-node-15-validator-0 1/1 Running 0 16m
aptos-node-16-validator-0 1/1 Running 0 16m
aptos-node-17-validator-0 1/1 Running 0 16m
aptos-node-18-validator-0 1/1 Running 0 16m
aptos-node-19-validator-0 1/1 Running 0 16m
aptos-node-2-validator-0 1/1 Running 0 16m
aptos-node-3-validator-0 1/1 Running 0 16m
aptos-node-4-validator-0 1/1 Running 0 16m
aptos-node-5-validator-0 1/1 Running 0 16m
aptos-node-6-validator-0 1/1 Running 0 16m
aptos-node-7-validator-0 1/1 Running 0 16m
aptos-node-8-validator-0 1/1 Running 0 16m
aptos-node-9-validator-0 1/1 Running 0 16m
genesis-aptos-genesis-eforge245-7rrkx 0/1 Completed 0 20m
land_blocking on 50fdc31064330efd94ef973254ce970547502a69github-actions[bot]
commented
1 year ago land_blocking failure on 50fdc31064330efd94ef973254ce970547502a69Test Failed: start emitter job
Caused by:
Unknown error Ledger on endpoint (http://aptos-node-3-fullnode.forge-e2e-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (3f428706) might still succeed.
Trailing Log Lines:
Error: start emitter job
Caused by:
Unknown error Ledger on endpoint (http://aptos-node-3-fullnode.forge-e2e-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (3f428706) might still succeed.
Test Statistics:
Test Failed: start emitter job
Caused by:
Unknown error Ledger on endpoint (http://aptos-node-3-fullnode.forge-e2e-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (3f428706) might still succeed.
Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:281"},"thread_name":"main","hostname":"forge-e2e-pr-6914-1678348197-50fdc31064330efd94ef973254ce970547","timestamp":"2023-03-09T07:59:26.292418Z","message":"Deleting namespace forge-e2e-pr-6914: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:389"},"thread_name":"main","hostname":"forge-e2e-pr-6914-1678348197-50fdc31064330efd94ef973254ce970547","timestamp":"2023-03-09T07:59:26.292441Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-6914"}
failures:
performance benchmark with full nodes
test result: FAILED. 0 passed; 1 failed; 0 filtered out
Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
NAME READY STATUS RESTARTS AGE
aptos-node-0-fullnode-eforge148-0 1/1 Running 0 6m39s
aptos-node-0-validator-0 1/1 Running 0 6m39s
aptos-node-1-fullnode-eforge148-0 1/1 Running 0 6m39s
aptos-node-1-validator-0 1/1 Running 0 6m39s
aptos-node-10-validator-0 1/1 Running 0 6m39s
aptos-node-11-validator-0 1/1 Running 0 6m38s
aptos-node-12-validator-0 1/1 Running 0 6m38s
aptos-node-13-validator-0 1/1 Running 0 6m38s
aptos-node-14-validator-0 1/1 Running 0 6m39s
aptos-node-15-validator-0 1/1 Running 0 6m39s
aptos-node-16-validator-0 1/1 Running 0 6m39s
aptos-node-17-validator-0 1/1 Running 0 6m39s
aptos-node-18-validator-0 1/1 Running 0 6m39s
aptos-node-19-validator-0 1/1 Running 0 6m39s
aptos-node-2-fullnode-eforge148-0 1/1 Running 0 6m39s
aptos-node-2-validator-0 1/1 Running 0 6m39s
aptos-node-3-fullnode-eforge148-0 1/1 Running 0 6m39s
aptos-node-3-validator-0 1/1 Running 0 6m38s
aptos-node-4-fullnode-eforge148-0 1/1 Running 0 6m39s
aptos-node-4-validator-0 1/1 Running 0 6m38s
aptos-node-5-fullnode-eforge148-0 1/1 Running 0 6m39s
aptos-node-5-validator-0 1/1 Running 0 6m39s
aptos-node-6-fullnode-eforge148-0 1/1 Running 0 6m39s
aptos-node-6-validator-0 1/1 Running 0 6m39s
aptos-node-7-fullnode-eforge148-0 1/1 Running 0 6m39s
aptos-node-7-validator-0 1/1 Running 0 6m39s
aptos-node-8-fullnode-eforge148-0 1/1 Running 0 6m38s
aptos-node-8-validator-0 1/1 Running 0 6m39s
aptos-node-9-fullnode-eforge148-0 1/1 Running 0 6m39s
aptos-node-9-validator-0 1/1 Running 0 6m39s
genesis-aptos-genesis-eforge148-tlljc 0/1 Completed 0 7m7s
framework_upgrade on cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> d3189b626a08763f2719d0804fff3c8649a3e5f6github-actions[bot]
commented
1 year ago land_blocking on d3189b626a08763f2719d0804fff3c8649a3e5f6github-actions[bot]
commented
1 year ago compat on testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> d3189b626a08763f2719d0804fff3c8649a3e5f6github-actions[bot]
commented
1 year ago consensus_only_perf_benchmark on consensus_only_perf_test_d3189b626a08763f2719d0804fff3c8649a3e5f6 ==> d3189b626a08763f2719d0804fff3c8649a3e5f6github-actions[bot]
commented
1 year ago land_blocking failure on d3189b626a08763f2719d0804fff3c8649a3e5f6Test Failed: start emitter job
Caused by:
Failed to create accounts: Account 8ea44f1cd4f71e32231dd2810d5f607b87a1cf35588b16d1420f97278142be7a couldn't mint
Caused by:
Unknown error Ledger on endpoint (http://aptos-node-9-fullnode.forge-e2e-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (c4005b61) might still succeed.
Trailing Log Lines:
Unknown error Ledger on endpoint (http://aptos-node-9-fullnode.forge-e2e-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (c4005b61) might still succeed.
Test Statistics:
Test Failed: start emitter job
Caused by:
Failed to create accounts: Account 8ea44f1cd4f71e32231dd2810d5f607b87a1cf35588b16d1420f97278142be7a couldn't mint
Caused by:
Unknown error Ledger on endpoint (http://aptos-node-9-fullnode.forge-e2e-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (c4005b61) might still succeed.
Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:281"},"thread_name":"main","hostname":"forge-e2e-pr-6914-1678353416-d3189b626a08763f2719d0804fff3c8649","timestamp":"2023-03-09T09:29:12.519717Z","message":"Deleting namespace forge-e2e-pr-6914: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:389"},"thread_name":"main","hostname":"forge-e2e-pr-6914-1678353416-d3189b626a08763f2719d0804fff3c8649","timestamp":"2023-03-09T09:29:12.519741Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-6914"}
failures:
Failed to run tests:
Tests Failed
performance benchmark with full nodes
test result: FAILED. 0 passed; 1 failed; 0 filtered out
Error: Tests Failed
Debugging output:
NAME READY STATUS RESTARTS AGE
aptos-node-0-fullnode-eforge8-0 1/1 Running 0 6m41s
aptos-node-0-validator-0 1/1 Running 0 6m41s
aptos-node-1-fullnode-eforge8-0 1/1 Running 0 6m41s
aptos-node-1-validator-0 1/1 Running 0 6m40s
aptos-node-10-validator-0 1/1 Running 0 6m41s
aptos-node-11-validator-0 1/1 Running 0 6m41s
aptos-node-12-validator-0 1/1 Running 0 6m40s
aptos-node-13-validator-0 1/1 Running 0 6m40s
aptos-node-14-validator-0 1/1 Running 0 6m41s
aptos-node-15-validator-0 1/1 Running 0 6m41s
aptos-node-16-validator-0 1/1 Running 0 6m41s
aptos-node-17-validator-0 1/1 Running 0 6m40s
aptos-node-18-validator-0 1/1 Running 0 6m41s
aptos-node-19-validator-0 1/1 Running 0 6m41s
aptos-node-2-fullnode-eforge8-0 1/1 Running 0 6m41s
aptos-node-2-validator-0 1/1 Running 0 6m40s
aptos-node-3-fullnode-eforge8-0 1/1 Running 0 6m41s
aptos-node-3-validator-0 1/1 Running 0 6m41s
aptos-node-4-fullnode-eforge8-0 1/1 Running 0 6m41s
aptos-node-4-validator-0 1/1 Running 0 6m40s
aptos-node-5-fullnode-eforge8-0 1/1 Running 0 6m41s
aptos-node-5-validator-0 1/1 Running 0 6m40s
aptos-node-6-fullnode-eforge8-0 1/1 Running 0 6m41s
aptos-node-6-validator-0 1/1 Running 0 6m41s
aptos-node-7-fullnode-eforge8-0 1/1 Running 0 6m41s
aptos-node-7-validator-0 1/1 Running 0 6m40s
aptos-node-8-fullnode-eforge8-0 1/1 Running 0 6m41s
aptos-node-8-validator-0 1/1 Running 0 6m41s
aptos-node-9-fullnode-eforge8-0 1/1 Running 0 6m41s
aptos-node-9-validator-0 1/1 Running 0 6m41s
genesis-aptos-genesis-eforge8-gfjf5 0/1 Completed 0 9m42s
compat failure on testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> d3189b626a08763f2719d0804fff3c8649a3e5f6Compatibility test results for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> d3189b626a08763f2719d0804fff3c8649a3e5f6 (PR)
1. Check liveness of validators at old version: testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b
compatibility::simple-validator-upgrade::liveness-check : 7884 TPS, 4843 ms latency, 7400 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: d3189b626a08763f2719d0804fff3c8649a3e5f6
compatibility::simple-validator-upgrade::single-validator-upgrade : 3045 TPS, 13673 ms latency, 21200 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: d3189b626a08763f2719d0804fff3c8649a3e5f6
Test Failed: Unknown error Ledger on endpoint (http://aptos-node-0-validator.forge-compat-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (0fd0c8e2) might still succeed.
Trailing Log Lines:
Error: Unknown error Ledger on endpoint (http://aptos-node-0-validator.forge-compat-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (0fd0c8e2) might still succeed.
Test Statistics:
Compatibility test results for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> d3189b626a08763f2719d0804fff3c8649a3e5f6 (PR)
1. Check liveness of validators at old version: testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b
compatibility::simple-validator-upgrade::liveness-check : 7884 TPS, 4843 ms latency, 7400 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: d3189b626a08763f2719d0804fff3c8649a3e5f6
compatibility::simple-validator-upgrade::single-validator-upgrade : 3045 TPS, 13673 ms latency, 21200 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: d3189b626a08763f2719d0804fff3c8649a3e5f6
Test Failed: Unknown error Ledger on endpoint (http://aptos-node-0-validator.forge-compat-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (0fd0c8e2) might still succeed.
Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:281"},"thread_name":"main","hostname":"forge-compat-pr-6914-1678353422-testnet-2d8b1b57553d869190f61df","timestamp":"2023-03-09T09:30:02.387609Z","message":"Deleting namespace forge-compat-pr-6914: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:389"},"thread_name":"main","hostname":"forge-compat-pr-6914-1678353422-testnet-2d8b1b57553d869190f61df","timestamp":"2023-03-09T09:30:02.387635Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-6914"}
failures:
compatibility::simple-validator-upgrade
test result: FAILED. 0 passed; 1 failed; 0 filtered out
Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
framework_upgrade failure on cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> d3189b626a08763f2719d0804fff3c8649a3e5f6Compatibility test results for cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> d3189b626a08763f2719d0804fff3c8649a3e5f6 (PR)
Upgrade the nodes to version: d3189b626a08763f2719d0804fff3c8649a3e5f6
Test Failed: Waiting for nodes to catch up to target version and epoch (Some(805), None) timed out after 10 seconds, current status: [("validator-0", 805, 6), ("validator-1", 805, 6), ("validator-2", 805, 6), ("validator-3", 783, 6), ("validator-4", 783, 6)]
Trailing Log Lines:
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:529"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-6914-1678353394-cb4ba0a57c998c60cbab","timestamp":"2023-03-09T09:36:17.698491Z","message":"Node validator-0 healthy @ version 805 > 100"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:529"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-6914-1678353394-cb4ba0a57c998c60cbab","timestamp":"2023-03-09T09:36:17.704448Z","message":"Node validator-4 healthy @ version 783 > 100"}
::error::Waiting for nodes to catch up to target version and epoch (Some(805), None) timed out after 10 seconds, current status: [("validator-0", 805, 6), ("validator-1", 805, 6), ("validator-2", 805, 6), ("validator-3", 783, 6), ("validator-4", 783, 6)]
test framework_upgrade::framework-upgrade ... FAILED
Error: Waiting for nodes to catch up to target version and epoch (Some(805), None) timed out after 10 seconds, current status: [("validator-0", 805, 6), ("validator-1", 805, 6), ("validator-2", 805, 6), ("validator-3", 783, 6), ("validator-4", 783, 6)]
Test Statistics:
Compatibility test results for cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> d3189b626a08763f2719d0804fff3c8649a3e5f6 (PR)
Upgrade the nodes to version: d3189b626a08763f2719d0804fff3c8649a3e5f6
Test Failed: Waiting for nodes to catch up to target version and epoch (Some(805), None) timed out after 10 seconds, current status: [("validator-0", 805, 6), ("validator-1", 805, 6), ("validator-2", 805, 6), ("validator-3", 783, 6), ("validator-4", 783, 6)]
Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:281"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-6914-1678353394-cb4ba0a57c998c60cbab","timestamp":"2023-03-09T09:36:27.845552Z","message":"Deleting namespace forge-framework-upgrade-pr-6914: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:389"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-6914-1678353394-cb4ba0a57c998c60cbab","timestamp":"2023-03-09T09:36:27.845574Z","message":"aptos-node resources for Forge removed in namespace: forge-framework-upgrade-pr-6914"}
failures:
framework_upgrade::framework-upgrade
test result: FAILED. 0 passed; 1 failed; 0 filtered out
Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
NAME READY STATUS RESTARTS AGE
aptos-node-0-validator-0 1/1 Running 0 9m17s
aptos-node-1-validator-0 1/1 Running 0 7m27s
aptos-node-2-validator-0 1/1 Running 0 5m36s
aptos-node-3-validator-0 1/1 Running 0 3m46s
aptos-node-4-validator-0 1/1 Running 0 116s
genesis-aptos-genesis-eforge124-gfkfm 0/1 Completed 0 17m
consensus_only_perf_benchmark failure on consensus_only_perf_test_d3189b626a08763f2719d0804fff3c8649a3e5f6 ==> d3189b626a08763f2719d0804fff3c8649a3e5f6Test Failed: start emitter job
Caused by:
Failed to create accounts: Account 0a5a7d8f209102e2662b9f0e61f001b32c228c766190bad594476c2735377a52 couldn't mint
Caused by:
Unknown error Ledger on endpoint (http://aptos-node-13-validator.forge-consensus-only-perf-test-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (7fe133ee) might still succeed.
Trailing Log Lines:
Unknown error Ledger on endpoint (http://aptos-node-13-validator.forge-consensus-only-perf-test-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (7fe133ee) might still succeed.
Test Statistics:
Test Failed: start emitter job
Caused by:
Failed to create accounts: Account 0a5a7d8f209102e2662b9f0e61f001b32c228c766190bad594476c2735377a52 couldn't mint
Caused by:
Unknown error Ledger on endpoint (http://aptos-node-13-validator.forge-consensus-only-perf-test-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (7fe133ee) might still succeed.
Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:281"},"thread_name":"main","hostname":"forge-consensus-only-perf-test-pr-6914-1678353424-consensus-onl","timestamp":"2023-03-09T09:41:49.539931Z","message":"Deleting namespace forge-consensus-only-perf-test-pr-6914: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:389"},"thread_name":"main","hostname":"forge-consensus-only-perf-test-pr-6914-1678353424-consensus-onl","timestamp":"2023-03-09T09:41:49.539953Z","message":"aptos-node resources for Forge removed in namespace: forge-consensus-only-perf-test-pr-6914"}
failures:
continuous progress test
test result: FAILED. 0 passed; 1 failed; 0 filtered out
Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
NAME READY STATUS RESTARTS AGE
aptos-node-0-validator-0 1/1 Running 0 18m
aptos-node-1-validator-0 1/1 Running 0 18m
aptos-node-10-validator-0 1/1 Running 0 18m
aptos-node-11-validator-0 1/1 Running 0 18m
aptos-node-12-validator-0 1/1 Running 0 18m
aptos-node-13-validator-0 1/1 Running 0 18m
aptos-node-14-validator-0 1/1 Running 0 18m
aptos-node-15-validator-0 1/1 Running 0 18m
aptos-node-16-validator-0 1/1 Running 0 18m
aptos-node-17-validator-0 1/1 Running 0 18m
aptos-node-18-validator-0 1/1 Running 0 18m
aptos-node-19-validator-0 1/1 Running 0 18m
aptos-node-2-validator-0 1/1 Running 0 18m
aptos-node-3-validator-0 1/1 Running 0 18m
aptos-node-4-validator-0 1/1 Running 0 18m
aptos-node-5-validator-0 1/1 Running 0 18m
aptos-node-6-validator-0 1/1 Running 0 18m
aptos-node-7-validator-0 1/1 Running 0 18m
aptos-node-8-validator-0 1/1 Running 0 18m
aptos-node-9-validator-0 1/1 Running 0 18m
genesis-aptos-genesis-eforge189-f6hj2 0/1 Completed 0 22m
consensus_only_perf_benchmark on consensus_only_perf_test_07ef23644b8276e0303c4df7d448646b25608d53 ==> 07ef23644b8276e0303c4df7d448646b25608d53github-actions[bot]
commented
1 year ago framework_upgrade on cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> 07ef23644b8276e0303c4df7d448646b25608d53github-actions[bot]
commented
1 year ago compat on testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 07ef23644b8276e0303c4df7d448646b25608d53github-actions[bot]
commented
1 year ago land_blocking on 07ef23644b8276e0303c4df7d448646b25608d53github-actions[bot]
commented
1 year ago land_blocking failure on 07ef23644b8276e0303c4df7d448646b25608d53Test Failed: start emitter job
Caused by:
Unknown error Ledger on endpoint (http://aptos-node-0-validator.forge-e2e-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (52b927d7) might still succeed.
Trailing Log Lines:
Error: start emitter job
Caused by:
Unknown error Ledger on endpoint (http://aptos-node-0-validator.forge-e2e-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (52b927d7) might still succeed.
Test Statistics:
Test Failed: start emitter job
Caused by:
Unknown error Ledger on endpoint (http://aptos-node-0-validator.forge-e2e-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (52b927d7) might still succeed.
Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:281"},"thread_name":"main","hostname":"forge-e2e-pr-6914-1678388858-07ef23644b8276e0303c4df7d448646b25","timestamp":"2023-03-09T19:19:19.272661Z","message":"Deleting namespace forge-e2e-pr-6914: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:389"},"thread_name":"main","hostname":"forge-e2e-pr-6914-1678388858-07ef23644b8276e0303c4df7d448646b25","timestamp":"2023-03-09T19:19:19.272684Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-6914"}
failures:
performance benchmark with full nodes
test result: FAILED. 0 passed; 1 failed; 0 filtered out
Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
NAME READY STATUS RESTARTS AGE
aptos-node-0-validator-0 1/1 Running 0 7m50s
aptos-node-1-validator-0 1/1 Running 0 7m50s
aptos-node-2-validator-0 1/1 Running 0 7m50s
aptos-node-3-validator-0 1/1 Running 0 7m50s
genesis-aptos-genesis-eforge0-qqdvg 0/1 Completed 0 10m
consensus_only_perf_benchmark failure on consensus_only_perf_test_07ef23644b8276e0303c4df7d448646b25608d53 ==> 07ef23644b8276e0303c4df7d448646b25608d53Test Failed: start emitter job
Caused by:
Unknown error Ledger on endpoint (http://aptos-node-11-validator.forge-consensus-only-perf-test-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (7abfb937) might still succeed.
Trailing Log Lines:
Error: start emitter job
Caused by:
Unknown error Ledger on endpoint (http://aptos-node-11-validator.forge-consensus-only-perf-test-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (7abfb937) might still succeed.
Test Statistics:
Test Failed: start emitter job
Caused by:
Unknown error Ledger on endpoint (http://aptos-node-11-validator.forge-consensus-only-perf-test-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (7abfb937) might still succeed.
Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:281"},"thread_name":"main","hostname":"forge-consensus-only-perf-test-pr-6914-1678388720-consensus-onl","timestamp":"2023-03-09T19:23:43.318068Z","message":"Deleting namespace forge-consensus-only-perf-test-pr-6914: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:389"},"thread_name":"main","hostname":"forge-consensus-only-perf-test-pr-6914-1678388720-consensus-onl","timestamp":"2023-03-09T19:23:43.318091Z","message":"aptos-node resources for Forge removed in namespace: forge-consensus-only-perf-test-pr-6914"}
failures:
continuous progress test
test result: FAILED. 0 passed; 1 failed; 0 filtered out
Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
NAME READY STATUS RESTARTS AGE
aptos-node-0-validator-0 1/1 Running 0 13m
aptos-node-1-validator-0 1/1 Running 0 13m
aptos-node-10-validator-0 1/1 Running 0 13m
aptos-node-11-validator-0 1/1 Running 0 13m
aptos-node-12-validator-0 1/1 Running 0 13m
aptos-node-13-validator-0 1/1 Running 0 13m
aptos-node-14-validator-0 1/1 Running 0 13m
aptos-node-15-validator-0 1/1 Running 0 13m
aptos-node-16-validator-0 1/1 Running 0 13m
aptos-node-17-validator-0 1/1 Running 0 13m
aptos-node-18-validator-0 1/1 Running 0 13m
aptos-node-19-validator-0 1/1 Running 0 13m
aptos-node-2-validator-0 1/1 Running 0 13m
aptos-node-3-validator-0 1/1 Running 0 13m
aptos-node-4-validator-0 1/1 Running 0 13m
aptos-node-5-validator-0 1/1 Running 0 13m
aptos-node-6-validator-0 1/1 Running 0 13m
aptos-node-7-validator-0 1/1 Running 0 13m
aptos-node-8-validator-0 1/1 Running 0 13m
aptos-node-9-validator-0 1/1 Running 0 13m
genesis-aptos-genesis-eforge57-2wlmw 0/1 Completed 0 16m
compat failure on testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 07ef23644b8276e0303c4df7d448646b25608d53Compatibility test results for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 07ef23644b8276e0303c4df7d448646b25608d53 (PR)
1. Check liveness of validators at old version: testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b
compatibility::simple-validator-upgrade::liveness-check : 8023 TPS, 4763 ms latency, 6900 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 07ef23644b8276e0303c4df7d448646b25608d53
compatibility::simple-validator-upgrade::single-validator-upgrade : 2997 TPS, 13917 ms latency, 18500 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: 07ef23644b8276e0303c4df7d448646b25608d53
Test Failed: Unknown error Ledger on endpoint (http://aptos-node-0-validator.forge-compat-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (6894bdbb) might still succeed.
Trailing Log Lines:
Error: Unknown error Ledger on endpoint (http://aptos-node-0-validator.forge-compat-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (6894bdbb) might still succeed.
Test Statistics:
Compatibility test results for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 07ef23644b8276e0303c4df7d448646b25608d53 (PR)
1. Check liveness of validators at old version: testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b
compatibility::simple-validator-upgrade::liveness-check : 8023 TPS, 4763 ms latency, 6900 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 07ef23644b8276e0303c4df7d448646b25608d53
compatibility::simple-validator-upgrade::single-validator-upgrade : 2997 TPS, 13917 ms latency, 18500 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: 07ef23644b8276e0303c4df7d448646b25608d53
Test Failed: Unknown error Ledger on endpoint (http://aptos-node-0-validator.forge-compat-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (6894bdbb) might still succeed.
Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:281"},"thread_name":"main","hostname":"forge-compat-pr-6914-1678388845-testnet-2d8b1b57553d869190f61df","timestamp":"2023-03-09T19:24:31.403335Z","message":"Deleting namespace forge-compat-pr-6914: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:389"},"thread_name":"main","hostname":"forge-compat-pr-6914-1678388845-testnet-2d8b1b57553d869190f61df","timestamp":"2023-03-09T19:24:31.403364Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-6914"}
failures:
compatibility::simple-validator-upgrade
test result: FAILED. 0 passed; 1 failed; 0 filtered out
Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
NAME READY STATUS RESTARTS AGE
aptos-node-0-validator-0 1/1 Running 0 3m47s
aptos-node-1-validator-0 1/1 Running 0 7m19s
aptos-node-2-validator-0 1/1 Running 0 12m
aptos-node-3-validator-0 1/1 Running 0 12m
aptos-node-4-validator-0 1/1 Running 0 12m
genesis-aptos-genesis-eforge211-j6q9h 0/1 Completed 0 14m
framework_upgrade failure on cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> 07ef23644b8276e0303c4df7d448646b25608d53Compatibility test results for cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> 07ef23644b8276e0303c4df7d448646b25608d53 (PR)
Upgrade the nodes to version: 07ef23644b8276e0303c4df7d448646b25608d53
Test Failed: Simulation failed with status: Move abort in 0x1::aptos_governance: EPROPOSAL_NOT_RESOLVABLE_YET(0x10006): Proposal is not ready to be resolved. Waiting on time or votes
Trailing Log Lines:
Script Hash: 0277034e
Voting proposal id 0
::error::Simulation failed with status: Move abort in 0x1::aptos_governance: EPROPOSAL_NOT_RESOLVABLE_YET(0x10006): Proposal is not ready to be resolved. Waiting on time or votes
test framework_upgrade::framework-upgrade ... FAILED
Error: Simulation failed with status: Move abort in 0x1::aptos_governance: EPROPOSAL_NOT_RESOLVABLE_YET(0x10006): Proposal is not ready to be resolved. Waiting on time or votes
Test Statistics:
Compatibility test results for cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> 07ef23644b8276e0303c4df7d448646b25608d53 (PR)
Upgrade the nodes to version: 07ef23644b8276e0303c4df7d448646b25608d53
Test Failed: Simulation failed with status: Move abort in 0x1::aptos_governance: EPROPOSAL_NOT_RESOLVABLE_YET(0x10006): Proposal is not ready to be resolved. Waiting on time or votes
Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:281"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-6914-1678388841-cb4ba0a57c998c60cbab","timestamp":"2023-03-09T19:26:34.124982Z","message":"Deleting namespace forge-framework-upgrade-pr-6914: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:389"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-6914-1678388841-cb4ba0a57c998c60cbab","timestamp":"2023-03-09T19:26:34.125006Z","message":"aptos-node resources for Forge removed in namespace: forge-framework-upgrade-pr-6914"}
failures:
framework_upgrade::framework-upgrade
test result: FAILED. 0 passed; 1 failed; 0 filtered out
Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
NAME READY STATUS RESTARTS AGE
aptos-node-1-validator-0 1/1 Running 0 7m47s
aptos-node-2-validator-0 1/1 Running 0 6m16s
aptos-node-3-validator-0 1/1 Running 0 4m56s
aptos-node-4-validator-0 1/1 Running 0 3m36s
genesis-aptos-genesis-eforge237-5wz2q 0/1 Completed 0 16m
consensus_only_perf_benchmark on consensus_only_perf_test_7460fc7d838c1e8596e0d697bcd9c48e477c4e6d ==> 7460fc7d838c1e8596e0d697bcd9c48e477c4e6dgithub-actions[bot]
commented
1 year ago framework_upgrade on cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> 7460fc7d838c1e8596e0d697bcd9c48e477c4e6dgithub-actions[bot]
commented
1 year ago compat on testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 7460fc7d838c1e8596e0d697bcd9c48e477c4e6dgithub-actions[bot]
commented
1 year ago land_blocking on 7460fc7d838c1e8596e0d697bcd9c48e477c4e6dgithub-actions[bot]
commented
1 year ago compat failure on testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 7460fc7d838c1e8596e0d697bcd9c48e477c4e6dCompatibility test results for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 7460fc7d838c1e8596e0d697bcd9c48e477c4e6d (PR)
1. Check liveness of validators at old version: testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b
compatibility::simple-validator-upgrade::liveness-check : 8003 TPS, 4793 ms latency, 7700 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 7460fc7d838c1e8596e0d697bcd9c48e477c4e6d
compatibility::simple-validator-upgrade::single-validator-upgrade : 3128 TPS, 14334 ms latency, 21600 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: 7460fc7d838c1e8596e0d697bcd9c48e477c4e6d
Test Failed: Unknown error Ledger on endpoint (http://aptos-node-0-validator.forge-compat-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (37838c72) might still succeed.
Trailing Log Lines:
Error: Unknown error Ledger on endpoint (http://aptos-node-0-validator.forge-compat-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (37838c72) might still succeed.
Test Statistics:
Compatibility test results for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 7460fc7d838c1e8596e0d697bcd9c48e477c4e6d (PR)
1. Check liveness of validators at old version: testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b
compatibility::simple-validator-upgrade::liveness-check : 8003 TPS, 4793 ms latency, 7700 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 7460fc7d838c1e8596e0d697bcd9c48e477c4e6d
compatibility::simple-validator-upgrade::single-validator-upgrade : 3128 TPS, 14334 ms latency, 21600 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: 7460fc7d838c1e8596e0d697bcd9c48e477c4e6d
Test Failed: Unknown error Ledger on endpoint (http://aptos-node-0-validator.forge-compat-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (37838c72) might still succeed.
Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:281"},"thread_name":"main","hostname":"forge-compat-pr-6914-1678401240-testnet-2d8b1b57553d869190f61df","timestamp":"2023-03-09T22:48:05.221564Z","message":"Deleting namespace forge-compat-pr-6914: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:389"},"thread_name":"main","hostname":"forge-compat-pr-6914-1678401240-testnet-2d8b1b57553d869190f61df","timestamp":"2023-03-09T22:48:05.221586Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-6914"}
Failed to run tests:
Tests Failed
failures:
compatibility::simple-validator-upgrade
test result: FAILED. 0 passed; 1 failed; 0 filtered out
Error: Tests Failed
Debugging output:
NAME READY STATUS RESTARTS AGE
aptos-node-0-validator-0 1/1 Running 0 3m48s
aptos-node-1-validator-0 1/1 Running 0 8m21s
aptos-node-2-validator-0 1/1 Running 0 12m
aptos-node-3-validator-0 1/1 Running 0 12m
aptos-node-4-validator-0 1/1 Running 0 12m
genesis-aptos-genesis-eforge202-h75k7 0/1 Completed 0 12m
framework_upgrade failure on cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> 7460fc7d838c1e8596e0d697bcd9c48e477c4e6dCompatibility test results for cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> 7460fc7d838c1e8596e0d697bcd9c48e477c4e6d (PR)
Upgrade the nodes to version: 7460fc7d838c1e8596e0d697bcd9c48e477c4e6d
Test Failed: Waiting for nodes to catch up to target version and epoch (Some(817), None) timed out after 10 seconds, current status: [("validator-0", 813, 6), ("validator-1", 817, 6), ("validator-2", 817, 6), ("validator-3", 817, 6), ("validator-4", 817, 6)]
Trailing Log Lines:
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:529"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-6914-1678401233-cb4ba0a57c998c60cbab","timestamp":"2023-03-09T22:51:28.086717Z","message":"Node validator-2 healthy @ version 817 > 100"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:529"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-6914-1678401233-cb4ba0a57c998c60cbab","timestamp":"2023-03-09T22:51:28.093499Z","message":"Node validator-4 healthy @ version 817 > 100"}
::error::Waiting for nodes to catch up to target version and epoch (Some(817), None) timed out after 10 seconds, current status: [("validator-0", 813, 6), ("validator-1", 817, 6), ("validator-2", 817, 6), ("validator-3", 817, 6), ("validator-4", 817, 6)]
test framework_upgrade::framework-upgrade ... FAILED
Error: Waiting for nodes to catch up to target version and epoch (Some(817), None) timed out after 10 seconds, current status: [("validator-0", 813, 6), ("validator-1", 817, 6), ("validator-2", 817, 6), ("validator-3", 817, 6), ("validator-4", 817, 6)]
Test Statistics:
Compatibility test results for cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> 7460fc7d838c1e8596e0d697bcd9c48e477c4e6d (PR)
Upgrade the nodes to version: 7460fc7d838c1e8596e0d697bcd9c48e477c4e6d
Test Failed: Waiting for nodes to catch up to target version and epoch (Some(817), None) timed out after 10 seconds, current status: [("validator-0", 813, 6), ("validator-1", 817, 6), ("validator-2", 817, 6), ("validator-3", 817, 6), ("validator-4", 817, 6)]
Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:281"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-6914-1678401233-cb4ba0a57c998c60cbab","timestamp":"2023-03-09T22:51:38.234912Z","message":"Deleting namespace forge-framework-upgrade-pr-6914: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:389"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-6914-1678401233-cb4ba0a57c998c60cbab","timestamp":"2023-03-09T22:51:38.234934Z","message":"aptos-node resources for Forge removed in namespace: forge-framework-upgrade-pr-6914"}
failures:
framework_upgrade::framework-upgrade
test result: FAILED. 0 passed; 1 failed; 0 filtered out
Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
NAME READY STATUS RESTARTS AGE
aptos-node-0-validator-0 1/1 Running 0 7m47s
aptos-node-1-validator-0 1/1 Running 0 5m56s
aptos-node-2-validator-0 1/1 Running 0 4m36s
aptos-node-3-validator-0 1/1 Running 0 3m16s
aptos-node-4-validator-0 1/1 Running 0 86s
genesis-aptos-genesis-eforge58-zrznj 0/1 Completed 0 15m
land_blocking on 51bb651ed6c3a27e50d1358e61e65e5cd1f8086egithub-actions[bot]
commented
1 year ago consensus_only_perf_benchmark on consensus_only_perf_test_51bb651ed6c3a27e50d1358e61e65e5cd1f8086e ==> 51bb651ed6c3a27e50d1358e61e65e5cd1f8086egithub-actions[bot]
commented
1 year ago framework_upgrade on cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> 51bb651ed6c3a27e50d1358e61e65e5cd1f8086egithub-actions[bot]
commented
1 year ago compat on testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 51bb651ed6c3a27e50d1358e61e65e5cd1f8086egithub-actions[bot]
commented
1 year ago compat failure on testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 51bb651ed6c3a27e50d1358e61e65e5cd1f8086eCompatibility test results for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 51bb651ed6c3a27e50d1358e61e65e5cd1f8086e (PR)
1. Check liveness of validators at old version: testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b
compatibility::simple-validator-upgrade::liveness-check : 8226 TPS, 4677 ms latency, 6500 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 51bb651ed6c3a27e50d1358e61e65e5cd1f8086e
compatibility::simple-validator-upgrade::single-validator-upgrade : 3158 TPS, 15136 ms latency, 21200 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: 51bb651ed6c3a27e50d1358e61e65e5cd1f8086e
Test Failed: Unknown error Ledger on endpoint (http://aptos-node-0-validator.forge-compat-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (bd27f5aa) might still succeed.
Trailing Log Lines:
Error: Unknown error Ledger on endpoint (http://aptos-node-0-validator.forge-compat-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (bd27f5aa) might still succeed.
Test Statistics:
Compatibility test results for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 51bb651ed6c3a27e50d1358e61e65e5cd1f8086e (PR)
1. Check liveness of validators at old version: testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b
compatibility::simple-validator-upgrade::liveness-check : 8226 TPS, 4677 ms latency, 6500 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 51bb651ed6c3a27e50d1358e61e65e5cd1f8086e
compatibility::simple-validator-upgrade::single-validator-upgrade : 3158 TPS, 15136 ms latency, 21200 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: 51bb651ed6c3a27e50d1358e61e65e5cd1f8086e
Test Failed: Unknown error Ledger on endpoint (http://aptos-node-0-validator.forge-compat-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (bd27f5aa) might still succeed.
Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:281"},"thread_name":"main","hostname":"forge-compat-pr-6914-1678751426-testnet-2d8b1b57553d869190f61df","timestamp":"2023-03-14T00:02:07.323516Z","message":"Deleting namespace forge-compat-pr-6914: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:389"},"thread_name":"main","hostname":"forge-compat-pr-6914-1678751426-testnet-2d8b1b57553d869190f61df","timestamp":"2023-03-14T00:02:07.323539Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-6914"}
failures:
compatibility::simple-validator-upgrade
test result: FAILED. 0 passed; 1 failed; 0 filtered out
Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
framework_upgrade failure on cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> 51bb651ed6c3a27e50d1358e61e65e5cd1f8086eCompatibility test results for cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> 51bb651ed6c3a27e50d1358e61e65e5cd1f8086e (PR)
Upgrade the nodes to version: 51bb651ed6c3a27e50d1358e61e65e5cd1f8086e
Test Failed: API error: Unknown error Ledger on endpoint (http://aptos-node-4-validator.forge-framework-upgrade-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (e15762d2) might still succeed.
Trailing Log Lines:
INCLUDING DEPENDENCY MoveStdlib
BUILDING RunScript
::error::API error: Unknown error Ledger on endpoint (http://aptos-node-4-validator.forge-framework-upgrade-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (e15762d2) might still succeed.
test framework_upgrade::framework-upgrade ... FAILED
Error: API error: Unknown error Ledger on endpoint (http://aptos-node-4-validator.forge-framework-upgrade-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (e15762d2) might still succeed.
Test Statistics:
Compatibility test results for cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> 51bb651ed6c3a27e50d1358e61e65e5cd1f8086e (PR)
Upgrade the nodes to version: 51bb651ed6c3a27e50d1358e61e65e5cd1f8086e
Test Failed: API error: Unknown error Ledger on endpoint (http://aptos-node-4-validator.forge-framework-upgrade-pr-6914.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (e15762d2) might still succeed.
Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:281"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-6914-1678751414-cb4ba0a57c998c60cbab","timestamp":"2023-03-14T00:04:14.198392Z","message":"Deleting namespace forge-framework-upgrade-pr-6914: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:389"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-6914-1678751414-cb4ba0a57c998c60cbab","timestamp":"2023-03-14T00:04:14.198418Z","message":"aptos-node resources for Forge removed in namespace: forge-framework-upgrade-pr-6914"}
failures:
framework_upgrade::framework-upgrade
test result: FAILED. 0 passed; 1 failed; 0 filtered out
Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
land_blocking on 6872a61348b8a37f013d786d4a1b705bc14acff5github-actions[bot]
commented
1 year ago framework_upgrade on cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> 6872a61348b8a37f013d786d4a1b705bc14acff5github-actions[bot]
commented
1 year ago compat on testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 6872a61348b8a37f013d786d4a1b705bc14acff5github-actions[bot]
commented
1 year ago consensus_only_perf_benchmark on consensus_only_perf_test_6872a61348b8a37f013d786d4a1b705bc14acff5 ==> 6872a61348b8a37f013d786d4a1b705bc14acff5
Description
Test Plan