Closed msmouse closed 3 days ago
🚨 1 job on the last run was significantly faster/slower than expected
Job | Duration | vs 7d avg | Delta |
---|---|---|---|
execution-performance / single-node-performance | 26m | 16m |
realistic_env_max_load
on 391f6a7be4e3afee22d15cac52a4b91c7b25654c
compat
on 78db50320d1ba1ca1a61adb189b0b4fde0098f27
==> 391f6a7be4e3afee22d15cac52a4b91c7b25654c
realistic_env_max_load
success on 391f6a7be4e3afee22d15cac52a4b91c7b25654c
two traffics test: inner traffic : committed: 14413.10 txn/s, latency: 2757.18 ms, (p50: 2700 ms, p70: 2700, p90: 3000 ms, p99: 3000 ms), latency samples: 5480180
two traffics test : committed: 99.92 txn/s, latency: 1517.02 ms, (p50: 1400 ms, p70: 1400, p90: 1500 ms, p99: 8000 ms), latency samples: 1740
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 1.989, avg: 1.557", "ConsensusProposalToOrdered: max: 0.315, avg: 0.291", "ConsensusOrderedToCommit: max: 0.383, avg: 0.371", "ConsensusProposalToCommit: max: 0.674, avg: 0.662"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.99s no progress at version 2627829 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.33s no progress at version 2627825 (avg 8.33s) [limit 15].
Test Ok
compat
success on 78db50320d1ba1ca1a61adb189b0b4fde0098f27
==> 391f6a7be4e3afee22d15cac52a4b91c7b25654c
Compatibility test results for 78db50320d1ba1ca1a61adb189b0b4fde0098f27 ==> 391f6a7be4e3afee22d15cac52a4b91c7b25654c (PR)
1. Check liveness of validators at old version: 78db50320d1ba1ca1a61adb189b0b4fde0098f27
compatibility::simple-validator-upgrade::liveness-check : committed: 15968.96 txn/s, latency: 2128.36 ms, (p50: 1900 ms, p70: 2100, p90: 2200 ms, p99: 5800 ms), latency samples: 513820
2. Upgrading first Validator to new version: 391f6a7be4e3afee22d15cac52a4b91c7b25654c
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 7835.09 txn/s, latency: 3658.99 ms, (p50: 4100 ms, p70: 4300, p90: 4300 ms, p99: 4400 ms), latency samples: 143920
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 8068.23 txn/s, latency: 4040.18 ms, (p50: 4300 ms, p70: 4400, p90: 4500 ms, p99: 4500 ms), latency samples: 269720
3. Upgrading rest of first batch to new version: 391f6a7be4e3afee22d15cac52a4b91c7b25654c
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 7651.79 txn/s, latency: 3738.95 ms, (p50: 4200 ms, p70: 4400, p90: 4500 ms, p99: 4600 ms), latency samples: 143240
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 7001.49 txn/s, latency: 4563.14 ms, (p50: 4500 ms, p70: 4600, p90: 7000 ms, p99: 7200 ms), latency samples: 234640
4. upgrading second batch to new version: 391f6a7be4e3afee22d15cac52a4b91c7b25654c
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 12146.47 txn/s, latency: 2261.69 ms, (p50: 2300 ms, p70: 2400, p90: 3300 ms, p99: 3500 ms), latency samples: 209340
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 11597.73 txn/s, latency: 2682.37 ms, (p50: 2500 ms, p70: 2600, p90: 4900 ms, p99: 6300 ms), latency samples: 375900
5. check swarm health
Compatibility test for 78db50320d1ba1ca1a61adb189b0b4fde0098f27 ==> 391f6a7be4e3afee22d15cac52a4b91c7b25654c passed
Test Ok
realistic_env_max_load
on d07f229804ab73eb1b9050f97fa22b677da282aa
compat
on 78db50320d1ba1ca1a61adb189b0b4fde0098f27
==> d07f229804ab73eb1b9050f97fa22b677da282aa
realistic_env_max_load
success on d07f229804ab73eb1b9050f97fa22b677da282aa
two traffics test: inner traffic : committed: 14338.77 txn/s, latency: 2769.45 ms, (p50: 2700 ms, p70: 2700, p90: 3000 ms, p99: 3300 ms), latency samples: 5451920
two traffics test : committed: 99.92 txn/s, latency: 1496.17 ms, (p50: 1400 ms, p70: 1400, p90: 1500 ms, p99: 9800 ms), latency samples: 1820
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 2.031, avg: 1.559", "ConsensusProposalToOrdered: max: 0.322, avg: 0.295", "ConsensusOrderedToCommit: max: 0.400, avg: 0.378", "ConsensusProposalToCommit: max: 0.701, avg: 0.673"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.18s no progress at version 42957 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.38s no progress at version 2810503 (avg 8.38s) [limit 15].
Test Ok
compat
success on 78db50320d1ba1ca1a61adb189b0b4fde0098f27
==> d07f229804ab73eb1b9050f97fa22b677da282aa
Compatibility test results for 78db50320d1ba1ca1a61adb189b0b4fde0098f27 ==> d07f229804ab73eb1b9050f97fa22b677da282aa (PR)
1. Check liveness of validators at old version: 78db50320d1ba1ca1a61adb189b0b4fde0098f27
compatibility::simple-validator-upgrade::liveness-check : committed: 14734.96 txn/s, latency: 2350.68 ms, (p50: 1800 ms, p70: 2000, p90: 3900 ms, p99: 9400 ms), latency samples: 557680
2. Upgrading first Validator to new version: d07f229804ab73eb1b9050f97fa22b677da282aa
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 7135.67 txn/s, latency: 3978.60 ms, (p50: 4500 ms, p70: 4700, p90: 4800 ms, p99: 4900 ms), latency samples: 131180
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 6737.84 txn/s, latency: 4684.13 ms, (p50: 4700 ms, p70: 4800, p90: 7100 ms, p99: 7400 ms), latency samples: 252620
3. Upgrading rest of first batch to new version: d07f229804ab73eb1b9050f97fa22b677da282aa
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 6145.92 txn/s, latency: 4372.51 ms, (p50: 4900 ms, p70: 5100, p90: 5200 ms, p99: 5400 ms), latency samples: 128540
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 6862.93 txn/s, latency: 4665.00 ms, (p50: 4900 ms, p70: 5000, p90: 6700 ms, p99: 6900 ms), latency samples: 234420
4. upgrading second batch to new version: d07f229804ab73eb1b9050f97fa22b677da282aa
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 2626.24 txn/s, submitted: 2626.39 txn/s, expired: 0.15 txn/s, latency: 2503.55 ms, (p50: 2500 ms, p70: 2600, p90: 4000 ms, p99: 4300 ms), latency samples: 188969
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 11378.06 txn/s, latency: 2792.49 ms, (p50: 2600 ms, p70: 2900, p90: 4700 ms, p99: 5900 ms), latency samples: 382620
5. check swarm health
Compatibility test for 78db50320d1ba1ca1a61adb189b0b4fde0098f27 ==> d07f229804ab73eb1b9050f97fa22b677da282aa passed
Test Ok
realistic_env_max_load
on d07f229804ab73eb1b9050f97fa22b677da282aa
framework_upgrade
on 78db50320d1ba1ca1a61adb189b0b4fde0098f27
==> d07f229804ab73eb1b9050f97fa22b677da282aa
compat
on 78db50320d1ba1ca1a61adb189b0b4fde0098f27
==> d07f229804ab73eb1b9050f97fa22b677da282aa
realistic_env_max_load
success on d07f229804ab73eb1b9050f97fa22b677da282aa
two traffics test: inner traffic : committed: 14002.19 txn/s, latency: 2836.65 ms, (p50: 2700 ms, p70: 2700, p90: 3000 ms, p99: 12600 ms), latency samples: 5323980
two traffics test : committed: 100.04 txn/s, latency: 1501.12 ms, (p50: 1400 ms, p70: 1400, p90: 1600 ms, p99: 6000 ms), latency samples: 1740
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 2.018, avg: 1.567", "ConsensusProposalToOrdered: max: 0.328, avg: 0.295", "ConsensusOrderedToCommit: max: 0.380, avg: 0.369", "ConsensusProposalToCommit: max: 0.675, avg: 0.664"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.04s no progress at version 1185119 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.65s no progress at version 5529143 (avg 8.52s) [limit 15].
Test Ok
framework_upgrade
success on 78db50320d1ba1ca1a61adb189b0b4fde0098f27
==> d07f229804ab73eb1b9050f97fa22b677da282aa
Compatibility test results for 78db50320d1ba1ca1a61adb189b0b4fde0098f27 ==> d07f229804ab73eb1b9050f97fa22b677da282aa (PR)
Upgrade the nodes to version: d07f229804ab73eb1b9050f97fa22b677da282aa
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1438.13 txn/s, submitted: 1441.33 txn/s, failed submission: 3.20 txn/s, expired: 3.20 txn/s, latency: 2175.39 ms, (p50: 1800 ms, p70: 2100, p90: 3600 ms, p99: 5600 ms), latency samples: 125880
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1344.07 txn/s, submitted: 1347.57 txn/s, failed submission: 3.50 txn/s, expired: 3.50 txn/s, latency: 2167.18 ms, (p50: 2100 ms, p70: 2400, p90: 2900 ms, p99: 5100 ms), latency samples: 122800
5. check swarm health
Compatibility test for 78db50320d1ba1ca1a61adb189b0b4fde0098f27 ==> d07f229804ab73eb1b9050f97fa22b677da282aa passed
Upgrade the remaining nodes to version: d07f229804ab73eb1b9050f97fa22b677da282aa
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1343.77 txn/s, submitted: 1345.99 txn/s, failed submission: 2.22 txn/s, expired: 2.22 txn/s, latency: 2439.45 ms, (p50: 2100 ms, p70: 2400, p90: 4500 ms, p99: 5700 ms), latency samples: 120940
Test Ok
compat
success on 78db50320d1ba1ca1a61adb189b0b4fde0098f27
==> d07f229804ab73eb1b9050f97fa22b677da282aa
Compatibility test results for 78db50320d1ba1ca1a61adb189b0b4fde0098f27 ==> d07f229804ab73eb1b9050f97fa22b677da282aa (PR)
1. Check liveness of validators at old version: 78db50320d1ba1ca1a61adb189b0b4fde0098f27
compatibility::simple-validator-upgrade::liveness-check : committed: 14767.05 txn/s, latency: 2142.48 ms, (p50: 1800 ms, p70: 1900, p90: 2400 ms, p99: 13600 ms), latency samples: 574740
2. Upgrading first Validator to new version: d07f229804ab73eb1b9050f97fa22b677da282aa
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 7807.58 txn/s, latency: 3532.51 ms, (p50: 3500 ms, p70: 4300, p90: 4700 ms, p99: 5000 ms), latency samples: 144380
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 7573.01 txn/s, latency: 4180.94 ms, (p50: 4300 ms, p70: 4400, p90: 6200 ms, p99: 6500 ms), latency samples: 253120
3. Upgrading rest of first batch to new version: d07f229804ab73eb1b9050f97fa22b677da282aa
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 7718.99 txn/s, latency: 3695.24 ms, (p50: 4000 ms, p70: 4300, p90: 4500 ms, p99: 4600 ms), latency samples: 143780
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 7703.67 txn/s, latency: 4147.44 ms, (p50: 4300 ms, p70: 4400, p90: 5900 ms, p99: 6100 ms), latency samples: 253580
4. upgrading second batch to new version: d07f229804ab73eb1b9050f97fa22b677da282aa
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 11732.26 txn/s, latency: 2346.08 ms, (p50: 2500 ms, p70: 2700, p90: 3000 ms, p99: 3200 ms), latency samples: 203760
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 11108.32 txn/s, latency: 2826.04 ms, (p50: 2700 ms, p70: 2800, p90: 4800 ms, p99: 5900 ms), latency samples: 360540
5. check swarm health
Compatibility test for 78db50320d1ba1ca1a61adb189b0b4fde0098f27 ==> d07f229804ab73eb1b9050f97fa22b677da282aa passed
Test Ok
this is to cherry-pick https://github.com/aptos-labs/aptos-core/pull/15361 to v1.24