Closed cockroach-teamcity closed 3 months ago
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test import/tpch/nodes=8 failed: VMs preempted during the test run: teamcity-16405142-1723268847-69-n8cpu4-0008 (us-east1-d), teamcity-16405142-1723268847-69-n8cpu4-0005 (us-east1-d)
**Other Failures:**
(test_runner.go:1277).runTest: test timed out (10h0m0s)
test artifacts and logs in: /artifacts/import/tpch/nodes=8/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test allocbench/nodes=7/cpu=8/kv/r=50/ops=skew failed: VMs preempted during the test run: teamcity-16414681-1723355265-10-n8cpu8-0007 (us-east1-d)
**Other Failures:**
(assertions.go:363).Fail:
Error Trace: github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:389
github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:339
github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:268
main/pkg/cmd/roachtest/test_runner.go:1248
src/runtime/asm_amd64.s:1695
Error: Received unexpected error:
monitor failure: context canceled
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| main/pkg/cmd/roachtest/monitor.go:142
| [...repeated from below...]
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait
| main/pkg/cmd/roachtest/monitor.go:237
| main.(*monitorImpl).WaitE
| main/pkg/cmd/roachtest/monitor.go:142
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBenchSample
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:388
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBench
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:339
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAllocationBenchSpec.func1
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:268
| main.(*testRunner).runTest.func2
| main/pkg/cmd/roachtest/test_runner.go:1248
| runtime.goexit
| src/runtime/asm_amd64.s:1695
Wraps: (4) context canceled
| EOF [owner=test-eng]
└─ Wraps: (5) EOF [owner=test-eng]
└─ Wraps: (6) context canceled
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *join.joinError (5) registry.ErrorWithOwnership (6) *errors.errorString
Test: allocbench/nodes=7/cpu=8/kv/r=50/ops=skew
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
| | github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:173
| | runtime.goexit
| | src/runtime/asm_amd64.s:1695
| Wraps: (3) _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/access=skew/run_1/ssh/ssh_080909.264066323_n8_cockroach-workload-r.log)
| Wraps: (4) TRANSIENT_ERROR(ssh_problem)
| Wraps: (5) exit status 255
| Error types: (1) *hintdetail.withDetail (2) *withstack.withStack (3) *errutil.withPrefix (4) errors.TransientError (5) *exec.ExitError
└─ Wraps: (11) secondary error attachment
| _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/access=skew/run_1/ssh/ssh_080841.693586527_n8_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
| (1) Node 8. Command with error:
| | ``````
| | ./cockroach workload run kv --db=bigwrites1 --read-percent=0 --min-block-bytes=32768 --max-block-bytes=32768 --max-rate=550 --zipfian --tolerate-errors --concurrency=512 --duration=30m0s {pgurl:1-7}
| | ``````
| | <no output>
| Wraps: (2) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).errWithDebug
| | github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:143
| | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).Run.func1
| | github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:173
| | runtime.goexit
| | src/runtime/asm_amd64.s:1695
| Wraps: (3) _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/access=skew/run_1/ssh/ssh_080841.693586527_n8_cockroach-workload-r.log)
| Wraps: (4) TRANSIENT_ERROR(ssh_problem)
| Wraps: (5) exit status 255
| Error types: (1) *hintdetail.withDetail (2) *withstack.withStack (3) *errutil.withPrefix (4) errors.TransientError (5) *exec.ExitError
└─ Wraps: (12) Node 8. Command with error:
| ``````
| ./cockroach workload run kv --db=bigwrites1 --read-percent=0 --min-block-bytes=32768 --max-block-bytes=32768 --max-rate=550 --zipfian --tolerate-errors --concurrency=512 --duration=30m0s {pgurl:1-7}
| ``````
| <truncated> ... 21 3944 workload/pgx_helpers.go:240 [-] 4340 error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
| W240811 07:57:30.761885 3841 workload/pgx_helpers.go:240 [-] 4341 error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
| W240811 07:57:30.761957 3642 workload/pgx_helpers.go:240 [-] 4342 error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
| W240811 07:57:30.762037 3876 workload/pgx_helpers.go:240 [-] 4343 error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
| Connection to 34.75.35.35 closed by remote host.
└─ Wraps: (13) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).errWithDebug
| github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:143
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).Run.func1
| github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:173
| runtime.goexit
| src/runtime/asm_amd64.s:1695
└─ Wraps: (14) _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/access=skew/run_1/ssh/ssh_075726.594515423_n8_cockroach-workload-r.log)
└─ Wraps: (15) TRANSIENT_ERROR(ssh_problem)
└─ Wraps: (16) exit status 255
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *join.joinError (5) *withstack.withStack (6) *errutil.withPrefix (7) *markers.withMark (8) *withstack.withStack (9) *errutil.withPrefix (10) *secondary.withSecondaryError (11) *secondary.withSecondaryError (12) *hintdetail.withDetail (13) *withstack.withStack (14) *errutil.withPrefix (15) errors.TransientError (16) *exec.ExitError
Test: allocbench/nodes=7/cpu=8/kv/r=0/access=skew
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/access=skew/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test failover/chaos/read-write/lease=expiration failed: VMs preempted during the test run: teamcity-16414681-1723355265-60-n10cpu2-0008 (us-east1-c), teamcity-16414681-1723355265-60-n10cpu2-0006 (us-east1-c)
**Other Failures:**
(failover.go:1774).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
(cluster.go:2419).Run: context canceled
(cluster.go:2419).Run: context canceled
(cluster.go:2419).Run: context canceled
(cluster.go:2419).Run: context canceled
test artifacts and logs in: /artifacts/failover/chaos/read-write/lease=expiration/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test ycsb/E/nodes=3 failed: VMs preempted during the test run: teamcity-16414681-1723355265-101-n4cpu8-0001 (us-east1-d), teamcity-16414681-1723355265-101-n4cpu8-0004 (us-east1-d)
**Other Failures:**
(cluster.go:2419).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/ycsb/E/nodes=3/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test import/tpch/nodes=8 failed: VMs preempted during the test run: teamcity-16414681-1723355265-75-n8cpu4-0006 (us-east1-d), teamcity-16414681-1723355265-75-n8cpu4-0007 (us-east1-d), teamcity-16414681-1723355265-75-n8cpu4-0002 (us-east1-d), teamcity-16414681-1723355265-75-n8cpu4-0001 (us-east1-d), teamcity-16414681-1723355265-75-n8cpu4-0003 (us-east1-d), teamcity-16414681-1723355265-75-n8cpu4-0004 (us-east1-d)
**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/import/tpch/nodes=8/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=true
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test cdc/cloud-sink-gcs/rangefeed=true failed: VMs preempted during the test run: teamcity-16423223-1723441621-31-n4cpu16-0003 (us-east1-b)
**Other Failures:**
(cluster.go:2419).Run: full command output in run_070500.523103643_n4_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
(cdc.go:379).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/cloud-sink-gcs/rangefeed=true/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test decommissionBench/nodes=4/warehouses=1000/while-down failed: VMs preempted during the test run: teamcity-16423223-1723441621-53-n5cpu16-0001 (us-east1-c)
**Other Failures:**
(cluster.go:2419).Run: full command output in run_081910.635394147_n1_cockroach-workload-f.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/decommissionBench/nodes=4/warehouses=1000/while-down/run_1/ssh/ssh_081910.691066034_n1_cockroach-workload-f.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/decommissionBench/nodes=4/warehouses=1000/while-down/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-16423223-1723441621-41-n6cpu16-0003 (us-central1-a), teamcity-16423223-1723441621-41-n6cpu16-0004 (us-central1-a)
**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/cpu_arch=arm64/run_1
Parameters:
ROACHTEST_arch=arm64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test failover/non-system/blackhole-recv failed: VMs preempted during the test run: teamcity-16423223-1723441621-63-n7cpu2-0002 (us-east1-d), teamcity-16423223-1723441621-63-n7cpu2-0006 (us-east1-d), teamcity-16423223-1723441621-63-n7cpu2-0004 (us-east1-d), teamcity-16423223-1723441621-63-n7cpu2-0005 (us-east1-d), teamcity-16423223-1723441621-63-n7cpu2-0001 (us-east1-d), teamcity-16423223-1723441621-63-n7cpu2-0003 (us-east1-d)
**Other Failures:**
(failover.go:1774).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
(cluster.go:2419).Run: context canceled
test artifacts and logs in: /artifacts/failover/non-system/blackhole-recv/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test failover/non-system/blackhole failed: VMs preempted during the test run: teamcity-16423223-1723441621-62-n7cpu2-0006 (us-east1-d), teamcity-16423223-1723441621-62-n7cpu2-0005 (us-east1-d), teamcity-16423223-1723441621-62-n7cpu2-0003 (us-east1-d), teamcity-16423223-1723441621-62-n7cpu2-0002 (us-east1-d), teamcity-16423223-1723441621-62-n7cpu2-0007 (us-east1-d)
**Other Failures:**
(failover.go:1774).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
(cluster.go:2419).Run: context canceled
test artifacts and logs in: /artifacts/failover/non-system/blackhole/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test decommissionBench/nodes=8/warehouses=3000 failed: VMs preempted during the test run: teamcity-16423223-1723441621-57-n9cpu16-0006 (us-east1-d), teamcity-16423223-1723441621-57-n9cpu16-0002 (us-east1-d), teamcity-16423223-1723441621-57-n9cpu16-0005 (us-east1-d), teamcity-16423223-1723441621-57-n9cpu16-0003 (us-east1-d), teamcity-16423223-1723441621-57-n9cpu16-0008 (us-east1-d), teamcity-16423223-1723441621-57-n9cpu16-0007 (us-east1-d), teamcity-16423223-1723441621-57-n9cpu16-0001 (us-east1-d)
**Other Failures:**
(decommissionbench.go:727).runDecommissionBench: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/decommissionBench/nodes=8/warehouses=3000/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test import/tpch/nodes=8 failed: VMs preempted during the test run: teamcity-16423223-1723441621-79-n8cpu4-0001 (us-east1-d), teamcity-16423223-1723441621-79-n8cpu4-0007 (us-east1-d)
**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/import/tpch/nodes=8/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=true
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test kv0/enc=false/nodes=3/cpu=32/mt-shared-process failed: VMs preempted during the test run: teamcity-16423223-1723441621-100-n4cpu32-0001 (us-east1-c)
**Other Failures:**
(cluster.go:2419).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/cpu=32/mt-shared-process/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test kv95/enc=false/nodes=3/lease=expiration failed: VMs preempted during the test run: teamcity-16423223-1723441621-108-n4cpu8-0002 (us-east1-c)
**Other Failures:**
(cluster.go:2419).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/lease=expiration/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test ycsb/D/nodes=3/cpu=32 failed: VMs preempted during the test run: teamcity-16423223-1723441621-111-n4cpu32-0001 (us-east1-c)
**Other Failures:**
(cluster.go:2305).Start: error persisted after 2 attempts: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa ubuntu@34.73.178.122:certs.tar /tmp/certs.tar3701531483
ssh: connect to host 34.73.178.122 port 22: Connection refused
: exit status 1
test artifacts and logs in: /artifacts/ycsb/D/nodes=3/cpu=32/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_encrypted=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test ycsb/D/nodes=3 failed: VMs preempted during the test run: teamcity-16423223-1723441621-112-n4cpu8-0004 (us-east1-c)
**Other Failures:**
(cluster.go:2419).Run: full command output in run_112204.181192195_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/ycsb/D/nodes=3/run_1/ssh/ssh_112204.208755942_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/D/nodes=3/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test sysbench/oltp_update_non_index/nodes=3/cpu=32/conc=256 failed: VMs preempted during the test run: teamcity-16423223-1723441621-141-n4cpu32-0004 (us-east1-c)
**Other Failures:**
(monitor.go:154).Wait: monitor failure: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/sysbench/oltp_update_non_index/nodes=3/cpu=32/conc=256/run_1/ssh/ssh_115611.789278223_n4_sysbench-dbdriverpgs.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/sysbench/oltp_update_non_index/nodes=3/cpu=32/conc=256/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test sysbench/oltp_write_only/nodes=3/cpu=32/conc=256 failed: VMs preempted during the test run: teamcity-16423223-1723441621-142-n4cpu32-0003 (us-east1-c), teamcity-16423223-1723441621-142-n4cpu32-0002 (us-east1-c)
**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/sysbench/oltp_write_only/nodes=3/cpu=32/conc=256/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test ycsb/E/nodes=3/cpu=32 failed: VMs preempted during the test run: teamcity-16423223-1723441621-101-n4cpu32-0004 (us-east1-c)
**Other Failures:**
(cluster.go:2419).Run: full command output in run_120400.971754331_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/ycsb/E/nodes=3/cpu=32/run_1/ssh/ssh_120400.996212611_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/E/nodes=3/cpu=32/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_encrypted=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test tpcc/headroom/n4cpu16 failed: VMs preempted during the test run: teamcity-16423223-1723441621-143-n4cpu16-0004 (us-east1-c)
**Other Failures:**
(monitor.go:154).Wait: monitor failure: full command output in run_115850.659111753_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpcc/headroom/n4cpu16/run_1/ssh/ssh_115850.684107321_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpcc/headroom/n4cpu16/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test ycsb/F/nodes=3/cpu=32/uniform failed: VMs preempted during the test run: teamcity-16423223-1723441621-154-n4cpu32-0002 (us-east1-c)
**Other Failures:**
(cluster.go:2419).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/ycsb/F/nodes=3/cpu=32/uniform/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test ycsb/F/nodes=3/cpu=32/isolation-level=read-committed failed: VMs preempted during the test run: teamcity-16423223-1723441621-156-n4cpu32-0002 (us-east1-c), teamcity-16423223-1723441621-156-n4cpu32-0003 (us-east1-c), teamcity-16423223-1723441621-156-n4cpu32-0001 (us-east1-c)
**Other Failures:**
(cluster.go:2419).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/ycsb/F/nodes=3/cpu=32/isolation-level=read-committed/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test tpccbench/nodes=3/cpu=16/lease=expiration failed: VMs preempted during the test run: teamcity-16423223-1723441621-146-n4cpu16-0001 (us-east1-b), teamcity-16423223-1723441621-146-n4cpu16-0003 (us-east1-b), teamcity-16423223-1723441621-146-n4cpu16-0004 (us-east1-b)
**Other Failures:**
(cluster.go:2305).Start: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=16/lease=expiration/run_1/ssh/ssh_134354.880611757_n3_upload-start-script.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=16/lease=expiration/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test backup/2TB/n10cpu4 failed: VMs preempted during the test run: teamcity-16441860-1723528050-19-n10cpu4-0002 (us-central1-a)
**Other Failures:**
(cluster.go:2419).Run: full command output in run_063757.250257763_n1_cockroach-workload-f.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/backup/2TB/n10cpu4/cpu_arch=arm64/run_1
Parameters:
ROACHTEST_arch=arm64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test failover/non-system/pause failed: VMs preempted during the test run: teamcity-16441860-1723528050-62-n7cpu2-0004 (us-east1-c), teamcity-16441860-1723528050-62-n7cpu2-0006 (us-east1-c)
**Other Failures:**
(failover.go:1774).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/failover/non-system/pause/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_encrypted=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test schemachange/during/tpcc failed: VMs preempted during the test run: teamcity-16441860-1723528050-53-n5cpu16-0004 (us-east1-b), teamcity-16441860-1723528050-53-n5cpu16-0001 (us-east1-b)
**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/schemachange/during/tpcc/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test failover/chaos/read-write/lease=expiration failed: VMs preempted during the test run: teamcity-16441860-1723528050-61-n10cpu2-0010 (us-east1-c)
**Other Failures:**
(monitor.go:154).Wait: monitor failure: full command output in run_090443.114963209_n10_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/failover/chaos/read-write/lease=expiration/run_1/ssh/ssh_090443.149656410_n10_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cluster.go:2419).Run: context canceled
(cluster.go:2419).Run: context canceled
(cluster.go:2419).Run: context canceled
(cluster.go:2419).Run: context canceled
test artifacts and logs in: /artifacts/failover/chaos/read-write/lease=expiration/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test kv/contention/nodes=4 failed: VMs preempted during the test run: teamcity-16441860-1723528050-92-n5cpu4-0003 (us-east1-d)
**Other Failures:**
(cluster.go:2419).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv/contention/nodes=4/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test tpccbench/nodes=3/cpu=4/backups=true failed: VMs preempted during the test run: teamcity-16441860-1723528050-77-n4cpu4-0001 (us-east1-d), teamcity-16441860-1723528050-77-n4cpu4-0003 (us-east1-d)
**Other Failures:**
(cluster.go:2305).Start: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=4/backups=true/run_1/ssh/ssh_111141.667297285_n3_upload-start-script.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=4/backups=true/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test import/tpch/nodes=8 failed: VMs preempted during the test run: teamcity-16441860-1723528050-78-n8cpu4-0001 (us-east1-d), teamcity-16441860-1723528050-78-n8cpu4-0004 (us-east1-d), teamcity-16441860-1723528050-78-n8cpu4-0005 (us-east1-d), teamcity-16441860-1723528050-78-n8cpu4-0002 (us-east1-d)
**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/import/tpch/nodes=8/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=true
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test kv50/enc=false/nodes=4/cpu=96/batch=64 failed: VMs preempted during the test run: teamcity-16441860-1723528050-143-n5cpu96-0001 (us-east1-c)
**Other Failures:**
(cluster.go:2419).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv50/enc=false/nodes=4/cpu=96/batch=64/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test tpccbench/nodes=3/cpu=4/mt-shared-process failed: VMs preempted during the test run: teamcity-16441860-1723528050-130-n4cpu4-0001 (us-east1-d), teamcity-16441860-1723528050-130-n4cpu4-0002 (us-east1-d)
**Other Failures:**
(cluster.go:2356).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=4/mt-shared-process/run_1/ssh/ssh_133914.968716171_n2_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=4/mt-shared-process/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test tpccbench/nodes=3/cpu=16 failed: VMs preempted during the test run: teamcity-16441860-1723528050-134-n4cpu16-0001 (us-east1-c), teamcity-16441860-1723528050-134-n4cpu16-0003 (us-east1-c), teamcity-16441860-1723528050-134-n4cpu16-0002 (us-east1-c)
**Other Failures:**
(cluster.go:2305).Start: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=16/run_1/ssh/ssh_134229.194441140_n3_upload-start-script.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=16/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test schemachange/indexschemachange/index/tpcc/w=800 failed: VMs preempted during the test run: teamcity-16441860-1723528050-120-n5cpu16-0002 (us-east1-c), teamcity-16441860-1723528050-120-n5cpu16-0003 (us-east1-c), teamcity-16441860-1723528050-120-n5cpu16-0001 (us-east1-c), teamcity-16441860-1723528050-120-n5cpu16-0005 (us-east1-c)
**Other Failures:**
(monitor.go:154).Wait: monitor failure: full command output in run_120828.214366495_n5_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/schemachange/indexschemachange/index/tpcc/w=800/run_1/ssh/ssh_120828.258992935_n5_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/schemachange/indexschemachange/index/tpcc/w=800/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test tpccbench/nodes=9/cpu=4/chaos/partition failed: VMs preempted during the test run: teamcity-16441860-1723528050-133-n10cpu4-0010 (us-east1-c), teamcity-16441860-1723528050-133-n10cpu4-0004 (us-east1-c)
**Other Failures:**
(cluster.go:2305).Start: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/run_1/ssh/ssh_133901.943705535_n4_upload-start-script.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
| | github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:173
| | runtime.goexit
| | src/runtime/asm_amd64.s:1695
| Wraps: (3) _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1/ssh/ssh_074047.496221086_n8_cockroach-workload-r.log)
| Wraps: (4) TRANSIENT_ERROR(ssh_problem)
| Wraps: (5) exit status 255
| Error types: (1) *hintdetail.withDetail (2) *withstack.withStack (3) *errutil.withPrefix (4) errors.TransientError (5) *exec.ExitError
└─ Wraps: (11) secondary error attachment
| _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1/ssh/ssh_074020.741673316_n8_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
| (1) Node 8. Command with error:
| | ``````
| | ./cockroach workload run kv --db=bigwrites --read-percent=0 --min-block-bytes=32768 --max-block-bytes=32768 --max-rate=550 --tolerate-errors --concurrency=512 --duration=30m0s {pgurl:1-7}
| | ``````
| | <no output>
| Wraps: (2) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).errWithDebug
| | github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:143
| | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).Run.func1
| | github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:173
| | runtime.goexit
| | src/runtime/asm_amd64.s:1695
| Wraps: (3) _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1/ssh/ssh_074020.741673316_n8_cockroach-workload-r.log)
| Wraps: (4) TRANSIENT_ERROR(ssh_problem)
| Wraps: (5) exit status 255
| Error types: (1) *hintdetail.withDetail (2) *withstack.withStack (3) *errutil.withPrefix (4) errors.TransientError (5) *exec.ExitError
└─ Wraps: (12) Node 8. Command with error:
| ``````
| ./cockroach workload run kv --db=bigwrites --read-percent=0 --min-block-bytes=32768 --max-block-bytes=32768 --max-rate=550 --tolerate-errors --concurrency=512 --duration=30m0s {pgurl:1-7}
| ``````
| <truncated> ... 3841 workload/pgx_helpers.go:240 [-] 7247 error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
| W240814 07:23:57.177414 3897 workload/pgx_helpers.go:240 [-] 7248 error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
| W240814 07:23:57.185020 3756 workload/pgx_helpers.go:240 [-] 7249 error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
| W240814 07:23:57.188423 3920 workload/pgx_helpers.go:240 [-] 7250 error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
| Connection to 35.243.128.46 closed by remote host.
└─ Wraps: (13) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).errWithDebug
| github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:143
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).Run.func1
| github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:173
| runtime.goexit
| src/runtime/asm_amd64.s:1695
└─ Wraps: (14) _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1/ssh/ssh_072353.026023236_n8_cockroach-workload-r.log)
└─ Wraps: (15) TRANSIENT_ERROR(ssh_problem)
└─ Wraps: (16) exit status 255
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *join.joinError (5) *withstack.withStack (6) *errutil.withPrefix (7) *markers.withMark (8) *withstack.withStack (9) *errutil.withPrefix (10) *secondary.withSecondaryError (11) *secondary.withSecondaryError (12) *hintdetail.withDetail (13) *withstack.withStack (14) *errutil.withPrefix (15) errors.TransientError (16) *exec.ExitError
Test: allocbench/nodes=7/cpu=8/kv/r=50/ops=skew
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
| | github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:173
| | runtime.goexit
| | src/runtime/asm_amd64.s:1695
| Wraps: (3) _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/ops=skew/run_1/ssh/ssh_074053.573752989_n8_cockroach-workload-r.log)
| Wraps: (4) TRANSIENT_ERROR(ssh_problem)
| Wraps: (5) exit status 255
| Error types: (1) *hintdetail.withDetail (2) *withstack.withStack (3) *errutil.withPrefix (4) errors.TransientError (5) *exec.ExitError
└─ Wraps: (11) secondary error attachment
| _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/ops=skew/run_1/ssh/ssh_074021.983742533_n8_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
| (1) Node 8. Command with error:
| | ``````
| | ./cockroach workload run kv --db=bigreads --read-percent=75 --min-block-bytes=1024 --max-block-bytes=1024 --max-rate=850 --span-percent=20 --span-limit=10000 --batch=10 --tolerate-errors --concurrency=512 --duration=30m0s {pgurl:1-7}
| | ``````
| | <no output>
| Wraps: (2) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).errWithDebug
| | github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:143
| | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).Run.func1
| | github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:173
| | runtime.goexit
| | src/runtime/asm_amd64.s:1695
| Wraps: (3) _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/ops=skew/run_1/ssh/ssh_074021.983742533_n8_cockroach-workload-r.log)
| Wraps: (4) TRANSIENT_ERROR(ssh_problem)
| Wraps: (5) exit status 255
| Error types: (1) *hintdetail.withDetail (2) *withstack.withStack (3) *errutil.withPrefix (4) errors.TransientError (5) *exec.ExitError
└─ Wraps: (12) Node 8. Command with error:
| ``````
| ./cockroach workload run kv --db=bigreads --read-percent=75 --min-block-bytes=1024 --max-block-bytes=1024 --max-rate=850 --span-percent=20 --span-limit=10000 --batch=10 --tolerate-errors --concurrency=512 --duration=30m0s {pgurl:1-7}
| ``````
| <truncated> ... ) WHERE k IN ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
| W240814 07:23:03.349910 4142 workload/pgx_helpers.go:240 [-] 4118 error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
| W240814 07:23:03.351205 4047 workload/pgx_helpers.go:240 [-] 4119 error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
| W240814 07:23:03.351274 4157 workload/pgx_helpers.go:240 [-] 4120 error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
| Connection to 35.231.119.12 closed by remote host.
└─ Wraps: (13) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).errWithDebug
| github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:143
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).Run.func1
| github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:173
| runtime.goexit
| src/runtime/asm_amd64.s:1695
└─ Wraps: (14) _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/ops=skew/run_1/ssh/ssh_072259.238652813_n8_cockroach-workload-r.log)
└─ Wraps: (15) TRANSIENT_ERROR(ssh_problem)
└─ Wraps: (16) exit status 255
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *join.joinError (5) *withstack.withStack (6) *errutil.withPrefix (7) *markers.withMark (8) *withstack.withStack (9) *errutil.withPrefix (10) *secondary.withSecondaryError (11) *secondary.withSecondaryError (12) *hintdetail.withDetail (13) *withstack.withStack (14) *errutil.withPrefix (15) errors.TransientError (16) *exec.ExitError
Test: allocbench/nodes=7/cpu=8/kv/r=95/ops=skew
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/ops=skew/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test allocbench/nodes=7/cpu=8/kv/r=95/access=skew failed: VMs preempted during the test run: teamcity-16460023-1723619632-09-n8cpu8-0004 (us-east1-c), teamcity-16460023-1723619632-09-n8cpu8-0006 (us-east1-c)
**Other Failures:**
(cluster.go:2411).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/access=skew/run_1/ssh/ssh_075214.212950743_n4_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/access=skew/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test kv95/enc=true/nodes=1 failed: VMs preempted during the test run: teamcity-16460023-1723619632-06-n2cpu8-0001 (us-east1-c)
**Other Failures:**
(cluster.go:2419).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv95/enc=true/nodes=1/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test admission-control/snapshot-overload-excise failed: VMs preempted during the test run: teamcity-16460023-1723619632-07-n4cpu4-0003 (us-east1-c), teamcity-16460023-1723619632-07-n4cpu4-0001 (us-east1-c)
**Other Failures:**
(cluster.go:2356).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/admission-control/snapshot-overload-excise/run_1/ssh/ssh_094411.331571239_n3_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cluster.go:2419).Run: context canceled
test artifacts and logs in: /artifacts/admission-control/snapshot-overload-excise/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test cdc/crdb-chaos failed: VMs preempted during the test run: teamcity-16460023-1723619632-22-n4cpu16-0003 (us-east1-b), teamcity-16460023-1723619632-22-n4cpu16-0001 (us-east1-b), teamcity-16460023-1723619632-22-n4cpu16-0002 (us-east1-b)
**Other Failures:**
(test_runner.go:1277).runTest: test timed out (3h0m0s)
test artifacts and logs in: /artifacts/cdc/crdb-chaos/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test kv95/enc=true/nodes=3 failed: VMs preempted during the test run: teamcity-16460023-1723619632-115-n4cpu8-0001 (us-east1-d)
**Other Failures:**
(cluster.go:2419).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv95/enc=true/nodes=3/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=true
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test ycsb/A/nodes=3/cpu=32 failed: VMs preempted during the test run: teamcity-16460023-1723619632-97-n4cpu32-0003 (us-east1-c), teamcity-16460023-1723619632-97-n4cpu32-0001 (us-east1-c), teamcity-16460023-1723619632-97-n4cpu32-0002 (us-east1-c)
**Other Failures:**
(cluster.go:2419).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/ycsb/A/nodes=3/cpu=32/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_encrypted=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test kv95/enc=false/nodes=3/cpu=96 failed: VMs preempted during the test run: teamcity-16460023-1723619632-118-n4cpu96-0001 (us-east1-d)
**Other Failures:**
(cluster.go:2419).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/cpu=96/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test sysbench/oltp_delete/nodes=3/cpu=32/conc=256 failed: VMs preempted during the test run: teamcity-16460023-1723619632-142-n4cpu32-0002 (us-east1-b)
**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/sysbench/oltp_delete/nodes=3/cpu=32/conc=256/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test ycsb/D/nodes=3/cpu=32 failed: VMs preempted during the test run: teamcity-16460023-1723619632-107-n4cpu32-0003 (us-east1-b), teamcity-16460023-1723619632-107-n4cpu32-0001 (us-east1-b), teamcity-16460023-1723619632-107-n4cpu32-0002 (us-east1-b)
**Other Failures:**
(cluster.go:2419).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/ycsb/D/nodes=3/cpu=32/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_encrypted=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test scrub/index-only/tpcc/w=100 failed: VMs preempted during the test run: teamcity-16460023-1723619632-140-n5cpu4-0001 (us-east1-b), teamcity-16460023-1723619632-140-n5cpu4-0004 (us-east1-b)
**Other Failures:**
(scrub.go:91).1: read tcp 172.17.0.3:60920 -> 35.231.103.98:26257: read: connection reset by peer
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/scrub/index-only/tpcc/w=100/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
| | github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:173
| | runtime.goexit
| | src/runtime/asm_amd64.s:1695
| Wraps: (3) _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1/ssh/ssh_062020.539604560_n8_cockroach-workload-r.log)
| Wraps: (4) TRANSIENT_ERROR(ssh_problem)
| Wraps: (5) exit status 255
| Error types: (1) *hintdetail.withDetail (2) *withstack.withStack (3) *errutil.withPrefix (4) errors.TransientError (5) *exec.ExitError
└─ Wraps: (11) secondary error attachment
| _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1/ssh/ssh_061953.779101768_n8_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
| (1) Node 8. Command with error:
| | ``````
| | ./cockroach workload run kv --db=smallreads --read-percent=95 --min-block-bytes=1 --max-block-bytes=1 --max-rate=16500 --tolerate-errors --concurrency=512 --duration=30m0s {pgurl:1-7}
| | ``````
| | <no output>
| Wraps: (2) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).errWithDebug
| | github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:143
| | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).Run.func1
| | github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:173
| | runtime.goexit
| | src/runtime/asm_amd64.s:1695
| Wraps: (3) _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1/ssh/ssh_061953.779101768_n8_cockroach-workload-r.log)
| Wraps: (4) TRANSIENT_ERROR(ssh_problem)
| Wraps: (5) exit status 255
| Error types: (1) *hintdetail.withDetail (2) *withstack.withStack (3) *errutil.withPrefix (4) errors.TransientError (5) *exec.ExitError
└─ Wraps: (12) Node 8. Command with error:
| ``````
| ./cockroach workload run kv --db=smallreads --read-percent=95 --min-block-bytes=1 --max-block-bytes=1 --max-rate=16500 --tolerate-errors --concurrency=512 --duration=30m0s {pgurl:1-7}
| ``````
| <truncated> ... 0 3753 workload/pgx_helpers.go:240 [-] 4719 error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
| W240815 05:54:41.339922 4017 workload/pgx_helpers.go:240 [-] 4720 error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
| W240815 05:54:41.344021 3936 workload/pgx_helpers.go:240 [-] 4721 error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
| W240815 05:54:41.321835 3846 workload/pgx_helpers.go:240 [-] 4696 error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
| Connection to 34.75.248.85 closed by remote host.
└─ Wraps: (13) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).errWithDebug
| github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:143
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).Run.func1
| github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:173
| runtime.goexit
| src/runtime/asm_amd64.s:1695
└─ Wraps: (14) _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1/ssh/ssh_055437.220474913_n8_cockroach-workload-r.log)
└─ Wraps: (15) TRANSIENT_ERROR(ssh_problem)
└─ Wraps: (16) exit status 255
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *join.joinError (5) *withstack.withStack (6) *errutil.withPrefix (7) *markers.withMark (8) *withstack.withStack (9) *errutil.withPrefix (10) *secondary.withSecondaryError (11) *secondary.withSecondaryError (12) *hintdetail.withDetail (13) *withstack.withStack (14) *errutil.withPrefix (15) errors.TransientError (16) *exec.ExitError
Test: allocbench/nodes=7/cpu=8/kv/r=50/ops=skew
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:
test allocbench/nodes=7/cpu=8/kv/r=0/ops=skew failed: VMs preempted during the test run: teamcity-16475370-1723700895-10-n8cpu8-0005 (us-east1-b), teamcity-16475370-1723700895-10-n8cpu8-0004 (us-east1-b)
**Other Failures:**
(cluster.go:2411).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/ops=skew/run_1/ssh/ssh_062719.102413926_n4_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/ops=skew/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help
See: roachtest README
See: How To Investigate (internal)
See: Grafana
Same failure on other branches
- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
/cc @cockroachdb/test-eng
This test on roachdash | Improve this report!
Jira issue: CRDB-40816