search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.97k stars 3.79k forks source link

roachtest: vm_preemption failed #118528

Closed cockroach-teamcity closed 1 week ago

cockroach-teamcity commented 8 months ago

roachtest.vm_preemption failed with artifacts on master @ ed3a25e3c9459cede2f80babbfc9d44a836b6c12:

test allocbench/nodes=7/cpu=8/kv/r=50/ops=skew failed due to VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13762710-1706682693-23-n8cpu8-0002

**Other Failure**
(cluster.go:2285).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-35774

cockroach-teamcity commented 8 months ago

roachtest.vm_preemption failed with artifacts on master @ ed3a25e3c9459cede2f80babbfc9d44a836b6c12:

test allocbench/nodes=7/cpu=8/kv/r=95/access=skew failed due to VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13762710-1706682693-24-n8cpu8-0004

**Other Failure**
(cluster.go:2285).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/access=skew/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.vm_preemption failed with artifacts on master @ ed3a25e3c9459cede2f80babbfc9d44a836b6c12:

test decommissionBench/nodes=6/warehouses=1000/drain-first/while-upreplicating/target=3/multi-region failed due to VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-west1-b/instances/teamcity-13762710-1706682693-81-n8cpu16-geo-0008, projects/cockroach-ephemeral/zones/us-west1-b/instances/teamcity-13762710-1706682693-81-n8cpu16-geo-0003

**Other Failure**
(decommissionbench.go:725).runDecommissionBench: monitor failure: full command output in run_152050.747873505_n8_workload-run-tpcc-wa.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/decommissionBench/nodes=6/warehouses=1000/drain-first/while-upreplicating/target=3/multi-region/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.vm_preemption failed with artifacts on master @ ed3a25e3c9459cede2f80babbfc9d44a836b6c12:

test ledger/nodes=6/multi-az failed due to VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-central1-f/instances/teamcity-13762710-1706682693-145-n7cpu16-geo-0001, projects/cockroach-ephemeral/zones/us-central1-f/instances/teamcity-13762710-1706682693-145-n7cpu16-geo-0007

**Other Failure**
(cluster.go:2293).Run: full command output in run_195903.908024278_n7_workload-run-ledger-.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ledger/nodes=6/multi-az/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.vm_preemption failed with artifacts on master @ ed3a25e3c9459cede2f80babbfc9d44a836b6c12:

test tpccbench/nodes=6/cpu=16/multi-az failed due to VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-central1-f/instances/teamcity-13762710-1706682693-172-n7cpu16-geo-0002

**Other Failure**
(cluster.go:2230).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=6/cpu=16/multi-az/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.vm_preemption failed with artifacts on master @ cc4fdffa8532d16544c48ef036689763f737dc6b:

test cdc/workload/kv100/nodes=5/cpu=16/ranges=100/control failed due to VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13776729-1706768946-62-n7cpu16-0006, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13776729-1706768946-62-n7cpu16-0001

**Other Failure**
(monitor.go:153).Wait: monitor failure: full command output in run_130124.799005526_n7_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100/control/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.vm_preemption failed with artifacts on master @ cc4fdffa8532d16544c48ef036689763f737dc6b:

test kv0/enc=false/nodes=3/cpu=96 failed due to VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13776729-1706768946-134-n4cpu96-0001, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13776729-1706768946-134-n4cpu96-0003, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13776729-1706768946-134-n4cpu96-0002, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13776729-1706768946-134-n4cpu96-0004

**Other Failure**
(cluster.go:2293).Run: context canceled
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/cpu=96/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.vm_preemption failed with artifacts on master @ fce4d4723519bc4ca6e9ef5da0ae19960c84752c:

test allocbench/nodes=7/cpu=8/kv/r=0/ops=skew failed due to VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13797902-1706855402-21-n8cpu8-0002

**Other Failure**
(cluster.go:2282).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/ops=skew/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.vm_preemption failed with artifacts on master @ 15961a19faca0e2b66df2d01a547549523ca70c7:

test tpccbench/nodes=12/cpu=16 failed due to VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-central1-b/instances/teamcity-13818437-1706941773-34-n13cpu16-0008

**Other Failure**
(cluster.go:2227).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=12/cpu=16/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.vm_preemption failed with artifacts on master @ 15961a19faca0e2b66df2d01a547549523ca70c7:

test tpccbench/nodes=12/cpu=16/lease=expiration failed due to VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-central1-b/instances/teamcity-13818437-1706941773-36-n13cpu16-0010

**Other Failure**
(cluster.go:2227).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=12/cpu=16/lease=expiration/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.vm_preemption failed with artifacts on master @ 3c41c509a87cba7a1fd3f5cfdb0f6badb78e3704:

test allocbench/nodes=7/cpu=8/kv/r=95/ops=skew failed due to VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13830521-1707028197-24-n8cpu8-0007

**Other Failure**
(cluster.go:2282).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/ops=skew/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.vm_preemption failed with artifacts on master @ 3c41c509a87cba7a1fd3f5cfdb0f6badb78e3704:

test kv95/enc=false/nodes=3/cpu=96 failed due to VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13830521-1707028197-167-n4cpu96-0004

**Other Failure**
(cluster.go:2290).Run: full command output in run_205754.519345260_n4_workload-run-kv-tole.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/cpu=96/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.vm_preemption failed with artifacts on master @ d272e9ef5589deff570efc023db6c70edfde311c:

test ycsb/C/nodes=3/cpu=32 failed due to VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13840930-1707114553-174-n4cpu32-0003

**Other Failure**
(cluster.go:2290).Run: full command output in run_213644.362105068_n4_workload-run-ycsb-in.log: COMMAND_PROBLEM: exit status 1
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/C/nodes=3/cpu=32/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.vm_preemption failed with artifacts on master @ d272e9ef5589deff570efc023db6c70edfde311c:

test ycsb/C/nodes=3/cpu=32/isolation-level=read-committed failed due to VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13840930-1707114553-175-n4cpu32-0001, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13840930-1707114553-175-n4cpu32-0004

**Other Failure**
(cluster.go:2290).Run: full command output in run_214248.301105115_n4_workload-run-ycsb-in.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/C/nodes=3/cpu=32/isolation-level=read-committed/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.vm_preemption failed with artifacts on master @ 715628abd134abfd2c0d966f9b7220a6715cc299:

test kv0/enc=false/nodes=1/cpu=32/mt-shared-process failed due to VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13860887-1707200979-132-n2cpu32-0001

**Other Failure**
(cluster.go:2176).Start: failed to find services to register: failed to find 1 open ports: expected 1 ports, got 0
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=1/cpu=32/mt-shared-process/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.vm_preemption failed with artifacts on master @ 715628abd134abfd2c0d966f9b7220a6715cc299:

test tpcc/headroom/isolation-level=read-committed/n4cpu16 failed due to VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13860887-1707200979-170-n4cpu16-0001

**Other Failure**
(cluster.go:2290).Run: full command output in run_223020.944432213_n4_cockroach-workload-c.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/tpcc/headroom/isolation-level=read-committed/n4cpu16/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.vm_preemption failed with artifacts on master @ d7d442e4a3c9dca7e01c4c6f4f00e2f28faa4374:

test cdc/workload/kv0/nodes=5/cpu=16/ranges=100/control failed due to VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13880717-1707287383-52-n7cpu16-0002

**Other Failure**
(monitor.go:153).Wait: monitor failure: full command output in run_124755.911215239_n7_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100/control/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.vm_preemption failed with artifacts on master @ d7d442e4a3c9dca7e01c4c6f4f00e2f28faa4374:

test cdc/scan/catchup/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null failed due to VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13880717-1707287383-49-n6cpu16-0003, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13880717-1707287383-49-n6cpu16-0006, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13880717-1707287383-49-n6cpu16-0005

**Other Failure**
(monitor.go:153).Wait: monitor failure: failed 5 attempts to get changefeed info: dial tcp 35.196.11.77:29000: connect: connection timed out
test artifacts and logs in: /artifacts/cdc/scan/catchup/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.vm_preemption failed with artifacts on master @ d7d442e4a3c9dca7e01c4c6f4f00e2f28faa4374:

test cdc/workload/kv0/nodes=5/cpu=16/ranges=100/server=scheduler/protocol=mux/format=json/sink=null failed due to VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13880717-1707287383-54-n7cpu16-0001, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13880717-1707287383-54-n7cpu16-0006, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13880717-1707287383-54-n7cpu16-0007, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13880717-1707287383-54-n7cpu16-0003, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13880717-1707287383-54-n7cpu16-0005

**Other Failure**
(monitor.go:153).Wait: monitor failure: full command output in run_125050.761140141_n7_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100/server=scheduler/protocol=mux/format=json/sink=null/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.vm_preemption failed with artifacts on master @ d7d442e4a3c9dca7e01c4c6f4f00e2f28faa4374:

test failover/non-system/disk-stall/lease=expiration failed due to VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13880717-1707287383-98-n7cpu2-0001, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13880717-1707287383-98-n7cpu2-0005, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13880717-1707287383-98-n7cpu2-0006, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13880717-1707287383-98-n7cpu2-0004, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13880717-1707287383-98-n7cpu2-0002, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13880717-1707287383-98-n7cpu2-0003, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13880717-1707287383-98-n7cpu2-0007

**Other Failure**
(failover.go:1774).sleepFor: sleep failed: context canceled
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2290).Run: context canceled
test artifacts and logs in: /artifacts/failover/non-system/disk-stall/lease=expiration/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.vm_preemption failed with artifacts on master @ d7d442e4a3c9dca7e01c4c6f4f00e2f28faa4374:

test ycsb/C/nodes=3/cpu=32 failed due to VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13880717-1707287383-168-n4cpu32-0002

**Other Failure**
(cluster.go:2290).Run: full command output in run_213213.826489550_n4_workload-run-ycsb-in.log: COMMAND_PROBLEM: exit status 1
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/C/nodes=3/cpu=32/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.vm_preemption failed with artifacts on master @ 7042601857042a057b1d4676735576cfbd37f36a:

test allocbench/nodes=7/cpu=8/kv/r=95/access=skew failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13900727-1707373779-23-n8cpu8-0005

**Other Failure**
(cluster.go:2282).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/access=skew/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.vm_preemption failed with artifacts on master @ 7042601857042a057b1d4676735576cfbd37f36a:

test c2c/initialscan/kv0 failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13900727-1707373779-34-n9cpu8-0003

**Other Failure**
(soon.go:64).SucceedsWithin: condition failed to evaluate within 30m0s: from cluster_to_cluster.go:685: no replicated time
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/initialscan/kv0/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.vm_preemption failed with artifacts on master @ 7042601857042a057b1d4676735576cfbd37f36a:

test tpcc/headroom/n4cpu16 failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13900727-1707373779-147-n4cpu16-0003, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13900727-1707373779-147-n4cpu16-0004

**Other Failure**
(monitor.go:153).Wait: monitor failure: full command output in run_212224.379801524_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/tpcc/headroom/n4cpu16/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.vm_preemption failed with artifacts on master @ 7042601857042a057b1d4676735576cfbd37f36a:

test tpccbench/nodes=6/cpu=16/multi-az failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-central1-f/instances/teamcity-13900727-1707373779-150-n7cpu16-geo-0001

**Other Failure**
(monitor.go:153).Wait: monitor failure: full command output in run_211849.850334565_n1_cockroach-workload-f.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=6/cpu=16/multi-az/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.vm_preemption failed with artifacts on master @ 353fded9fe270b3eee4c85480ac1b9ec819f23b0:

test kv50/enc=false/nodes=4/cpu=96/batch=64 failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13919397-1707460148-174-n5cpu96-0001, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13919397-1707460148-174-n5cpu96-0005, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13919397-1707460148-174-n5cpu96-0002

**Other Failure**
(cluster.go:2290).Run: full command output in run_233354.885924130_n5_workload-run-kv-tole.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv50/enc=false/nodes=4/cpu=96/batch=64/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.vm_preemption failed with artifacts on master @ b2e31876366324c2ebe5c2ad8bbd644997e90864:

test allocbench/nodes=7/cpu=8/kv/r=95/ops=skew failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13935478-1707546591-24-n8cpu8-0002, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13935478-1707546591-24-n8cpu8-0004

**Other Failure**
(cluster.go:2290).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/ops=skew/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.vm_preemption failed with artifacts on master @ b2e31876366324c2ebe5c2ad8bbd644997e90864:

test decommissionBench/nodes=4/warehouses=1000/duration=1h0m0s failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13958022-1707719383-75-n5cpu16-0003

**Other Failure**
(decommissionbench.go:847).runDecommissionBenchLong: monitor failure: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/decommissionBench/nodes=4/warehouses=1000/duration=1h0m0s/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.vm_preemption failed with artifacts on master @ b2e31876366324c2ebe5c2ad8bbd644997e90864:

test weekly/tpcc/headroom failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-central1-b/instances/teamcity-13935561-1707546617-32-n4cpu16-0001

**Other Failure**
(monitor.go:153).Wait: monitor failure: full command output in run_120624.130468120_n4_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/weekly/tpcc/headroom/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.vm_preemption failed with artifacts on master @ b2e31876366324c2ebe5c2ad8bbd644997e90864:

test restore/tpce/400GB/gce/nodes=4/cpus=8/lowmem failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13958022-1707719383-156-n4cpu8lm-0002, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13958022-1707719383-156-n4cpu8lm-0001

**Other Failure**
(monitor.go:153).Wait: monitor failure: pq: job 942720023536173057: node liveness error: restarting in background
test artifacts and logs in: /artifacts/restore/tpce/400GB/gce/nodes=4/cpus=8/lowmem/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.vm_preemption failed with artifacts on master @ 814a375d4c0e79d875c42452725f05f6c27294e3:

test failover/chaos/read-write failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13975278-1707805779-82-n10cpu2-0004

**Other Failure**
(cluster.go:2235).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
test artifacts and logs in: /artifacts/failover/chaos/read-write/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.vm_preemption failed with artifacts on master @ 814a375d4c0e79d875c42452725f05f6c27294e3:

test tpccbench/nodes=6/cpu=16/multi-az failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-central1-f/instances/teamcity-13975278-1707805779-169-n7cpu16-geo-0002, projects/cockroach-ephemeral/zones/us-central1-f/instances/teamcity-13975278-1707805779-169-n7cpu16-geo-0001

**Other Failure**
(cluster.go:2235).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=6/cpu=16/multi-az/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.vm_preemption failed with artifacts on master @ 254dbd247fb8ed352a11439063b29f23a0767f28:

test allocbench/nodes=7/cpu=8/kv/r=0/ops=skew failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13990670-1707892139-21-n8cpu8-0006

**Other Failure**
(cluster.go:2290).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/ops=skew/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.vm_preemption failed with artifacts on master @ 254dbd247fb8ed352a11439063b29f23a0767f28:

test decommissionBench/nodes=8/warehouses=3000 failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13990670-1707892139-93-n9cpu16-0003, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13990670-1707892139-93-n9cpu16-0009

**Other Failure**
(decommissionbench.go:721).runDecommissionBench: monitor failure: full command output in run_154606.007181434_n9_workload-run-tpcc-wa.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/decommissionBench/nodes=8/warehouses=3000/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.vm_preemption failed with artifacts on master @ 254dbd247fb8ed352a11439063b29f23a0767f28:

test kv95limited-spans/enc=false/nodes=1/cpu=32/seq/no-load-splitting failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13990670-1707892139-145-n2cpu32-0002, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13990670-1707892139-145-n2cpu32-0001

**Other Failure**
(cluster.go:2298).Run: context canceled
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95limited-spans/enc=false/nodes=1/cpu=32/seq/no-load-splitting/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.vm_preemption failed with artifacts on master @ 254dbd247fb8ed352a11439063b29f23a0767f28:

test ycsb/C/nodes=3/cpu=32/mvcc-range-keys=global failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13990670-1707892139-176-n4cpu32-0001

**Other Failure**
(cluster.go:2298).Run: full command output in run_204942.305479801_n4_workload-run-ycsb-in.log: COMMAND_PROBLEM: exit status 1
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/C/nodes=3/cpu=32/mvcc-range-keys=global/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.vm_preemption failed with artifacts on master @ cc6ca026319024800395293b0fb18f05dd8eb50e:

test cdc/initial-scan-only/parquet failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14006082-1707978563-43-n4cpu16-0002

**Other Failure**
(cdc.go:378).Close: error shutting down prometheus/grafana: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/cdc/initial-scan-only/parquet/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.vm_preemption failed with artifacts on master @ cc6ca026319024800395293b0fb18f05dd8eb50e:

test failover/chaos/read-write failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14006082-1707978563-88-n10cpu2-0006, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14006082-1707978563-88-n10cpu2-0004

**Other Failure**
(cluster.go:2298).Run: full command output in run_141639.792052953_n6_sudo-dmsetup-resume-.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
test artifacts and logs in: /artifacts/failover/chaos/read-write/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.vm_preemption failed with artifacts on master @ cc6ca026319024800395293b0fb18f05dd8eb50e:

test failover/chaos/read-write/lease=expiration failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14006082-1707978563-90-n10cpu2-0003

**Other Failure**
(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/failover.go:1592
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/failover.go:319
                                main/pkg/cmd/roachtest/monitor.go:119
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                                src/runtime/asm_amd64.s:1650
    Error:          Received unexpected error:
                    dial tcp 34.138.114.150:29000: connect: connection timed out
    Test:           failover/chaos/read-write/lease=expiration
(require.go:1360).NoError: FailNow called
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
test artifacts and logs in: /artifacts/failover/chaos/read-write/lease=expiration/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.vm_preemption failed with artifacts on master @ cc6ca026319024800395293b0fb18f05dd8eb50e:

test failover/non-system/disk-stall/lease=expiration failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14006082-1707978563-99-n7cpu2-0004

**Other Failure**
(cluster.go:2298).Run: full command output in run_145938.034518148_n4_sudo-dmsetup-resume-.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2298).Run: context canceled
test artifacts and logs in: /artifacts/failover/non-system/disk-stall/lease=expiration/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.vm_preemption failed with artifacts on master @ cc6ca026319024800395293b0fb18f05dd8eb50e:

test kv95/enc=true/nodes=3 failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14006082-1707978563-142-n4cpu8-0002, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14006082-1707978563-142-n4cpu8-0004

**Other Failure**
(cluster.go:2298).Run: full command output in run_182149.842212236_n4_workload-run-kv-tole.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=true/nodes=3/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.vm_preemption failed with artifacts on master @ 7d0697b632066ee78735fc57e8150222d5576d0d:

test allocbench/nodes=7/cpu=8/kv/r=0/access=skew failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14023201-1708064961-19-n8cpu8-0007

**Other Failure**
(cluster.go:2290).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/access=skew/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.vm_preemption failed with artifacts on master @ 7d0697b632066ee78735fc57e8150222d5576d0d:

test indexes/2/nodes=6/multi-region failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14023201-1708064961-122-n7cpu16-geo-0001

**Other Failure**
(cluster.go:2298).Run: full command output in run_162624.742352979_n7_workload-run-indexes.log: COMMAND_PROBLEM: exit status 1
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/indexes/2/nodes=6/multi-region/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.vm_preemption failed with artifacts on master @ 7d0697b632066ee78735fc57e8150222d5576d0d:

test ycsb/E/nodes=3/cpu=32 failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14023201-1708064961-176-n4cpu32-0002

**Other Failure**
(cluster.go:2298).Run: full command output in run_211346.932363687_n4_workload-run-ycsb-in.log: COMMAND_PROBLEM: exit status 1
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/E/nodes=3/cpu=32/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.vm_preemption failed with artifacts on master @ 0b7ae19e2b94b851ed8812914f57032aab699811:

test tpccbench/nodes=6/cpu=16/multi-az failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-central1-f/instances/teamcity-14039084-1708151422-168-n7cpu16-geo-0001

**Other Failure**
(monitor.go:153).Wait: monitor failure: dial tcp 34.171.42.106:29000: connect: connection timed out
test artifacts and logs in: /artifacts/tpccbench/nodes=6/cpu=16/multi-az/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.vm_preemption failed with artifacts on master @ e39dafe6d8c153301ff43ed2b3ed3e13af9ec72a:

test allocbench/nodes=7/cpu=8/kv/r=95/ops=skew failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14059812-1708324110-24-n8cpu8-0008

**Other Failure**
(cluster.go:2184).Start: one or more parallel execution failure(s): error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/ops=skew/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.vm_preemption failed with artifacts on master @ e39dafe6d8c153301ff43ed2b3ed3e13af9ec72a:

test allocbench/nodes=7/cpu=8/kv/r=95/access=skew failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14059812-1708324110-23-n8cpu8-0007

**Other Failure**
(cluster.go:2290).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/access=skew/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.vm_preemption failed with artifacts on master @ a36097be277adef635f55d317579ca79b450bfef:

test cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14071859-1708410544-64-n7cpu16-0006

**Other Failure**
(monitor.go:153).Wait: monitor failure: failed 5 attempts to get changefeed info: dial tcp 34.148.61.18:29000: connect: connection timed out
test artifacts and logs in: /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.vm_preemption failed with artifacts on master @ a36097be277adef635f55d317579ca79b450bfef:

test failover/chaos/read-only/lease=expiration failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14071859-1708410544-79-n10cpu2-0003

**Other Failure**
(cluster.go:2235).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
test artifacts and logs in: /artifacts/failover/chaos/read-only/lease=expiration/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.vm_preemption failed with artifacts on master @ a36097be277adef635f55d317579ca79b450bfef:

test failover/chaos/read-only failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14071859-1708410544-77-n10cpu2-0007

**Other Failure**
(cluster.go:2298).Run: full command output in run_135532.940833528_n7_sudo-iptables-A-OUTP.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
test artifacts and logs in: /artifacts/failover/chaos/read-only/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.vm_preemption failed with artifacts on master @ a36097be277adef635f55d317579ca79b450bfef:

test decommissionBench/nodes=8/warehouses=3000 failed: VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14071859-1708410544-88-n9cpu16-0004, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14071859-1708410544-88-n9cpu16-0005, projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14071859-1708410544-88-n9cpu16-0009

**Other Failure**
(cluster.go:2298).Run: full command output in run_145443.225051170_n1_cockroach-workload-f.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/decommissionBench/nodes=8/warehouses=3000/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!