search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.96k stars 3.79k forks source link

roachtest: vm_preemption failed #125727

Open cockroach-teamcity opened 3 months ago

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 35c3e0c133c56c91399e730f49bc488cf6dd4c11:

test allocbench/nodes=7/cpu=8/kv/r=95/ops=skew failed: VMs preempted during the test run: teamcity-15671088-1718430600-08-n8cpu8-0003 (us-east1-c), teamcity-15671088-1718430600-08-n8cpu8-0006 (us-east1-c), teamcity-15671088-1718430600-08-n8cpu8-0008 (us-east1-c), teamcity-15671088-1718430600-08-n8cpu8-0001 (us-east1-c)

**Other Failures:**
(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:396
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:344
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:269
                                main/pkg/cmd/roachtest/test_runner.go:1213
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    monitor failure: EOF [owner=test-eng]
                    (1) attached stack trace
                      -- stack trace:
                      | main.(*monitorImpl).WaitE
                      |     main/pkg/cmd/roachtest/monitor.go:142
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBenchSample
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:395
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBench
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:344
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAllocationBenchSpec.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:269
                      | main.(*testRunner).runTest.func2
                      |     main/pkg/cmd/roachtest/test_runner.go:1213
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) monitor failure
                    Wraps: (3) secondary error attachment
                      | context canceled
                      | (1) context canceled
                      | Error types: (1) *errors.errorString
                    Wraps: (4) EOF [owner=test-eng]
                    Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *secondary.withSecondaryError (4) registry.ErrorWithOwnership
    Test:           allocbench/nodes=7/cpu=8/kv/r=95/ops=skew
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/ops=skew/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-39614

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 35c3e0c133c56c91399e730f49bc488cf6dd4c11:

test cdc/pubsub-sink failed: VMs preempted during the test run: teamcity-15671088-1718430600-30-n4cpu16-0003 (us-east1-d), teamcity-15671088-1718430600-30-n4cpu16-0004 (us-east1-d), teamcity-15671088-1718430600-30-n4cpu16-0001 (us-east1-d), teamcity-15671088-1718430600-30-n4cpu16-0002 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: full command output in run_075637.009992036_n4_workload-run-tpcc-wa.log: COMMAND_PROBLEM: exit status 1
(cdc.go:388).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/pubsub-sink/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 35c3e0c133c56c91399e730f49bc488cf6dd4c11:

test cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/control failed: VMs preempted during the test run: teamcity-15671088-1718430600-37-n7cpu16-0001 (us-east1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/control/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 35c3e0c133c56c91399e730f49bc488cf6dd4c11:

test kv0/enc=false/nodes=3/cpu=32/mt-shared-process failed: VMs preempted during the test run: teamcity-15671088-1718430600-82-n4cpu32-0003 (us-east1-c), teamcity-15671088-1718430600-82-n4cpu32-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2184).StartServiceForVirtualCluster: one or more parallel execution failure(s): error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv0/enc=false/nodes=3/cpu=32/mt-shared-process/run_1/ssh/ssh_120249.820696131_n1_create-tenant-cert-bundle.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/cpu=32/mt-shared-process/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 35c3e0c133c56c91399e730f49bc488cf6dd4c11:

test kv95/enc=false/nodes=3/cpu=32/size=64kb failed: VMs preempted during the test run: teamcity-15671088-1718430600-86-n4cpu32-0004 (us-east1-c), teamcity-15671088-1718430600-86-n4cpu32-0002 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/cpu=32/size=64kb/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 35c3e0c133c56c91399e730f49bc488cf6dd4c11:

test cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-15682008-1718516956-31-n7cpu16-0004 (us-east1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: failed to list cockroach processes: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null/run_1/ssh/ssh_081354.192298902_n4_list-processes.log): TRANSIENT_ERROR(ssh_problem): exit status 255 [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 35c3e0c133c56c91399e730f49bc488cf6dd4c11:

test cdc/tpcc-1000/sink=cloudstorage failed: VMs preempted during the test run: teamcity-15682008-1718516956-32-n4cpu16-0003 (us-east1-b), teamcity-15682008-1718516956-32-n4cpu16-0002 (us-east1-b), teamcity-15682008-1718516956-32-n4cpu16-0004 (us-east1-b)

**Other Failures:**
(cluster.go:2398).Run: full command output in run_075951.961768582_n4_workload-run-tpcc-wa.log: COMMAND_PROBLEM: exit status 1
(cdc.go:388).Close: error shutting down prometheus/grafana: get /tmp/prometheus/prometheus-snapshot.tar.gz failed
test artifacts and logs in: /artifacts/cdc/tpcc-1000/sink=cloudstorage/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 35c3e0c133c56c91399e730f49bc488cf6dd4c11:

test kv0/enc=false/nodes=3/size=64kb failed: VMs preempted during the test run: teamcity-15682008-1718516956-85-n4cpu8-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/size=64kb/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 35c3e0c133c56c91399e730f49bc488cf6dd4c11:

test kv95/enc=false/nodes=3/size=4kb failed: VMs preempted during the test run: teamcity-15682008-1718516956-90-n4cpu8-0002 (us-east1-c), teamcity-15682008-1718516956-90-n4cpu8-0001 (us-east1-c), teamcity-15682008-1718516956-90-n4cpu8-0004 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/size=4kb/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 35c3e0c133c56c91399e730f49bc488cf6dd4c11:

test kv95/enc=false/nodes=3/tracing failed: VMs preempted during the test run: teamcity-15693235-1718603442-88-n4cpu8-0003 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/tracing/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 35c3e0c133c56c91399e730f49bc488cf6dd4c11:

test tpcc/headroom/n4cpu16 failed: VMs preempted during the test run: teamcity-15693235-1718603442-102-n4cpu16-0004 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: full command output in run_114227.270104496_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpcc/headroom/n4cpu16/run_1/ssh/ssh_114227.310853615_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpcc/headroom/n4cpu16/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 35c3e0c133c56c91399e730f49bc488cf6dd4c11:

test ycsb/C/nodes=3/cpu=32/mvcc-range-keys=global failed: VMs preempted during the test run: teamcity-15693235-1718603442-109-n4cpu32-0002 (us-east1-b)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/C/nodes=3/cpu=32/mvcc-range-keys=global/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 35c3e0c133c56c91399e730f49bc488cf6dd4c11:

test ycsb/B/nodes=3/cpu=32/mvcc-range-keys=global failed: VMs preempted during the test run: teamcity-15693235-1718603442-108-n4cpu32-0001 (us-east1-d), teamcity-15693235-1718603442-108-n4cpu32-0004 (us-east1-d), teamcity-15693235-1718603442-108-n4cpu32-0003 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/B/nodes=3/cpu=32/mvcc-range-keys=global/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test cdc/scan/catchup-cold/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-15711854-1718689748-28-n6cpu16-0003 (us-east1-c), teamcity-15711854-1718689748-28-n6cpu16-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: full command output in run_080823.511231258_n6_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/cdc/scan/catchup-cold/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test cdc/scan/catchup/nodes=5/cpu=16/rows=1G/ranges=100/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-15711854-1718689748-29-n6cpu16-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: full command output in run_080855.059230699_n6_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/cdc/scan/catchup/nodes=5/cpu=16/rows=1G/ranges=100/protocol=mux/format=json/sink=null/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test decommissionBench/nodes=4/warehouses=1000/duration=1h0m0s failed: VMs preempted during the test run: teamcity-15711854-1718689748-41-n5cpu16-0003 (us-east1-d)

**Other Failures:**
(decommissionbench.go:847).runDecommissionBenchLong: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/decommissionBench/nodes=4/warehouses=1000/duration=1h0m0s/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test tpccbench/nodes=3/cpu=16/enc=true failed: VMs preempted during the test run: teamcity-15711854-1718689748-24-n4cpu16-0001 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=16/enc=true/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test indexes/2/nodes=6/multi-region failed: VMs preempted during the test run: teamcity-15711854-1718689748-65-n7cpu16-geo-0005 (europe-west2-b)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/indexes/2/nodes=6/multi-region/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test cdc/scan/catchup/nodes=5/cpu=16/rows=1G/ranges=100/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-15727917-1718776181-26-n6cpu16-0002 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/scan/catchup/nodes=5/cpu=16/rows=1G/ranges=100/protocol=mux/format=json/sink=null/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test failover/chaos/read-only/lease=expiration failed: VMs preempted during the test run: teamcity-15727917-1718776181-49-n10cpu2-0009 (us-east1-c), teamcity-15727917-1718776181-49-n10cpu2-0006 (us-east1-c), teamcity-15727917-1718776181-49-n10cpu2-0010 (us-east1-c)

**Other Failures:**
(failover.go:1775).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2398).Run: context canceled
(cluster.go:2398).Run: context canceled
(cluster.go:2398).Run: context canceled
(cluster.go:2398).Run: context canceled
test artifacts and logs in: /artifacts/failover/chaos/read-only/lease=expiration/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test failover/non-system/deadlock failed: VMs preempted during the test run: teamcity-15727917-1718776181-54-n7cpu2-0006 (us-east1-d)

**Other Failures:**
(failover.go:1775).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/failover/non-system/deadlock/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test schemachange/index/tpcc/w=100 failed: VMs preempted during the test run: teamcity-15727917-1718776181-91-n5cpu4-0001 (us-east1-b)

**Other Failures:**
(schemachange.go:511).runAndLogStmts: pq: server is not accepting clients, try another node
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/schemachange/index/tpcc/w=100/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test allocbench/nodes=7/cpu=8/kv/r=0/access=skew failed: VMs preempted during the test run: teamcity-15741119-1718862567-07-n8cpu8-0004 (us-east1-d), teamcity-15741119-1718862567-07-n8cpu8-0003 (us-east1-d), teamcity-15741119-1718862567-07-n8cpu8-0008 (us-east1-d), teamcity-15741119-1718862567-07-n8cpu8-0007 (us-east1-d), teamcity-15741119-1718862567-07-n8cpu8-0001 (us-east1-d), teamcity-15741119-1718862567-07-n8cpu8-0005 (us-east1-d)

**Other Failures:**
(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:396
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:344
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:269
                                main/pkg/cmd/roachtest/test_runner.go:1213
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    monitor failure: EOF [owner=test-eng]
                    (1) attached stack trace
                      -- stack trace:
                      | main.(*monitorImpl).WaitE
                      |     main/pkg/cmd/roachtest/monitor.go:142
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBenchSample
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:395
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBench
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:344
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAllocationBenchSpec.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:269
                      | main.(*testRunner).runTest.func2
                      |     main/pkg/cmd/roachtest/test_runner.go:1213
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) monitor failure
                    Wraps: (3) secondary error attachment
                      | context canceled
                      | (1) context canceled
                      | Error types: (1) *errors.errorString
                    Wraps: (4) EOF [owner=test-eng]
                    Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *secondary.withSecondaryError (4) registry.ErrorWithOwnership
    Test:           allocbench/nodes=7/cpu=8/kv/r=0/access=skew
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/access=skew/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test allocbench/nodes=7/cpu=8/kv/r=0/ops=skew failed: VMs preempted during the test run: teamcity-15741119-1718862567-15-n8cpu8-0003 (us-east1-c), teamcity-15741119-1718862567-15-n8cpu8-0002 (us-east1-c), teamcity-15741119-1718862567-15-n8cpu8-0006 (us-east1-c), teamcity-15741119-1718862567-15-n8cpu8-0001 (us-east1-c), teamcity-15741119-1718862567-15-n8cpu8-0005 (us-east1-c), teamcity-15741119-1718862567-15-n8cpu8-0008 (us-east1-c)

**Other Failures:**
(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:396
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:344
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:269
                                main/pkg/cmd/roachtest/test_runner.go:1213
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    monitor failure: EOF [owner=test-eng]
                    (1) attached stack trace
                      -- stack trace:
                      | main.(*monitorImpl).WaitE
                      |     main/pkg/cmd/roachtest/monitor.go:142
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBenchSample
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:395
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBench
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:344
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAllocationBenchSpec.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:269
                      | main.(*testRunner).runTest.func2
                      |     main/pkg/cmd/roachtest/test_runner.go:1213
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) monitor failure
                    Wraps: (3) secondary error attachment
                      | context canceled
                      | (1) context canceled
                      | Error types: (1) *errors.errorString
                    Wraps: (4) EOF [owner=test-eng]
                    Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *secondary.withSecondaryError (4) registry.ErrorWithOwnership
    Test:           allocbench/nodes=7/cpu=8/kv/r=0/ops=skew
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/ops=skew/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test c2c/MultiRegion/SameRegions/kv0 failed: VMs preempted during the test run: teamcity-15741119-1718862567-21-n9cpu8-geo-0005 (us-central1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/MultiRegion/SameRegions/kv0/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test cdc/workload/kv100/nodes=5/cpu=16/ranges=100/control failed: VMs preempted during the test run: teamcity-15741119-1718862567-37-n7cpu16-0006 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100/control/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test cdc/workload/kv100/nodes=5/cpu=16/ranges=100/server=scheduler/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-15741119-1718862567-40-n7cpu16-0006 (us-east1-b), teamcity-15741119-1718862567-40-n7cpu16-0007 (us-east1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: full command output in run_082508.709081124_n7_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100/server=scheduler/protocol=mux/format=json/sink=null/run_1/ssh/ssh_082508.740659086_n7_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100/server=scheduler/protocol=mux/format=json/sink=null/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test multitenant/tpch/separate_process/enable_direct_scans failed: VMs preempted during the test run: teamcity-15741119-1718862567-102-n1cpu4-0001 (us-east1-c)

**Other Failures:**
(multitenant_tpch.go:70).func2: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/multitenant/tpch/separate_process/enable_direct_scans/run_1/ssh/ssh_113840.090649914_n1_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/multitenant/tpch/separate_process/enable_direct_scans/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test kv0/enc=false/nodes=3/batch=16 failed: VMs preempted during the test run: teamcity-15741119-1718862567-92-n4cpu8-0003 (us-east1-c), teamcity-15741119-1718862567-92-n4cpu8-0001 (us-east1-c), teamcity-15741119-1718862567-92-n4cpu8-0002 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/batch=16/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test kv95limited-spans/enc=false/nodes=1/cpu=32/seq/no-load-splitting failed: VMs preempted during the test run: teamcity-15741119-1718862567-99-n2cpu32-0002 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: full command output in run_113338.134224256_n2_workload-run-kv-tole.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv95limited-spans/enc=false/nodes=1/cpu=32/seq/no-load-splitting/run_1/ssh/ssh_113338.172750411_n2_workload-run-kv-tole.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95limited-spans/enc=false/nodes=1/cpu=32/seq/no-load-splitting/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test kv0/enc=false/nodes=3/size=64kb failed: VMs preempted during the test run: teamcity-15741119-1718862567-95-n4cpu8-0003 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/size=64kb/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test kv/contention/nodes=4 failed: VMs preempted during the test run: teamcity-15741119-1718862567-90-n5cpu4-0005 (us-east1-d), teamcity-15741119-1718862567-90-n5cpu4-0001 (us-east1-d), teamcity-15741119-1718862567-90-n5cpu4-0003 (us-east1-d), teamcity-15741119-1718862567-90-n5cpu4-0004 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv/contention/nodes=4/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test tpccbench/nodes=9/cpu=4/chaos/partition failed: VMs preempted during the test run: teamcity-15741119-1718862567-120-n10cpu4-0005 (us-east1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test tpccbench/nodes=3/cpu=4 failed: VMs preempted during the test run: teamcity-15741119-1718862567-119-n4cpu4-0002 (us-east1-b), teamcity-15741119-1718862567-119-n4cpu4-0001 (us-east1-b)

**Other Failures:**
(cluster.go:2335).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=4/run_1/ssh/ssh_133401.266931465_n2_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=4/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test tpccbench/nodes=9/cpu=4/multi-region failed: VMs preempted during the test run: teamcity-15741119-1718862567-121-n12cpu4hm-geo-0001 (us-east1-b)

**Other Failures:**
(cluster.go:2284).Start: one or more parallel execution failure(s): error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1/ssh/ssh_144814.113440480_n1_init-certs.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test cdc/sink-chaos failed: VMs preempted during the test run: teamcity-15757305-1718948912-45-n4cpu16-0002 (us-east1-d), teamcity-15757305-1718948912-45-n4cpu16-0003 (us-east1-d), teamcity-15757305-1718948912-45-n4cpu16-0004 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: full command output in run_072204.322388706_n4_CONFLUENTCURRENTmntd.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/sink-chaos/run_1/ssh/ssh_072204.369494773_n4_CONFLUENTCURRENTmntd.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cluster.go:2398).Run: context canceled
(cdc.go:388).Close: error shutting down prometheus/grafana: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/sink-chaos/run_1/ssh/ssh_072243.503898158_n4_sudo-rm-rf-tmppromet.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/sink-chaos/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-15757305-1718948912-49-n6cpu16-0005 (us-east1-d), teamcity-15757305-1718948912-49-n6cpu16-0001 (us-east1-d), teamcity-15757305-1718948912-49-n6cpu16-0003 (us-east1-d), teamcity-15757305-1718948912-49-n6cpu16-0004 (us-east1-d), teamcity-15757305-1718948912-49-n6cpu16-0002 (us-east1-d), teamcity-15757305-1718948912-49-n6cpu16-0006 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: full command output in run_072754.311331661_n6_cockroach-workload-i.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/run_1/ssh/ssh_072754.332994686_n6_cockroach-workload-i.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test kv0/enc=false/nodes=1/size=64kb/conc=4096 failed: VMs preempted during the test run: teamcity-15757305-1718948912-98-n2cpu8-0001 (us-east1-b), teamcity-15757305-1718948912-98-n2cpu8-0002 (us-east1-b)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=1/size=64kb/conc=4096/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test kv0/enc=true/nodes=3 failed: VMs preempted during the test run: teamcity-15757305-1718948912-100-n4cpu8-0002 (us-east1-b), teamcity-15757305-1718948912-100-n4cpu8-0004 (us-east1-b)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=true/nodes=3/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test kv0/enc=false/nodes=3/cpu=32 failed: VMs preempted during the test run: teamcity-15757305-1718948912-99-n4cpu32-0002 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/cpu=32/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test kv95/enc=false/nodes=3/cpu=32/mt-shared-process failed: VMs preempted during the test run: teamcity-15757305-1718948912-105-n4cpu32-0001 (us-east1-b)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/cpu=32/mt-shared-process/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test tpccbench/nodes=3/cpu=4/mt-shared-process failed: VMs preempted during the test run: teamcity-15757305-1718948912-120-n4cpu4-0002 (us-east1-d)

**Other Failures:**
(cluster.go:2184).StartServiceForVirtualCluster: one or more parallel execution failure(s): error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=4/mt-shared-process/run_1/ssh/ssh_110806.980022719_n2_dist-local-certs.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=4/mt-shared-process/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test ycsb/B/nodes=3/cpu=32 failed: VMs preempted during the test run: teamcity-15757305-1718948912-126-n4cpu32-0004 (us-east1-b), teamcity-15757305-1718948912-126-n4cpu32-0003 (us-east1-b)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/B/nodes=3/cpu=32/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test zfs/ycsb/A/nodes=3/cpu=32 failed: VMs preempted during the test run: teamcity-15757305-1718948912-129-n4cpu32-0002 (us-east1-b)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/zfs/ycsb/A/nodes=3/cpu=32/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-15773217-1719035367-30-n6cpu16-0002 (us-east1-c), teamcity-15773217-1719035367-30-n6cpu16-0003 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test tpccbench/nodes=3/cpu=16/mt-shared-process failed: VMs preempted during the test run: teamcity-15773217-1719035367-101-n4cpu16-0003 (us-east1-c), teamcity-15773217-1719035367-101-n4cpu16-0001 (us-east1-c), teamcity-15773217-1719035367-101-n4cpu16-0002 (us-east1-c)

**Other Failures:**
(cluster.go:2335).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=16/mt-shared-process/run_1/ssh/ssh_131520.627036124_n3_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=16/mt-shared-process/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test allocbench/nodes=7/cpu=8/kv/r=0/access=skew failed: VMs preempted during the test run: teamcity-15782535-1719121864-05-n8cpu8-0005 (us-east1-d), teamcity-15782535-1719121864-05-n8cpu8-0007 (us-east1-d)

**Other Failures:**
(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:396
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:344
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:269
                                main/pkg/cmd/roachtest/test_runner.go:1213
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    monitor failure: EOF [owner=test-eng]
                    (1) attached stack trace
                      -- stack trace:
                      | main.(*monitorImpl).WaitE
                      |     main/pkg/cmd/roachtest/monitor.go:142
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBenchSample
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:395
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBench
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:344
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAllocationBenchSpec.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:269
                      | main.(*testRunner).runTest.func2
                      |     main/pkg/cmd/roachtest/test_runner.go:1213
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) monitor failure
                    Wraps: (3) secondary error attachment
                      | context canceled
                      | (1) context canceled
                      | Error types: (1) *errors.errorString
                    Wraps: (4) EOF [owner=test-eng]
                    Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *secondary.withSecondaryError (4) registry.ErrorWithOwnership
    Test:           allocbench/nodes=7/cpu=8/kv/r=0/access=skew
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/access=skew/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test c2c/tpcc/warehouses=1000/duration=60/cutover=30 failed: VMs preempted during the test run: teamcity-15782535-1719121864-19-n9cpu8-0003 (us-east1-b), teamcity-15782535-1719121864-19-n9cpu8-0002 (us-east1-b), teamcity-15782535-1719121864-19-n9cpu8-0006 (us-east1-b)

**Other Failures:**
(soon.go:64).SucceedsWithin: condition failed to evaluate within 1h30m0s: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "Cockroach CA")
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/tpcc/warehouses=1000/duration=60/cutover=30/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test failover/chaos/read-write failed: VMs preempted during the test run: teamcity-15782535-1719121864-44-n10cpu2-0006 (us-east1-b)

**Other Failures:**
(failover.go:1775).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2398).Run: context canceled
(cluster.go:2398).Run: context canceled
(cluster.go:2398).Run: context canceled
(cluster.go:2398).Run: context canceled
test artifacts and logs in: /artifacts/failover/chaos/read-write/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test failover/chaos/read-write/lease=expiration failed: VMs preempted during the test run: teamcity-15782535-1719121864-45-n10cpu2-0008 (us-east1-d), teamcity-15782535-1719121864-45-n10cpu2-0007 (us-east1-d), teamcity-15782535-1719121864-45-n10cpu2-0006 (us-east1-d)

**Other Failures:**
(failover.go:1775).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2398).Run: context canceled
(cluster.go:2398).Run: context canceled
(cluster.go:2398).Run: context canceled
(cluster.go:2398).Run: context canceled
test artifacts and logs in: /artifacts/failover/chaos/read-write/lease=expiration/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.2-rc @ 4037be1494ff8f3873edfa2510a7fa79545b413e:

test tpccbench/nodes=9/cpu=4/multi-region failed: VMs preempted during the test run: teamcity-15782535-1719121864-99-n12cpu4hm-geo-0005 (us-west1-b), teamcity-15782535-1719121864-99-n12cpu4hm-geo-0007 (us-west1-b), teamcity-15782535-1719121864-99-n12cpu4hm-geo-0006 (us-west1-b), teamcity-15782535-1719121864-99-n12cpu4hm-geo-0008 (us-west1-b)

**Other Failures:**
(cluster.go:2335).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1/ssh/ssh_154016.336461641_n7_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!