roachtest: vm_preemption failed

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test allocbench/nodes=7/cpu=8/kv/r=95/access=skew failed: VMs preempted during the test run: teamcity-16458788-1723614389-10-n8cpu8-0002 (us-east1-b)

**Other Failures:**
(cluster.go:2424).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/access=skew/run_1/ssh/ssh_065537.125101178_n2_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/access=skew/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-41294

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test cdc/workload/kv100/nodes=5/cpu=16/ranges=100/control failed: VMs preempted during the test run: teamcity-16458788-1723614389-36-n7cpu16-0005 (us-central1-a), teamcity-16458788-1723614389-36-n7cpu16-0002 (us-central1-a), teamcity-16458788-1723614389-36-n7cpu16-0004 (us-central1-a)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100/control/cpu_arch=arm64/run_1

Parameters:

• ROACHTEST_arch=arm64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test cdc/tpcc-1000/sink=cloudstorage failed: VMs preempted during the test run: teamcity-16458788-1723614389-22-n4cpu16-0003 (us-east1-c), teamcity-16458788-1723614389-22-n4cpu16-0002 (us-east1-c), teamcity-16458788-1723614389-22-n4cpu16-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2432).Run: full command output in run_063321.878403662_n4_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
(cdc.go:378).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/tpcc-1000/sink=cloudstorage/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test failover/partial/lease-liveness/lease=expiration failed: VMs preempted during the test run: teamcity-16458788-1723614389-53-n8cpu2-0007 (us-east1-c)

**Other Failures:**
(failover.go:1779).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
(cluster.go:2432).Run: context canceled
test artifacts and logs in: /artifacts/failover/partial/lease-liveness/lease=expiration/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=2
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test failover/partial/lease-liveness failed: VMs preempted during the test run: teamcity-16458788-1723614389-52-n8cpu2-0006 (us-east1-c), teamcity-16458788-1723614389-52-n8cpu2-0001 (us-east1-c)

**Other Failures:**
(failover.go:1779).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
(cluster.go:2432).Run: context canceled
test artifacts and logs in: /artifacts/failover/partial/lease-liveness/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=2
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test schemachange/indexschemachange/index/tpcc/w=800 failed: VMs preempted during the test run: teamcity-16458788-1723614389-39-n5cpu16-0004 (us-east1-d), teamcity-16458788-1723614389-39-n5cpu16-0001 (us-east1-d), teamcity-16458788-1723614389-39-n5cpu16-0005 (us-east1-d), teamcity-16458788-1723614389-39-n5cpu16-0003 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/schemachange/indexschemachange/index/tpcc/w=800/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test tpccbench/nodes=3/cpu=16/enc=true failed: VMs preempted during the test run: teamcity-16458788-1723614389-127-n4cpu16-0001 (us-east1-d)

**Other Failures:**
(cluster.go:2369).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=16/enc=true/run_1/ssh/ssh_110354.805091508_n1_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=16/enc=true/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=true
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test ycsb/D/nodes=3/cpu=32/isolation-level=read-committed failed: VMs preempted during the test run: teamcity-16458788-1723614389-134-n4cpu32-0002 (us-east1-d), teamcity-16458788-1723614389-134-n4cpu32-0001 (us-east1-d), teamcity-16458788-1723614389-134-n4cpu32-0003 (us-east1-d)

**Other Failures:**
(cluster.go:2432).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/ycsb/D/nodes=3/cpu=32/isolation-level=read-committed/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=32
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test tpccbench/nodes=3/cpu=4/enc=true failed: VMs preempted during the test run: teamcity-16458788-1723614389-115-n4cpu4-0001 (us-east1-b)

**Other Failures:**
(cluster.go:2369).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=4/enc=true/run_1/ssh/ssh_112928.984248549_n1_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=4/enc=true/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=true
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test allocbench/nodes=7/cpu=8/kv/r=0/ops=skew failed: VMs preempted during the test run: teamcity-16475340-1723700850-14-n8cpu8-0003 (us-east1-c)

**Other Failures:**
(cluster.go:2424).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/ops=skew/run_1/ssh/ssh_062615.520884664_n3_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/ops=skew/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test cdc/tpcc-1000/sink=kafka failed: VMs preempted during the test run: teamcity-16475340-1723700850-46-n4cpu16-0002 (us-east1-b)

**Other Failures:**
(cluster.go:2432).Run: full command output in run_073015.301595777_n4_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
(cdc.go:378).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/tpcc-1000/sink=kafka/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test allocbench/nodes=7/cpu=8/kv/r=0/access=skew failed: VMs preempted during the test run: teamcity-16475340-1723700850-13-n8cpu8-0001 (us-central1-a)

**Other Failures:**
(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:389
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:339
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:268
                                main/pkg/cmd/roachtest/test_runner.go:1223
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    monitor failure: context canceled
                    (1) attached stack trace
                      -- stack trace:
                      | main.(*monitorImpl).WaitE
                      |     main/pkg/cmd/roachtest/monitor.go:142
                      | [...repeated from below...]
                    Wraps: (2) monitor failure
                    Wraps: (3) attached stack trace
                      -- stack trace:
                      | main.(*monitorImpl).wait
                      |     main/pkg/cmd/roachtest/monitor.go:237
                      | main.(*monitorImpl).WaitE
                      |     main/pkg/cmd/roachtest/monitor.go:142
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBenchSample
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:388
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBench
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:339
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAllocationBenchSpec.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:268
                      | main.(*testRunner).runTest.func2
                      |     main/pkg/cmd/roachtest/test_runner.go:1223
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (4) context canceled
                      | EOF [owner=test-eng]
                        └─ Wraps: (5) EOF [owner=test-eng]
                        └─ Wraps: (6) context canceled
                    Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *join.joinError (5) registry.ErrorWithOwnership (6) *errors.errorString
    Test:           allocbench/nodes=7/cpu=8/kv/r=0/access=skew
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/access=skew/cpu_arch=arm64/run_1

Parameters:

• ROACHTEST_arch=arm64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test kv0/enc=false/nodes=3 failed: VMs preempted during the test run: teamcity-16475340-1723700850-96-n4cpu8-0003 (us-central1-a)

**Other Failures:**
(cluster.go:2432).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/cpu_arch=arm64/run_1

Parameters:

• ROACHTEST_arch=arm64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test kv0/enc=false/nodes=3/cpu=32/mvcc-range-keys=global failed: VMs preempted during the test run: teamcity-16475340-1723700850-98-n4cpu32-0002 (us-east1-c)

**Other Failures:**
(cluster.go:2432).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/cpu=32/mvcc-range-keys=global/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=32
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test kv95/enc=false/nodes=4/ssds=8/raid0 failed: VMs preempted during the test run: teamcity-16475340-1723700850-114-n5cpu8-0005 (us-east1-b)

**Other Failures:**
(cluster.go:2432).Run: full command output in run_101136.863239276_n5_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv95/enc=false/nodes=4/ssds=8/raid0/run_1/ssh/ssh_101136.888590587_n5_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=4/ssds=8/raid0/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=8
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test tpccbench/nodes=9/cpu=4/chaos/partition failed: VMs preempted during the test run: teamcity-16475340-1723700850-142-n10cpu4-0007 (us-east1-d), teamcity-16475340-1723700850-142-n10cpu4-0004 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

                      |   |     github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:173
                      |   | runtime.goexit
                      |   |     src/runtime/asm_amd64.s:1695
                      | Wraps: (3) _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/ops=skew/run_1/ssh/ssh_071934.666453653_n8_cockroach-workload-r.log)
                      | Wraps: (4) TRANSIENT_ERROR(ssh_problem)
                      | Wraps: (5) exit status 255
                      | Error types: (1) *hintdetail.withDetail (2) *withstack.withStack (3) *errutil.withPrefix (4) errors.TransientError (5) *exec.ExitError
                                    └─ Wraps: (11) secondary error attachment
                      | _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/ops=skew/run_1/ssh/ssh_071904.452810134_n8_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
                      | (1) Node 8. Command with error:
                      |   | ``````
                      |   | ./cockroach workload run kv --db=smallreads --read-percent=95 --min-block-bytes=1 --max-block-bytes=1 --max-rate=16500 --tolerate-errors --concurrency=512 --duration=30m0s {pgurl:1-7}
                      |   | ``````
                      |   | <no output>
                      | Wraps: (2) attached stack trace
                      |   -- stack trace:
                      |   | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).errWithDebug
                      |   |     github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:143
                      |   | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).Run.func1
                      |   |     github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:173
                      |   | runtime.goexit
                      |   |     src/runtime/asm_amd64.s:1695
                      | Wraps: (3) _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/ops=skew/run_1/ssh/ssh_071904.452810134_n8_cockroach-workload-r.log)
                      | Wraps: (4) TRANSIENT_ERROR(ssh_problem)
                      | Wraps: (5) exit status 255
                      | Error types: (1) *hintdetail.withDetail (2) *withstack.withStack (3) *errutil.withPrefix (4) errors.TransientError (5) *exec.ExitError
                                      └─ Wraps: (12) Node 8. Command with error:
                      | ``````
                      | ./cockroach workload run kv --db=smallreads --read-percent=95 --min-block-bytes=1 --max-block-bytes=1 --max-rate=16500 --tolerate-errors --concurrency=512 --duration=30m0s {pgurl:1-7}
                      | ``````
                      | <truncated> ... 3932 workload/pgx_helpers.go:240  [-] 4123  error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
                      | W240816 06:58:41.575413 4127 workload/pgx_helpers.go:240  [-] 3967  error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
                      | W240816 06:58:41.695186 4307 workload/pgx_helpers.go:240  [-] 4128  error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
                      | W240816 06:58:41.702285 3791 workload/pgx_helpers.go:240  [-] 4129  error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
                      | Connection to 34.139.108.208 closed by remote host.
                                        └─ Wraps: (13) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).errWithDebug
                      |     github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:143
                      | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).Run.func1
                      |     github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:173
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                                          └─ Wraps: (14) _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/ops=skew/run_1/ssh/ssh_065837.560413379_n8_cockroach-workload-r.log)
                                            └─ Wraps: (15) TRANSIENT_ERROR(ssh_problem)
                                              └─ Wraps: (16) exit status 255
                    Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *join.joinError (5) *withstack.withStack (6) *errutil.withPrefix (7) *markers.withMark (8) *withstack.withStack (9) *errutil.withPrefix (10) *secondary.withSecondaryError (11) *secondary.withSecondaryError (12) *hintdetail.withDetail (13) *withstack.withStack (14) *errutil.withPrefix (15) errors.TransientError (16) *exec.ExitError
    Test:           allocbench/nodes=7/cpu=8/kv/r=95/ops=skew
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/ops=skew/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test cdc/initial-scan failed: VMs preempted during the test run: teamcity-16491888-1723787355-29-n4cpu16-0004 (us-east1-c)

**Other Failures:**
(cluster.go:2432).Run: full command output in run_070259.979233659_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/initial-scan/run_1/ssh/ssh_070300.026792166_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:378).Close: error shutting down prometheus/grafana: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/initial-scan/run_1/ssh/ssh_071939.224817160_n4_sudo-rm-rf-tmppromet.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/initial-scan/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test cdc/initial-scan-only/parquet failed: VMs preempted during the test run: teamcity-16491888-1723787355-30-n4cpu16-0001 (us-east1-c), teamcity-16491888-1723787355-30-n4cpu16-0004 (us-east1-c), teamcity-16491888-1723787355-30-n4cpu16-0003 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: pq: server is not accepting clients, try another node
test artifacts and logs in: /artifacts/cdc/initial-scan-only/parquet/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test cdc/workload/kv100/nodes=5/cpu=16/ranges=100/server=processor/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-16491888-1723787355-36-n7cpu16-0003 (us-east1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100/server=processor/protocol=mux/format=json/sink=null/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test kv0/enc=false/nodes=3/lease=expiration failed: VMs preempted during the test run: teamcity-16491888-1723787355-87-n4cpu8-0002 (us-east1-d)

**Other Failures:**
(cluster.go:2432).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/lease=expiration/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test replicate/up/1to3 failed: VMs preempted during the test run: teamcity-16491888-1723787355-106-n4cpu4-0002 (us-east1-d), teamcity-16491888-1723787355-106-n4cpu4-0001 (us-east1-d)

**Other Failures:**
(allocator.go:57).1: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/replicate/up/1to3/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test tpccbench/nodes=9/cpu=4/chaos/partition failed: VMs preempted during the test run: teamcity-16491888-1723787355-124-n10cpu4-0004 (us-central1-a), teamcity-16491888-1723787355-124-n10cpu4-0003 (us-central1-a), teamcity-16491888-1723787355-124-n10cpu4-0006 (us-central1-a)

**Other Failures:**
(cluster.go:2369).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/cpu_arch=arm64/run_1/ssh/ssh_112209.089798559_n6_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/cpu_arch=arm64/run_1

Parameters:

• ROACHTEST_arch=arm64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test ycsb/D/nodes=3/cpu=32 failed: VMs preempted during the test run: teamcity-16491888-1723787355-128-n4cpu32-0004 (us-east1-b)

**Other Failures:**
(cluster.go:2432).Run: full command output in run_112255.841716489_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/ycsb/D/nodes=3/cpu=32/run_1/ssh/ssh_112255.870257553_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/D/nodes=3/cpu=32/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=32
• ROACHTEST_encrypted=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test cdc/scan/catchup/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-16507469-1723873554-33-n6cpu16-0005 (us-east1-b), teamcity-16507469-1723873554-33-n6cpu16-0002 (us-east1-b), teamcity-16507469-1723873554-33-n6cpu16-0003 (us-east1-b), teamcity-16507469-1723873554-33-n6cpu16-0001 (us-east1-b)

**Other Failures:**
(cluster.go:2432).Run: full command output in run_065920.039705567_n6_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/cdc/scan/catchup/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test ycsb/D/nodes=3 failed: VMs preempted during the test run: teamcity-16507469-1723873554-139-n4cpu8-0001 (us-central1-a)

**Other Failures:**
(cluster.go:2432).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/ycsb/D/nodes=3/cpu_arch=arm64/run_1

Parameters:

• ROACHTEST_arch=arm64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test tpccbench/nodes=6/cpu=16/multi-az failed: VMs preempted during the test run: teamcity-16507469-1723873554-129-n7cpu16-geo-0002 (us-central1-f)

**Other Failures:**
(cluster.go:2369).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=6/cpu=16/multi-az/run_1/ssh/ssh_113102.666310857_n2_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=6/cpu=16/multi-az/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test ycsb/D/nodes=3/cpu=32 failed: VMs preempted during the test run: teamcity-16507469-1723873554-141-n4cpu32-0003 (us-east1-d)

**Other Failures:**
(cluster.go:2432).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/ycsb/D/nodes=3/cpu=32/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=32
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test tpccbench/nodes=3/cpu=4 failed: VMs preempted during the test run: teamcity-16507469-1723873554-127-n4cpu4-0001 (us-east1-d), teamcity-16507469-1723873554-127-n4cpu4-0003 (us-east1-d), teamcity-16507469-1723873554-127-n4cpu4-0002 (us-east1-d)

**Other Failures:**
(cluster.go:2369).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=4/run_1/ssh/ssh_113910.176901419_n2_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=4/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test kv95/enc=false/nodes=3/tracing failed: VMs preempted during the test run: teamcity-16516865-1723959954-90-n4cpu8-0002 (us-east1-d), teamcity-16516865-1723959954-90-n4cpu8-0001 (us-east1-d)

**Other Failures:**
(cluster.go:2432).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/tracing/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test ycsb/F/nodes=3/cpu=32 failed: VMs preempted during the test run: teamcity-16516865-1723959954-139-n4cpu32-0003 (us-east1-b)

**Other Failures:**
(cluster.go:2432).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/ycsb/F/nodes=3/cpu=32/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=32
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test allocbench/nodes=7/cpu=8/kv/r=50/ops=skew failed: VMs preempted during the test run: teamcity-16526232-1724046421-11-n8cpu8-0005 (us-east1-c), teamcity-16526232-1724046421-11-n8cpu8-0007 (us-east1-c), teamcity-16526232-1724046421-11-n8cpu8-0001 (us-east1-c), teamcity-16526232-1724046421-11-n8cpu8-0003 (us-east1-c), teamcity-16526232-1724046421-11-n8cpu8-0008 (us-east1-c), teamcity-16526232-1724046421-11-n8cpu8-0004 (us-east1-c), teamcity-16526232-1724046421-11-n8cpu8-0006 (us-east1-c)

**Other Failures:**
(cluster.go:2424).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1/ssh/ssh_062646.337134588_n6_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test allocbench/nodes=7/cpu=8/kv/r=0/access=skew failed: VMs preempted during the test run: teamcity-16526232-1724046421-10-n8cpu8-0007 (us-east1-c), teamcity-16526232-1724046421-10-n8cpu8-0006 (us-east1-c), teamcity-16526232-1724046421-10-n8cpu8-0002 (us-east1-c), teamcity-16526232-1724046421-10-n8cpu8-0005 (us-east1-c), teamcity-16526232-1724046421-10-n8cpu8-0004 (us-east1-c), teamcity-16526232-1724046421-10-n8cpu8-0003 (us-east1-c)

**Other Failures:**
(cluster.go:2424).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/access=skew/run_1/ssh/ssh_062726.295048245_n3_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/access=skew/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test cdc/ledger failed: VMs preempted during the test run: teamcity-16526232-1724046421-29-n4cpu16-0001 (us-east1-c), teamcity-16526232-1724046421-29-n4cpu16-0002 (us-east1-c), teamcity-16526232-1724046421-29-n4cpu16-0003 (us-east1-c)

**Other Failures:**
(cluster.go:2432).Run: full command output in run_070200.398832547_n4_workload-run-ledger-.log: COMMAND_PROBLEM: exit status 1
(ts_util.go:59).mustGetMetrics: roachtestutil.addCookies: unable to extract sessionID: roachtestutil.GetSessionID: failed to authenticate: context canceled
(cdc.go:378).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/ledger/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test failover/non-system/pause failed: VMs preempted during the test run: teamcity-16526232-1724046421-65-n7cpu2-0005 (us-east1-c)

**Other Failures:**
(failover.go:1779).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/failover/non-system/pause/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=2
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test failover/non-system/pause/lease=expiration failed: VMs preempted during the test run: teamcity-16526232-1724046421-66-n7cpu2-0006 (us-east1-c)

**Other Failures:**
(failover.go:1779).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/failover/non-system/pause/lease=expiration/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=2
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test kv95/enc=false/nodes=3/cpu=32/mvcc-range-keys=global failed: VMs preempted during the test run: teamcity-16526232-1724046421-102-n4cpu32-0001 (us-east1-d)

**Other Failures:**
(cluster.go:2432).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/cpu=32/mvcc-range-keys=global/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=32
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test kv0/enc=false/nodes=3/cpu=32/size=64kb failed: VMs preempted during the test run: teamcity-16526232-1724046421-92-n4cpu32-0003 (us-east1-c), teamcity-16526232-1724046421-92-n4cpu32-0002 (us-east1-c)

**Other Failures:**
(cluster.go:2432).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/cpu=32/size=64kb/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=32
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test kv95/enc=false/nodes=3/lease=expiration failed: VMs preempted during the test run: teamcity-16526232-1724046421-103-n4cpu8-0002 (us-east1-c), teamcity-16526232-1724046421-103-n4cpu8-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2432).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/lease=expiration/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test kv95/enc=false/nodes=3 failed: VMs preempted during the test run: teamcity-16526232-1724046421-100-n4cpu8-0002 (us-east1-d)

**Other Failures:**
(cluster.go:2432).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test ycsb/A/nodes=3/cpu=32 failed: VMs preempted during the test run: teamcity-16526232-1724046421-142-n4cpu32-0004 (us-east1-b)

**Other Failures:**
(cluster.go:2432).Run: full command output in run_110146.164292139_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/ycsb/A/nodes=3/cpu=32/run_1/ssh/ssh_110146.212276529_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/A/nodes=3/cpu=32/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=32
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test tpccbench/nodes=9/cpu=4/multi-region failed: VMs preempted during the test run: teamcity-16526232-1724046421-137-n12cpu4hm-geo-0002 (us-east1-b), teamcity-16526232-1724046421-137-n12cpu4hm-geo-0004 (us-east1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test allocbench/nodes=7/cpu=8/kv/r=95/ops=skew failed: VMs preempted during the test run: teamcity-16544056-1724132918-11-n8cpu8-0003 (us-east1-b)

**Other Failures:**
(cluster.go:2424).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/ops=skew/run_1/ssh/ssh_062628.047668449_n3_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/ops=skew/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test cdc/webhook-sink failed: VMs preempted during the test run: teamcity-16544056-1724132918-28-n4cpu16-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2432).Run: full command output in run_070659.067905290_n4_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
(cdc.go:378).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/webhook-sink/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test kv0/enc=false/nodes=1 failed: VMs preempted during the test run: teamcity-16544056-1724132918-77-n2cpu8-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2432).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=1/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test restore/tpce/400GB/gce/nodes=4/cpus=8/lowmem failed: VMs preempted during the test run: teamcity-16544056-1724132918-104-n4cpu8lm-0002 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/restore/tpce/400GB/gce/nodes=4/cpus=8/lowmem/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test ycsb/B/nodes=3 failed: VMs preempted during the test run: teamcity-16544056-1724132918-126-n4cpu8-0001 (us-east1-b)

**Other Failures:**
(cluster.go:2432).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/ycsb/B/nodes=3/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test tpcdsvec failed: VMs preempted during the test run: teamcity-16544056-1724132918-123-n3cpu4-0001 (us-east1-b), teamcity-16544056-1724132918-123-n3cpu4-0003 (us-east1-b)

**Other Failures:**
(tpcdsvec.go:109).1: pgx conn: failed to connect to ``host=34.75.108.121 user=roachprod database=``: server error (ERROR: server is not accepting clients, try another node (SQLSTATE 57P01))
test artifacts and logs in: /artifacts/tpcdsvec/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test import/tpch/nodes=8 failed: VMs preempted during the test run: teamcity-16544056-1724132918-61-n8cpu4-0007 (us-east1-c), teamcity-16544056-1724132918-61-n8cpu4-0005 (us-east1-c), teamcity-16544056-1724132918-61-n8cpu4-0002 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/import/tpch/nodes=8/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=true
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test scrub/index-only/tpcc/w=100 failed: VMs preempted during the test run: teamcity-16544056-1724132918-107-n5cpu4-0005 (us-east1-c)

**Other Failures:**
(cluster.go:2432).Run: full command output in run_114438.684608075_n5_cockroach-workload-c.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/scrub/index-only/tpcc/w=100/run_1/ssh/ssh_114438.708848947_n5_cockroach-workload-c.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/scrub/index-only/tpcc/w=100/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test allocbench/nodes=7/cpu=8/kv/r=0/access=skew failed: VMs preempted during the test run: teamcity-16558678-1724219180-10-n8cpu8-0006 (us-east1-c), teamcity-16558678-1724219180-10-n8cpu8-0002 (us-east1-c), teamcity-16558678-1724219180-10-n8cpu8-0005 (us-east1-c), teamcity-16558678-1724219180-10-n8cpu8-0007 (us-east1-c)

**Other Failures:**
(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:389
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:339
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:268
                                main/pkg/cmd/roachtest/test_runner.go:1223
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    monitor failure: context canceled
                    (1) attached stack trace
                      -- stack trace:
                      | main.(*monitorImpl).WaitE
                      |     main/pkg/cmd/roachtest/monitor.go:142
                      | [...repeated from below...]
                    Wraps: (2) monitor failure
                    Wraps: (3) attached stack trace
                      -- stack trace:
                      | main.(*monitorImpl).wait
                      |     main/pkg/cmd/roachtest/monitor.go:237
                      | main.(*monitorImpl).WaitE
                      |     main/pkg/cmd/roachtest/monitor.go:142
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBenchSample
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:388
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBench
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:339
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAllocationBenchSpec.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:268
                      | main.(*testRunner).runTest.func2
                      |     main/pkg/cmd/roachtest/test_runner.go:1223
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (4) context canceled
                      | EOF [owner=test-eng]
                        └─ Wraps: (5) EOF [owner=test-eng]
                        └─ Wraps: (6) context canceled
                    Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *join.joinError (5) registry.ErrorWithOwnership (6) *errors.errorString
    Test:           allocbench/nodes=7/cpu=8/kv/r=0/access=skew
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/access=skew/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!