roachtest: vm_preemption failed

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test allocbench/nodes=7/cpu=8/kv/r=95/access=skew failed: VMs preempted during the test run: teamcity-16298889-1722577605-11-n8cpu8-0002 (us-east1-c), teamcity-16298889-1722577605-11-n8cpu8-0007 (us-east1-c)

**Other Failures:**
(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:389
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:339
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:268
                                main/pkg/cmd/roachtest/test_runner.go:1248
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    monitor failure: context canceled
                    (1) attached stack trace
                      -- stack trace:
                      | main.(*monitorImpl).WaitE
                      |     main/pkg/cmd/roachtest/monitor.go:142
                      | [...repeated from below...]
                    Wraps: (2) monitor failure
                    Wraps: (3) attached stack trace
                      -- stack trace:
                      | main.(*monitorImpl).wait
                      |     main/pkg/cmd/roachtest/monitor.go:237
                      | main.(*monitorImpl).WaitE
                      |     main/pkg/cmd/roachtest/monitor.go:142
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBenchSample
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:388
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBench
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:339
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAllocationBenchSpec.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:268
                      | main.(*testRunner).runTest.func2
                      |     main/pkg/cmd/roachtest/test_runner.go:1248
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (4) context canceled
                      | EOF [owner=test-eng]
                        └─ Wraps: (5) EOF [owner=test-eng]
                        └─ Wraps: (6) context canceled
                    Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *join.joinError (5) registry.ErrorWithOwnership (6) *errors.errorString
    Test:           allocbench/nodes=7/cpu=8/kv/r=95/access=skew
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/access=skew/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-40816

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:

test admission-control/elastic-io failed: VMs preempted during the test run: teamcity-16475370-1723700895-09-n2cpu8-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2419).Run: context canceled
(test_runner.go:1277).runTest: test timed out (1h0m0s)
test artifacts and logs in: /artifacts/admission-control/elastic-io/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:

test cdc/kafka-chaos failed: VMs preempted during the test run: teamcity-16475370-1723700895-33-n4cpu16-0002 (us-central1-a), teamcity-16475370-1723700895-33-n4cpu16-0001 (us-central1-a), teamcity-16475370-1723700895-33-n4cpu16-0003 (us-central1-a)

**Other Failures:**
(cluster.go:2419).Run: full command output in run_064929.896262283_n4_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
(cdc.go:379).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/kafka-chaos/cpu_arch=arm64/run_1

Parameters:

• ROACHTEST_arch=arm64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:

test cdc/scan/catchup/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-16475370-1723700895-35-n6cpu16-0004 (us-east1-b), teamcity-16475370-1723700895-35-n6cpu16-0001 (us-east1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/scan/catchup/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:

test decommissionBench/nodes=6/warehouses=1000/drain-first/while-upreplicating/target=3/multi-region failed: VMs preempted during the test run: teamcity-16475370-1723700895-51-n8cpu16-geo-0004 (us-west1-b), teamcity-16475370-1723700895-51-n8cpu16-geo-0003 (us-west1-b), teamcity-16475370-1723700895-51-n8cpu16-geo-0002 (us-east1-b)

**Other Failures:**
(decommissionbench.go:727).runDecommissionBench: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/decommissionBench/nodes=6/warehouses=1000/drain-first/while-upreplicating/target=3/multi-region/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:

test kv95/enc=false/nodes=1/cpu=32 failed: VMs preempted during the test run: teamcity-16475370-1723700895-101-n2cpu32-0002 (us-east1-c)

**Other Failures:**
(cluster.go:2419).Run: full command output in run_093522.974720762_n2_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv95/enc=false/nodes=1/cpu=32/run_1/ssh/ssh_093523.051392510_n2_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=1/cpu=32/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=32
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:

test kv0/enc=true/nodes=3 failed: VMs preempted during the test run: teamcity-16475370-1723700895-100-n4cpu8-0004 (us-east1-c)

**Other Failures:**
(cluster.go:2419).Run: full command output in run_093522.948660052_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv0/enc=true/nodes=3/run_1/ssh/ssh_093523.011705592_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=true/nodes=3/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=true
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:

test sysbench/oltp_read_write/nodes=3/cpu=32/conc=256 failed: VMs preempted during the test run: teamcity-16475370-1723700895-133-n4cpu32-0004 (us-east1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/sysbench/oltp_read_write/nodes=3/cpu=32/conc=256/run_1/ssh/ssh_111241.841556199_n4_sysbench-dbdriverpgs.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/sysbench/oltp_read_write/nodes=3/cpu=32/conc=256/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=32
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:

test tpccbench/nodes=3/cpu=16/enc=true failed: VMs preempted during the test run: teamcity-16475370-1723700895-143-n4cpu16-0002 (us-east1-c), teamcity-16475370-1723700895-143-n4cpu16-0001 (us-east1-c), teamcity-16475370-1723700895-143-n4cpu16-0004 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=16/enc=true/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=true
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:

test sysbench/oltp_update_index/nodes=3/cpu=32/conc=256 failed: VMs preempted during the test run: teamcity-16475370-1723700895-135-n4cpu32-0001 (us-east1-c), teamcity-16475370-1723700895-135-n4cpu32-0002 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/sysbench/oltp_update_index/nodes=3/cpu=32/conc=256/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=32
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 953d09cce5a01cc0053e138516d80a27454f027c:

test zfs/ycsb/A/nodes=3/cpu=8 failed: VMs preempted during the test run: teamcity-16475370-1723700895-154-n4cpu8-0001 (us-east1-d)

**Other Failures:**
(cluster.go:2419).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/zfs/ycsb/A/nodes=3/cpu=8/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=zfs
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ b0d361aa4ae46893a7157dc4f010923dc0918984:

test cdc/tpcc-1000/sink=cloudstorage failed: VMs preempted during the test run: teamcity-16491918-1723787410-37-n4cpu16-0003 (us-east1-c), teamcity-16491918-1723787410-37-n4cpu16-0001 (us-east1-c), teamcity-16491918-1723787410-37-n4cpu16-0004 (us-east1-c), teamcity-16491918-1723787410-37-n4cpu16-0002 (us-east1-c)

**Other Failures:**
(cluster.go:2320).Start: one or more parallel execution failure(s): error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/tpcc-1000/sink=cloudstorage/run_1/ssh/ssh_072013.345336256_n1_init-certs.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/tpcc-1000/sink=cloudstorage/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ b0d361aa4ae46893a7157dc4f010923dc0918984:

test cdc/scan/catchup-cold/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-16491918-1723787410-35-n6cpu16-0001 (us-east1-c), teamcity-16491918-1723787410-35-n6cpu16-0006 (us-east1-c), teamcity-16491918-1723787410-35-n6cpu16-0004 (us-east1-c), teamcity-16491918-1723787410-35-n6cpu16-0005 (us-east1-c)

**Other Failures:**
(cluster.go:2434).Run: full command output in run_071624.246041536_n6_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/cdc/scan/catchup-cold/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ b0d361aa4ae46893a7157dc4f010923dc0918984:

test cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-16491918-1723787410-36-n6cpu16-0004 (us-east1-c), teamcity-16491918-1723787410-36-n6cpu16-0002 (us-east1-c), teamcity-16491918-1723787410-36-n6cpu16-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2434).Run: full command output in run_071828.059907474_n6_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100/protocol=mux/format=json/sink=null/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ b0d361aa4ae46893a7157dc4f010923dc0918984:

test cdc/workload/kv0/nodes=5/cpu=16/ranges=100/server=processor/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-16491918-1723787410-40-n7cpu16-0006 (us-east1-b), teamcity-16491918-1723787410-40-n7cpu16-0001 (us-east1-b), teamcity-16491918-1723787410-40-n7cpu16-0007 (us-east1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100/server=processor/protocol=mux/format=json/sink=null/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ b0d361aa4ae46893a7157dc4f010923dc0918984:

test decommissionBench/nodes=4/warehouses=1000/duration=1h0m0s failed: VMs preempted during the test run: teamcity-16491918-1723787410-55-n5cpu16-0004 (us-east1-b)

**Other Failures:**
(decommissionbench.go:853).runDecommissionBenchLong: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/decommissionBench/nodes=4/warehouses=1000/duration=1h0m0s/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ b0d361aa4ae46893a7157dc4f010923dc0918984:

test schemachange/during/tpcc failed: VMs preempted during the test run: teamcity-16491918-1723787410-54-n5cpu16-0003 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/schemachange/during/tpcc/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ b0d361aa4ae46893a7157dc4f010923dc0918984:

test kv95/enc=false/nodes=3/lease=expiration failed: VMs preempted during the test run: teamcity-16491918-1723787410-104-n4cpu8-0003 (us-central1-a)

**Other Failures:**
(cluster.go:2434).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/lease=expiration/cpu_arch=arm64/run_1

Parameters:

• ROACHTEST_arch=arm64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ b0d361aa4ae46893a7157dc4f010923dc0918984:

test kv0/enc=false/nodes=3/size=4kb failed: VMs preempted during the test run: teamcity-16491918-1723787410-98-n4cpu8-0002 (us-east1-d)

**Other Failures:**
(cluster.go:2434).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/size=4kb/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ b0d361aa4ae46893a7157dc4f010923dc0918984:

test kv0/enc=false/nodes=1/cpu=32 failed: VMs preempted during the test run: teamcity-16491918-1723787410-90-n2cpu32-0001 (us-east1-d)

**Other Failures:**
(cluster.go:2434).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=1/cpu=32/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=32
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ b0d361aa4ae46893a7157dc4f010923dc0918984:

test kv95/enc=false/nodes=3 failed: VMs preempted during the test run: teamcity-16491918-1723787410-102-n4cpu8-0004 (us-east1-d)

**Other Failures:**
(cluster.go:2434).Run: full command output in run_101828.685829553_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv95/enc=false/nodes=3/run_1/ssh/ssh_101828.733693804_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ b0d361aa4ae46893a7157dc4f010923dc0918984:

test kv0/enc=false/nodes=3/size=64kb failed: VMs preempted during the test run: teamcity-16491918-1723787410-100-n4cpu8-0004 (us-east1-d), teamcity-16491918-1723787410-100-n4cpu8-0001 (us-east1-d), teamcity-16491918-1723787410-100-n4cpu8-0002 (us-east1-d)

**Other Failures:**
(cluster.go:2434).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/size=64kb/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ b0d361aa4ae46893a7157dc4f010923dc0918984:

test kv95/enc=false/nodes=1/cpu=32 failed: VMs preempted during the test run: teamcity-16491918-1723787410-101-n2cpu32-0002 (us-east1-d)

**Other Failures:**
(cluster.go:2434).Run: full command output in run_101739.647517813_n2_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv95/enc=false/nodes=1/cpu=32/run_1/ssh/ssh_101739.676103773_n2_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=1/cpu=32/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=32
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ b0d361aa4ae46893a7157dc4f010923dc0918984:

test ycsb/A/nodes=3 failed: VMs preempted during the test run: teamcity-16491918-1723787410-92-n4cpu8-0003 (us-east1-d)

**Other Failures:**
(cluster.go:2434).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/ycsb/A/nodes=3/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ b0d361aa4ae46893a7157dc4f010923dc0918984:

test sysbench/oltp_update_non_index/nodes=3/cpu=32/conc=256 failed: VMs preempted during the test run: teamcity-16491918-1723787410-127-n4cpu32-0002 (us-east1-b)

**Other Failures:**
(cluster.go:2434).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/sysbench/oltp_update_non_index/nodes=3/cpu=32/conc=256/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=32
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ b0d361aa4ae46893a7157dc4f010923dc0918984:

test restore/tpce/400GB/gce/nodes=4/cpus=8/lowmem failed: VMs preempted during the test run: teamcity-16491918-1723787410-121-n4cpu8lm-0001 (us-east1-c), teamcity-16491918-1723787410-121-n4cpu8lm-0003 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/restore/tpce/400GB/gce/nodes=4/cpus=8/lowmem/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ b0d361aa4ae46893a7157dc4f010923dc0918984:

test tpccbench/nodes=3/cpu=16/lease=expiration failed: VMs preempted during the test run: teamcity-16491918-1723787410-130-n4cpu16-0004 (us-east1-c), teamcity-16491918-1723787410-130-n4cpu16-0001 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=16/lease=expiration/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ b0d361aa4ae46893a7157dc4f010923dc0918984:

test tpcc/headroom/isolation-level=read-committed/n4cpu16 failed: VMs preempted during the test run: teamcity-16491918-1723787410-128-n4cpu16-0003 (us-central1-a)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpcc/headroom/isolation-level=read-committed/n4cpu16/cpu_arch=arm64/run_1

Parameters:

• ROACHTEST_arch=arm64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=true
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ b0d361aa4ae46893a7157dc4f010923dc0918984:

test ycsb/E/nodes=3/cpu=32/uniform failed: VMs preempted during the test run: teamcity-16491918-1723787410-96-n4cpu32-0004 (us-east1-c)

**Other Failures:**
(cluster.go:2434).Run: full command output in run_115221.847632219_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/ycsb/E/nodes=3/cpu=32/uniform/run_1/ssh/ssh_115221.873986858_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/E/nodes=3/cpu=32/uniform/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=32
• ROACHTEST_encrypted=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ b0d361aa4ae46893a7157dc4f010923dc0918984:

test schemachange/indexschemachange/index/tpcc/w=800 failed: VMs preempted during the test run: teamcity-16491918-1723787410-123-n5cpu16-0004 (us-east1-c), teamcity-16491918-1723787410-123-n5cpu16-0001 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/schemachange/indexschemachange/index/tpcc/w=800/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ b0d361aa4ae46893a7157dc4f010923dc0918984:

test ycsb/D/nodes=3/cpu=32/uniform failed: VMs preempted during the test run: teamcity-16491918-1723787410-143-n4cpu32-0003 (us-east1-c)

**Other Failures:**
(cluster.go:2434).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/ycsb/D/nodes=3/cpu=32/uniform/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=32
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ b0d361aa4ae46893a7157dc4f010923dc0918984:

test tpccbench/nodes=9/cpu=4/multi-region failed: VMs preempted during the test run: teamcity-16491918-1723787410-132-n12cpu4hm-geo-0007 (us-west1-b), teamcity-16491918-1723787410-132-n12cpu4hm-geo-0006 (us-west1-b), teamcity-16491918-1723787410-132-n12cpu4hm-geo-0008 (us-west1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ acb89a54d7923817d5fbdc11cadc91c60fe4345f:

test cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-16507499-1723873728-29-n6cpu16-0003 (us-east1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
failed to list cockroach processes: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100/protocol=mux/format=json/sink=null/run_1/ssh/ssh_071946.012083305_n3_list-processes.log): TRANSIENT_ERROR(ssh_problem): exit status 255 [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100/protocol=mux/format=json/sink=null/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ acb89a54d7923817d5fbdc11cadc91c60fe4345f:

test tpcc-nowait/isolation-level=snapshot/nodes=3/w=1 failed: VMs preempted during the test run: teamcity-16507499-1723873728-28-n4cpu16-0003 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpcc-nowait/isolation-level=snapshot/nodes=3/w=1/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ acb89a54d7923817d5fbdc11cadc91c60fe4345f:

test failover/partial/lease-liveness failed: VMs preempted during the test run: teamcity-16507499-1723873728-58-n8cpu2-0003 (us-east1-c)

**Other Failures:**
(failover.go:1774).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
(cluster.go:2434).Run: context canceled
test artifacts and logs in: /artifacts/failover/partial/lease-liveness/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=2
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ acb89a54d7923817d5fbdc11cadc91c60fe4345f:

test import/tpch/nodes=8 failed: VMs preempted during the test run: teamcity-16507499-1723873728-65-n8cpu4-0003 (us-east1-c), teamcity-16507499-1723873728-65-n8cpu4-0001 (us-east1-c), teamcity-16507499-1723873728-65-n8cpu4-0002 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/import/tpch/nodes=8/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ acb89a54d7923817d5fbdc11cadc91c60fe4345f:

test import/tpcc/warehouses=1000/nodes=4 failed: VMs preempted during the test run: teamcity-16507499-1723873728-63-n4cpu4-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2434).Run: context canceled
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/import/tpcc/warehouses=1000/nodes=4/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ acb89a54d7923817d5fbdc11cadc91c60fe4345f:

test kv0/enc=false/nodes=1/cpu=32 failed: VMs preempted during the test run: teamcity-16507499-1723873728-82-n2cpu32-0001 (us-east1-b)

**Other Failures:**
(cluster.go:2434).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=1/cpu=32/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=32
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ acb89a54d7923817d5fbdc11cadc91c60fe4345f:

test tpccbench/nodes=3/cpu=16 failed: VMs preempted during the test run: teamcity-16507499-1723873728-130-n4cpu16-0002 (us-central1-a), teamcity-16507499-1723873728-130-n4cpu16-0001 (us-central1-a), teamcity-16507499-1723873728-130-n4cpu16-0003 (us-central1-a)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=16/cpu_arch=arm64/run_1

Parameters:

• ROACHTEST_arch=arm64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ acb89a54d7923817d5fbdc11cadc91c60fe4345f:

test tpccbench/nodes=6/cpu=16/multi-az failed: VMs preempted during the test run: teamcity-16507499-1723873728-133-n7cpu16-geo-0002 (us-central1-f), teamcity-16507499-1723873728-133-n7cpu16-geo-0001 (us-central1-f)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpccbench/nodes=6/cpu=16/multi-az/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ acb89a54d7923817d5fbdc11cadc91c60fe4345f:

test tpccbench/nodes=3/cpu=4/backups=true failed: VMs preempted during the test run: teamcity-16507499-1723873728-132-n4cpu4-0004 (us-east1-d), teamcity-16507499-1723873728-132-n4cpu4-0003 (us-east1-d)

**Other Failures:**
(cluster.go:2320).Start: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=4/backups=true/run_1/ssh/ssh_113946.275401842_n3_upload-start-script.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=4/backups=true/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ acb89a54d7923817d5fbdc11cadc91c60fe4345f:

test allocbench/nodes=7/cpu=8/kv/r=0/access=skew failed: VMs preempted during the test run: teamcity-16516895-1723960029-03-n8cpu8-0002 (us-east1-c)

**Other Failures:**
(cluster.go:2426).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/access=skew/run_1/ssh/ssh_075747.818442128_n2_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/access=skew/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ acb89a54d7923817d5fbdc11cadc91c60fe4345f:

test cdc/tpcc-1000/sink=cloudstorage failed: VMs preempted during the test run: teamcity-16516895-1723960029-34-n4cpu16-0001 (us-east1-c), teamcity-16516895-1723960029-34-n4cpu16-0002 (us-east1-c), teamcity-16516895-1723960029-34-n4cpu16-0004 (us-east1-c)

**Other Failures:**
(cluster.go:2434).Run: full command output in run_072043.039010418_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/tpcc-1000/sink=cloudstorage/run_1/ssh/ssh_072043.164319666_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:379).Close: error shutting down prometheus/grafana: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/tpcc-1000/sink=cloudstorage/run_1/ssh/ssh_083319.642252065_n4_sudo-rm-rf-tmppromet.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/tpcc-1000/sink=cloudstorage/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ acb89a54d7923817d5fbdc11cadc91c60fe4345f:

test kv0/enc=false/nodes=1/cpu=32/mt-shared-process failed: VMs preempted during the test run: teamcity-16516895-1723960029-90-n2cpu32-0001 (us-east1-d)

**Other Failures:**
(cluster.go:2434).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=1/cpu=32/mt-shared-process/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=32
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ acb89a54d7923817d5fbdc11cadc91c60fe4345f:

test ycsb/A/nodes=3 failed: VMs preempted during the test run: teamcity-16516895-1723960029-91-n4cpu8-0002 (us-east1-d)

**Other Failures:**
(cluster.go:2434).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/ycsb/A/nodes=3/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ acb89a54d7923817d5fbdc11cadc91c60fe4345f:

test cdc/scan/catchup/nodes=5/cpu=16/rows=1G/ranges=100/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-16526262-1724046578-26-n6cpu16-0006 (us-east1-c), teamcity-16526262-1724046578-26-n6cpu16-0003 (us-east1-c), teamcity-16526262-1724046578-26-n6cpu16-0005 (us-east1-c)

**Other Failures:**
(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_bench.go:307
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_bench.go:98
                                main/pkg/cmd/roachtest/test_runner.go:1256
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    pq: server is not accepting clients, try another node
    Test:           cdc/scan/catchup/nodes=5/cpu=16/rows=1G/ranges=100/protocol=mux/format=json/sink=null
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/cdc/scan/catchup/nodes=5/cpu=16/rows=1G/ranges=100/protocol=mux/format=json/sink=null/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ acb89a54d7923817d5fbdc11cadc91c60fe4345f:

test cdc/workload/kv0/nodes=5/cpu=16/ranges=100/server=scheduler/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-16526262-1724046578-28-n7cpu16-0007 (us-east1-c), teamcity-16526262-1724046578-28-n7cpu16-0001 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100/server=scheduler/protocol=mux/format=json/sink=null/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ acb89a54d7923817d5fbdc11cadc91c60fe4345f:

test decommissionBench/nodes=6/warehouses=1000/drain-first/while-upreplicating/target=2/multi-region failed: VMs preempted during the test run: teamcity-16526262-1724046578-39-n8cpu16-geo-0005 (us-central1-b)

**Other Failures:**
(cluster.go:2320).Start: one or more parallel execution failure(s): error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/decommissionBench/nodes=6/warehouses=1000/drain-first/while-upreplicating/target=2/multi-region/run_1/ssh/ssh_075843.943969433_n5_dist-local-certs.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/decommissionBench/nodes=6/warehouses=1000/drain-first/while-upreplicating/target=2/multi-region/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ acb89a54d7923817d5fbdc11cadc91c60fe4345f:

test cdc/scan/catchup-cold/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-16526262-1724046578-25-n6cpu16-0005 (us-central1-a)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
failed to list cockroach processes: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/scan/catchup-cold/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/cpu_arch=arm64/run_1/ssh/ssh_080830.454018191_n5_list-processes.log): TRANSIENT_ERROR(ssh_problem): exit status 255 [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/scan/catchup-cold/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/cpu_arch=arm64/run_1

Parameters:

• ROACHTEST_arch=arm64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ acb89a54d7923817d5fbdc11cadc91c60fe4345f:

test kv0/enc=false/nodes=3/cpu=32/seq failed: VMs preempted during the test run: teamcity-16526262-1724046578-85-n4cpu32-0003 (us-east1-c), teamcity-16526262-1724046578-85-n4cpu32-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2434).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/cpu=32/seq/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=32
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ acb89a54d7923817d5fbdc11cadc91c60fe4345f:

test ycsb/E/nodes=3 failed: VMs preempted during the test run: teamcity-16526262-1724046578-88-n4cpu8-0003 (us-east1-d)

**Other Failures:**
(cluster.go:2434).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/ycsb/E/nodes=3/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128948 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!