roachtest: vm_preemption failed

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test allocbench/nodes=7/cpu=8/kv/r=95/access=skew failed: VMs preempted during the test run: teamcity-16458788-1723614389-10-n8cpu8-0002 (us-east1-b)

**Other Failures:**
(cluster.go:2424).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/access=skew/run_1/ssh/ssh_065537.125101178_n2_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/access=skew/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-41294

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test allocbench/nodes=7/cpu=8/kv/r=95/ops=skew failed: VMs preempted during the test run: teamcity-16558678-1724219180-12-n8cpu8-0002 (us-east1-c), teamcity-16558678-1724219180-12-n8cpu8-0006 (us-east1-c), teamcity-16558678-1724219180-12-n8cpu8-0005 (us-east1-c)

**Other Failures:**
(cluster.go:2424).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/ops=skew/run_1/ssh/ssh_065630.519160429_n5_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/ops=skew/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test kv95/enc=true/nodes=1 failed: VMs preempted during the test run: teamcity-16558678-1724219180-08-n2cpu8-0001 (us-east1-b)

**Other Failures:**
(cluster.go:2432).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv95/enc=true/nodes=1/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/control failed: VMs preempted during the test run: teamcity-16558678-1724219180-36-n7cpu16-0003 (us-east1-d), teamcity-16558678-1724219180-36-n7cpu16-0002 (us-east1-d), teamcity-16558678-1724219180-36-n7cpu16-0006 (us-east1-d), teamcity-16558678-1724219180-36-n7cpu16-0007 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/control/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test decommissionBench/nodes=6/warehouses=1000/drain-first/while-upreplicating/target=2/multi-region failed: VMs preempted during the test run: teamcity-16558678-1724219180-45-n8cpu16-geo-0007 (us-east1-b), teamcity-16558678-1724219180-45-n8cpu16-geo-0002 (us-east1-b)

**Other Failures:**
(decommissionbench.go:727).runDecommissionBench: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/decommissionBench/nodes=6/warehouses=1000/drain-first/while-upreplicating/target=2/multi-region/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test decommissionBench/nodes=8/warehouses=3000 failed: VMs preempted during the test run: teamcity-16558678-1724219180-54-n9cpu16-0009 (us-east1-b), teamcity-16558678-1724219180-54-n9cpu16-0006 (us-east1-c)

**Other Failures:**
(decommissionbench.go:727).runDecommissionBench: monitor failure: context canceled
failed to list cockroach processes: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/decommissionBench/nodes=8/warehouses=3000/run_1/ssh/ssh_082449.724443643_n6_list-processes.log): TRANSIENT_ERROR(ssh_problem): exit status 255 [owner=test-eng]
test artifacts and logs in: /artifacts/decommissionBench/nodes=8/warehouses=3000/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test import/tpcc/warehouses=1000/nodes=4 failed: VMs preempted during the test run: teamcity-16558678-1724219180-70-n4cpu4-0003 (us-east1-c), teamcity-16558678-1724219180-70-n4cpu4-0004 (us-east1-c)

**Other Failures:**
(cluster.go:2432).Run: context canceled
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/import/tpcc/warehouses=1000/nodes=4/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=true
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test failover/partial/lease-gateway failed: VMs preempted during the test run: teamcity-16558678-1724219180-61-n8cpu2-0001 (us-east1-c), teamcity-16558678-1724219180-61-n8cpu2-0003 (us-east1-c), teamcity-16558678-1724219180-61-n8cpu2-0002 (us-east1-c)

**Other Failures:**
(failover.go:1779).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
(cluster.go:2432).Run: context canceled
test artifacts and logs in: /artifacts/failover/partial/lease-gateway/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test import/tpch/nodes=8 failed: VMs preempted during the test run: teamcity-16558678-1724219180-72-n8cpu4-0003 (us-east1-c), teamcity-16558678-1724219180-72-n8cpu4-0002 (us-east1-c), teamcity-16558678-1724219180-72-n8cpu4-0005 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/import/tpch/nodes=8/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test ycsb/A/nodes=3/cpu=32/mvcc-range-keys=global failed: VMs preempted during the test run: teamcity-16558678-1724219180-127-n4cpu32-0004 (us-east1-b)

**Other Failures:**
(cluster.go:2432).Run: full command output in run_110641.853686188_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/ycsb/A/nodes=3/cpu=32/mvcc-range-keys=global/run_1/ssh/ssh_110641.880912919_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/A/nodes=3/cpu=32/mvcc-range-keys=global/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test tpccbench/nodes=9/cpu=4/chaos/partition failed: VMs preempted during the test run: teamcity-16558678-1724219180-120-n10cpu4-0008 (us-east1-b), teamcity-16558678-1724219180-120-n10cpu4-0005 (us-east1-b), teamcity-16558678-1724219180-120-n10cpu4-0002 (us-east1-b)

**Other Failures:**
(cluster.go:2369).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/run_1/ssh/ssh_111730.632968904_n2_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test zfs/ycsb/A/nodes=3/cpu=32 failed: VMs preempted during the test run: teamcity-16558678-1724219180-131-n4cpu32-0002 (us-east1-b)

**Other Failures:**
(cluster.go:2432).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/zfs/ycsb/A/nodes=3/cpu=32/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_encrypted=false
ROACHTEST_fs=zfs
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test tpccbench/nodes=3/cpu=16/lease=expiration failed: VMs preempted during the test run: teamcity-16558678-1724219180-118-n4cpu16-0002 (us-east1-d)

**Other Failures:**
(cluster.go:2318).Start: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=16/lease=expiration/run_1/ssh/ssh_111125.110925366_n2_upload-start-script.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=16/lease=expiration/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test kv50/enc=false/nodes=1/conc=8192 failed: VMs preempted during the test run: teamcity-16573782-1724305556-06-n2cpu8-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2432).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv50/enc=false/nodes=1/conc=8192/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

                      |   |     github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:173
                      |   | runtime.goexit
                      |   |     src/runtime/asm_amd64.s:1695
                      | Wraps: (3) _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/access=skew/run_1/ssh/ssh_061256.956770587_n8_cockroach-workload-r.log)
                      | Wraps: (4) TRANSIENT_ERROR(ssh_problem)
                      | Wraps: (5) exit status 255
                      | Error types: (1) *hintdetail.withDetail (2) *withstack.withStack (3) *errutil.withPrefix (4) errors.TransientError (5) *exec.ExitError
                                    └─ Wraps: (11) secondary error attachment
                      | _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/access=skew/run_1/ssh/ssh_061230.201959611_n8_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
                      | (1) Node 8. Command with error:
                      |   | ``````
                      |   | ./cockroach workload run kv --db=smallreads2 --read-percent=95 --min-block-bytes=1 --max-block-bytes=1 --max-rate=16500 --zipfian --tolerate-errors --concurrency=512 --duration=30m0s {pgurl:1-7}
                      |   | ``````
                      |   | <no output>
                      | Wraps: (2) attached stack trace
                      |   -- stack trace:
                      |   | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).errWithDebug
                      |   |     github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:143
                      |   | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).Run.func1
                      |   |     github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:173
                      |   | runtime.goexit
                      |   |     src/runtime/asm_amd64.s:1695
                      | Wraps: (3) _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/access=skew/run_1/ssh/ssh_061230.201959611_n8_cockroach-workload-r.log)
                      | Wraps: (4) TRANSIENT_ERROR(ssh_problem)
                      | Wraps: (5) exit status 255
                      | Error types: (1) *hintdetail.withDetail (2) *withstack.withStack (3) *errutil.withPrefix (4) errors.TransientError (5) *exec.ExitError
                                      └─ Wraps: (12) Node 8. Command with error:
                      | ``````
                      | ./cockroach workload run kv --db=smallreads2 --read-percent=95 --min-block-bytes=1 --max-block-bytes=1 --max-rate=16500 --zipfian --tolerate-errors --concurrency=512 --duration=30m0s {pgurl:1-7}
                      | ``````
                      | <truncated> ...  4270 workload/pgx_helpers.go:240  [-] 4036  error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
                      | W240822 05:51:02.115283 4221 workload/pgx_helpers.go:240  [-] 4037  error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
                      | W240822 05:51:02.122500 4179 workload/pgx_helpers.go:240  [-] 4038  error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
                      | W240822 05:51:02.130638 4238 workload/pgx_helpers.go:240  [-] 4039  error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
                      | Connection to 34.75.223.217 closed by remote host.
                                        └─ Wraps: (13) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).errWithDebug
                      |     github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:143
                      | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).Run.func1
                      |     github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:173
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                                          └─ Wraps: (14) _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/access=skew/run_1/ssh/ssh_055057.925027986_n8_cockroach-workload-r.log)
                                            └─ Wraps: (15) TRANSIENT_ERROR(ssh_problem)
                                              └─ Wraps: (16) exit status 255
                    Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *join.joinError (5) *withstack.withStack (6) *errutil.withPrefix (7) *markers.withMark (8) *withstack.withStack (9) *errutil.withPrefix (10) *secondary.withSecondaryError (11) *secondary.withSecondaryError (12) *hintdetail.withDetail (13) *withstack.withStack (14) *errutil.withPrefix (15) errors.TransientError (16) *exec.ExitError
    Test:           allocbench/nodes=7/cpu=8/kv/r=95/access=skew
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/access=skew/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

                      |   |     github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:173
                      |   | runtime.goexit
                      |   |     src/runtime/asm_amd64.s:1695
                      | Wraps: (3) _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1/ssh/ssh_061256.195308814_n8_cockroach-workload-r.log)
                      | Wraps: (4) TRANSIENT_ERROR(ssh_problem)
                      | Wraps: (5) exit status 255
                      | Error types: (1) *hintdetail.withDetail (2) *withstack.withStack (3) *errutil.withPrefix (4) errors.TransientError (5) *exec.ExitError
                                    └─ Wraps: (11) secondary error attachment
                      | _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1/ssh/ssh_061229.880642216_n8_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
                      | (1) Node 8. Command with error:
                      |   | ``````
                      |   | ./cockroach workload run kv --db=smallreads --read-percent=95 --min-block-bytes=1 --max-block-bytes=1 --max-rate=16500 --tolerate-errors --concurrency=512 --duration=30m0s {pgurl:1-7}
                      |   | ``````
                      |   | <no output>
                      | Wraps: (2) attached stack trace
                      |   -- stack trace:
                      |   | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).errWithDebug
                      |   |     github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:143
                      |   | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).Run.func1
                      |   |     github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:173
                      |   | runtime.goexit
                      |   |     src/runtime/asm_amd64.s:1695
                      | Wraps: (3) _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1/ssh/ssh_061229.880642216_n8_cockroach-workload-r.log)
                      | Wraps: (4) TRANSIENT_ERROR(ssh_problem)
                      | Wraps: (5) exit status 255
                      | Error types: (1) *hintdetail.withDetail (2) *withstack.withStack (3) *errutil.withPrefix (4) errors.TransientError (5) *exec.ExitError
                                      └─ Wraps: (12) Node 8. Command with error:
                      | ``````
                      | ./cockroach workload run kv --db=smallreads --read-percent=95 --min-block-bytes=1 --max-block-bytes=1 --max-rate=16500 --tolerate-errors --concurrency=512 --duration=30m0s {pgurl:1-7}
                      | ``````
                      | <truncated> ... 3919 workload/pgx_helpers.go:240  [-] 5074  error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
                      | W240822 05:52:57.752473 3736 workload/pgx_helpers.go:240  [-] 5075  error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
                      | W240822 05:52:57.757861 3926 workload/pgx_helpers.go:240  [-] 5076  error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
                      | W240822 05:52:57.758030 3991 workload/pgx_helpers.go:240  [-] 5077  error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
                      | Connection to 34.138.150.106 closed by remote host.
                                        └─ Wraps: (13) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).errWithDebug
                      |     github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:143
                      | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).Run.func1
                      |     github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:173
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                                          └─ Wraps: (14) _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1/ssh/ssh_055253.642392052_n8_cockroach-workload-r.log)
                                            └─ Wraps: (15) TRANSIENT_ERROR(ssh_problem)
                                              └─ Wraps: (16) exit status 255
                    Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *join.joinError (5) *withstack.withStack (6) *errutil.withPrefix (7) *markers.withMark (8) *withstack.withStack (9) *errutil.withPrefix (10) *secondary.withSecondaryError (11) *secondary.withSecondaryError (12) *hintdetail.withDetail (13) *withstack.withStack (14) *errutil.withPrefix (15) errors.TransientError (16) *exec.ExitError
    Test:           allocbench/nodes=7/cpu=8/kv/r=50/ops=skew
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test allocbench/nodes=7/cpu=8/kv/r=0/access=skew failed: VMs preempted during the test run: teamcity-16573782-1724305556-07-n8cpu8-0006 (us-east1-c), teamcity-16573782-1724305556-07-n8cpu8-0003 (us-east1-c), teamcity-16573782-1724305556-07-n8cpu8-0001 (us-east1-c), teamcity-16573782-1724305556-07-n8cpu8-0004 (us-east1-c)

**Other Failures:**
(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:389
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:339
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:268
                                main/pkg/cmd/roachtest/test_runner.go:1223
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    monitor failure: context canceled
                    (1) attached stack trace
                      -- stack trace:
                      | main.(*monitorImpl).WaitE
                      |     main/pkg/cmd/roachtest/monitor.go:142
                      | [...repeated from below...]
                    Wraps: (2) monitor failure
                    Wraps: (3) attached stack trace
                      -- stack trace:
                      | main.(*monitorImpl).wait
                      |     main/pkg/cmd/roachtest/monitor.go:237
                      | main.(*monitorImpl).WaitE
                      |     main/pkg/cmd/roachtest/monitor.go:142
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBenchSample
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:388
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBench
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:339
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAllocationBenchSpec.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:268
                      | main.(*testRunner).runTest.func2
                      |     main/pkg/cmd/roachtest/test_runner.go:1223
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (4) context canceled
                      | EOF [owner=test-eng]
                        └─ Wraps: (5) EOF [owner=test-eng]
                        └─ Wraps: (6) context canceled
                    Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *join.joinError (5) registry.ErrorWithOwnership (6) *errors.errorString
    Test:           allocbench/nodes=7/cpu=8/kv/r=0/access=skew
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/access=skew/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test schemachange/leasing-benchmark failed: VMs preempted during the test run: teamcity-16573782-1724305556-117-n3cpu4-geo-0002 (us-east1-b)

**Other Failures:**
(multiregion_leasing.go:190).func2: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/schemachange/leasing-benchmark/run_1/ssh/ssh_103810.826874915_n2_time-p-cockroach-sql.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/schemachange/leasing-benchmark/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test schemachange/indexschemachange/index/tpcc/w=800 failed: VMs preempted during the test run: teamcity-16573782-1724305556-116-n5cpu16-0005 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: full command output in run_103453.960738871_n5_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/schemachange/indexschemachange/index/tpcc/w=800/run_1/ssh/ssh_103454.012952001_n5_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/schemachange/indexschemachange/index/tpcc/w=800/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test tpccbench/nodes=3/cpu=4/mt-shared-process failed: VMs preempted during the test run: teamcity-16573782-1724305556-125-n4cpu4-0004 (us-east1-c), teamcity-16573782-1724305556-125-n4cpu4-0002 (us-east1-c)

**Other Failures:**
(cluster.go:2369).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=4/mt-shared-process/run_1/ssh/ssh_111224.393095489_n2_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=4/mt-shared-process/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test decommissionBench/nodes=6/warehouses=1000/drain-first/while-upreplicating/target=2/multi-region failed: VMs preempted during the test run: teamcity-16588908-1724392226-48-n8cpu16-geo-0007 (us-east1-b)

**Other Failures:**
(cluster.go:2318).Start: failed to find services to register: failed to find 1 open ports: TRANSIENT_ERROR(open_ports): output:: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/decommissionBench/nodes=6/warehouses=1000/drain-first/while-upreplicating/target=2/multi-region/run_1/ssh/ssh_075842.496977419_n7_find-ports.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(decommissionbench.go:727).runDecommissionBench: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/decommissionBench/nodes=6/warehouses=1000/drain-first/while-upreplicating/target=2/multi-region/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test kv95/enc=true/nodes=3 failed: VMs preempted during the test run: teamcity-16588908-1724392226-99-n4cpu8-0003 (us-east1-c)

**Other Failures:**
(cluster.go:2432).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv95/enc=true/nodes=3/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=true
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test loqrecovery/workload=tpcc/rangeSize=default failed: VMs preempted during the test run: teamcity-16588908-1724392226-102-n6cpu4-0001 (us-east1-b)

**Other Failures:**
(cluster.go:2432).Run: context canceled
test artifacts and logs in: /artifacts/loqrecovery/workload=tpcc/rangeSize=default/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test kv95/enc=false/nodes=3/cpu=96 failed: VMs preempted during the test run: teamcity-16588908-1724392226-96-n4cpu96-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2432).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/cpu=96/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ f629e9a3a88152ab73ee154eb5fe1e7c819e1af3:

test ycsb/F/nodes=3 failed: VMs preempted during the test run: teamcity-16588908-1724392226-143-n4cpu8-0003 (us-east1-c)

**Other Failures:**
(cluster.go:2432).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/ycsb/F/nodes=3/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ a1cbd782a33a51f7500444373ccdaa81eb2ed971:

test cdc/initial-scan failed: VMs preempted during the test run: teamcity-16600428-1724478520-27-n4cpu16-0003 (us-east1-b), teamcity-16600428-1724478520-27-n4cpu16-0001 (us-east1-b)

**Other Failures:**
(cluster.go:2432).Run: full command output in run_065243.871279250_n4_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
(cdc.go:378).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/initial-scan/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ a1cbd782a33a51f7500444373ccdaa81eb2ed971:

test ycsb/A/nodes=3/cpu=32/mvcc-range-keys=global failed: VMs preempted during the test run: teamcity-16600428-1724478520-142-n4cpu32-0004 (us-east1-d)

**Other Failures:**
(cluster.go:2432).Run: full command output in run_103834.841506401_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/ycsb/A/nodes=3/cpu=32/mvcc-range-keys=global/run_1/ssh/ssh_103834.868171302_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/A/nodes=3/cpu=32/mvcc-range-keys=global/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ a1cbd782a33a51f7500444373ccdaa81eb2ed971:

test scrub/index-only/tpcc/w=100 failed: VMs preempted during the test run: teamcity-16600428-1724478520-119-n5cpu4-0002 (us-east1-d), teamcity-16600428-1724478520-119-n5cpu4-0004 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/scrub/index-only/tpcc/w=100/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ a1cbd782a33a51f7500444373ccdaa81eb2ed971:

test allocbench/nodes=7/cpu=8/kv/r=0/access=skew failed: VMs preempted during the test run: teamcity-16608158-1724564705-10-n8cpu8-0005 (us-central1-a)

**Other Failures:**
(cluster.go:2424).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/access=skew/cpu_arch=arm64/run_1/ssh/ssh_062541.324685185_n5_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/access=skew/cpu_arch=arm64/run_1

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ a1cbd782a33a51f7500444373ccdaa81eb2ed971:

test cdc/cloud-sink-gcs/assume-role failed: VMs preempted during the test run: teamcity-16608158-1724564705-25-n4cpu16-0001 (us-central1-a)

**Other Failures:**
(cluster.go:2432).Run: full command output in run_070003.999254888_n4_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
(cdc.go:378).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/cloud-sink-gcs/assume-role/cpu_arch=arm64/run_1

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ a1cbd782a33a51f7500444373ccdaa81eb2ed971:

                      |   |     github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:173
                      |   | runtime.goexit
                      |   |     src/runtime/asm_amd64.s:1695
                      | Wraps: (3) _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1/ssh/ssh_081607.838338810_n8_cockroach-workload-r.log)
                      | Wraps: (4) TRANSIENT_ERROR(ssh_problem)
                      | Wraps: (5) exit status 255
                      | Error types: (1) *hintdetail.withDetail (2) *withstack.withStack (3) *errutil.withPrefix (4) errors.TransientError (5) *exec.ExitError
                                    └─ Wraps: (11) secondary error attachment
                      | _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1/ssh/ssh_081537.875130073_n8_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
                      | (1) Node 8. Command with error:
                      |   | ``````
                      |   | ./cockroach workload run kv --db=bigwrites --read-percent=0 --min-block-bytes=32768 --max-block-bytes=32768 --max-rate=550 --tolerate-errors --concurrency=512 --duration=30m0s {pgurl:1-7}
                      |   | ``````
                      |   | <no output>
                      | Wraps: (2) attached stack trace
                      |   -- stack trace:
                      |   | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).errWithDebug
                      |   |     github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:143
                      |   | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).Run.func1
                      |   |     github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:173
                      |   | runtime.goexit
                      |   |     src/runtime/asm_amd64.s:1695
                      | Wraps: (3) _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1/ssh/ssh_081537.875130073_n8_cockroach-workload-r.log)
                      | Wraps: (4) TRANSIENT_ERROR(ssh_problem)
                      | Wraps: (5) exit status 255
                      | Error types: (1) *hintdetail.withDetail (2) *withstack.withStack (3) *errutil.withPrefix (4) errors.TransientError (5) *exec.ExitError
                                      └─ Wraps: (12) Node 8. Command with error:
                      | ``````
                      | ./cockroach workload run kv --db=bigwrites --read-percent=0 --min-block-bytes=32768 --max-block-bytes=32768 --max-rate=550 --tolerate-errors --concurrency=512 --duration=30m0s {pgurl:1-7}
                      | ``````
                      | <truncated> ... 3876 workload/pgx_helpers.go:240  [-] 7167  error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
                      | W240825 07:56:08.673815 3598 workload/pgx_helpers.go:240  [-] 7168  error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
                      | W240825 07:56:08.674004 3959 workload/pgx_helpers.go:240  [-] 7169  error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
                      | W240825 07:56:08.696088 3647 workload/pgx_helpers.go:240  [-] 7170  error preparing statement. name=kv-2 sql=SELECT k, v FROM kv AS OF SYSTEM TIME follower_read_timestamp() WHERE k IN ($1) ERROR: relation "kv" is offline: importing (SQLSTATE XXUUU)
                      | Connection to 34.139.109.118 closed by remote host.
                                        └─ Wraps: (13) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).errWithDebug
                      |     github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:143
                      | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).Run.func1
                      |     github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:173
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                                          └─ Wraps: (14) _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1/ssh/ssh_075604.438307636_n8_cockroach-workload-r.log)
                                            └─ Wraps: (15) TRANSIENT_ERROR(ssh_problem)
                                              └─ Wraps: (16) exit status 255
                    Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *join.joinError (5) *withstack.withStack (6) *errutil.withPrefix (7) *markers.withMark (8) *withstack.withStack (9) *errutil.withPrefix (10) *secondary.withSecondaryError (11) *secondary.withSecondaryError (12) *hintdetail.withDetail (13) *withstack.withStack (14) *errutil.withPrefix (15) errors.TransientError (16) *exec.ExitError
    Test:           allocbench/nodes=7/cpu=8/kv/r=50/ops=skew
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ a1cbd782a33a51f7500444373ccdaa81eb2ed971:

test allocbench/nodes=7/cpu=8/kv/r=95/ops=skew failed: VMs preempted during the test run: teamcity-16608158-1724564705-13-n8cpu8-0008 (us-east1-b), teamcity-16608158-1724564705-13-n8cpu8-0007 (us-east1-c)

**Other Failures:**
(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:389
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:339
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:268
                                main/pkg/cmd/roachtest/test_runner.go:1223
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    monitor failure: context canceled
                    (1) attached stack trace
                      -- stack trace:
                      | main.(*monitorImpl).WaitE
                      |     main/pkg/cmd/roachtest/monitor.go:142
                      | [...repeated from below...]
                    Wraps: (2) monitor failure
                    Wraps: (3) attached stack trace
                      -- stack trace:
                      | main.(*monitorImpl).wait
                      |     main/pkg/cmd/roachtest/monitor.go:237
                      | main.(*monitorImpl).WaitE
                      |     main/pkg/cmd/roachtest/monitor.go:142
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBenchSample
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:388
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBench
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:339
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAllocationBenchSpec.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:268
                      | main.(*testRunner).runTest.func2
                      |     main/pkg/cmd/roachtest/test_runner.go:1223
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (4) context canceled
                      | EOF [owner=test-eng]
                        └─ Wraps: (5) EOF [owner=test-eng]
                        └─ Wraps: (6) context canceled
                    Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *join.joinError (5) registry.ErrorWithOwnership (6) *errors.errorString
    Test:           allocbench/nodes=7/cpu=8/kv/r=95/ops=skew
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/ops=skew/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ a1cbd782a33a51f7500444373ccdaa81eb2ed971:

test allocbench/nodes=7/cpu=8/kv/r=95/access=skew failed: VMs preempted during the test run: teamcity-16608158-1724564705-12-n8cpu8-0003 (us-east1-c)

**Other Failures:**
(cluster.go:2424).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/access=skew/run_1/ssh/ssh_082504.052069979_n3_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/access=skew/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ a1cbd782a33a51f7500444373ccdaa81eb2ed971:

test kv/contention/nodes=4 failed: VMs preempted during the test run: teamcity-16608158-1724564705-80-n5cpu4-0001 (us-east1-b)

**Other Failures:**
(cluster.go:2432).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv/contention/nodes=4/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ a1cbd782a33a51f7500444373ccdaa81eb2ed971:

test schemachange/indexschemachange/index/tpcc/w=800 failed: VMs preempted during the test run: teamcity-16608158-1724564705-108-n5cpu16-0004 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/schemachange/indexschemachange/index/tpcc/w=800/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ a1cbd782a33a51f7500444373ccdaa81eb2ed971:

test ycsb/A/nodes=3 failed: VMs preempted during the test run: teamcity-16608158-1724564705-126-n4cpu8-0002 (us-east1-c)

**Other Failures:**
(cluster.go:2432).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/ycsb/A/nodes=3/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ a1cbd782a33a51f7500444373ccdaa81eb2ed971:

test tpccbench/nodes=3/cpu=4/enc=true failed: VMs preempted during the test run: teamcity-16608158-1724564705-123-n4cpu4-0001 (us-central1-a)

**Other Failures:**
(cluster.go:2369).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=4/enc=true/cpu_arch=arm64/run_1/ssh/ssh_105215.925142925_n1_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=4/enc=true/cpu_arch=arm64/run_1

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=true
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ a1cbd782a33a51f7500444373ccdaa81eb2ed971:

test tpccbench/nodes=3/cpu=16/lease=expiration failed: VMs preempted during the test run: teamcity-16608158-1724564705-122-n4cpu16-0003 (us-east1-c), teamcity-16608158-1724564705-122-n4cpu16-0002 (us-east1-c)

**Other Failures:**
(cluster.go:2369).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=16/lease=expiration/run_1/ssh/ssh_105526.399664961_n2_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=16/lease=expiration/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ a1cbd782a33a51f7500444373ccdaa81eb2ed971:

test tpccbench/nodes=3/cpu=16/enc=true failed: VMs preempted during the test run: teamcity-16608158-1724564705-121-n4cpu16-0004 (us-east1-c), teamcity-16608158-1724564705-121-n4cpu16-0002 (us-east1-c)

**Other Failures:**
(cluster.go:2369).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=16/enc=true/run_1/ssh/ssh_111138.508828521_n2_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=16/enc=true/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=true
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ a1cbd782a33a51f7500444373ccdaa81eb2ed971:

test ycsb/C/nodes=3/cpu=32 failed: VMs preempted during the test run: teamcity-16608158-1724564705-132-n4cpu32-0004 (us-east1-c)

**Other Failures:**
(cluster.go:2432).Run: full command output in run_112333.575741020_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/ycsb/C/nodes=3/cpu=32/run_1/ssh/ssh_112333.599884747_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/C/nodes=3/cpu=32/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ a1cbd782a33a51f7500444373ccdaa81eb2ed971:

test scrub/index-only/tpcc/w=100 failed: VMs preempted during the test run: teamcity-16608158-1724564705-110-n5cpu4-0002 (us-central1-a)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/scrub/index-only/tpcc/w=100/cpu_arch=arm64/run_1

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ a1cbd782a33a51f7500444373ccdaa81eb2ed971:

test backup/2TB/n10cpu4 failed: VMs preempted during the test run: teamcity-16614829-1724651108-20-n10cpu4-0004 (us-east1-d)

**Other Failures:**
(cluster.go:2432).Run: full command output in run_062445.903536711_n1_cockroach-workload-f.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/backup/2TB/n10cpu4/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ a1cbd782a33a51f7500444373ccdaa81eb2ed971:

test cdc/workload/kv0/nodes=5/cpu=16/ranges=100/server=scheduler/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-16614829-1724651108-36-n7cpu16-0004 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100/server=scheduler/protocol=mux/format=json/sink=null/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ a1cbd782a33a51f7500444373ccdaa81eb2ed971:

test kv0/enc=false/nodes=3/batch=16 failed: VMs preempted during the test run: teamcity-16614829-1724651108-97-n4cpu8-0003 (us-central1-a)

**Other Failures:**
(cluster.go:2432).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/batch=16/cpu_arch=arm64/run_1

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ a1cbd782a33a51f7500444373ccdaa81eb2ed971:

test kv0/enc=false/nodes=3/cpu=32/mvcc-range-keys=global failed: VMs preempted during the test run: teamcity-16614829-1724651108-94-n4cpu32-0002 (us-east1-d)

**Other Failures:**
(cluster.go:2432).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/cpu=32/mvcc-range-keys=global/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ a1cbd782a33a51f7500444373ccdaa81eb2ed971:

test kv0/enc=false/nodes=3/size=64kb failed: VMs preempted during the test run: teamcity-16614829-1724651108-95-n4cpu8-0002 (us-east1-d)

**Other Failures:**
(cluster.go:2432).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/size=64kb/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ a1cbd782a33a51f7500444373ccdaa81eb2ed971:

                      |   -- stack trace:
                      |   | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).errWithDebug
                      |   |     github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:143
                      |   | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).Run.func1
                      |   |     github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:173
                      |   | runtime.goexit
                      |   |     src/runtime/asm_amd64.s:1695
                      | Wraps: (3) _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/replicate/rebalance/3to5/run_1/ssh/ssh_111021.992730812_n6_sudo-systemctl-stop-.log)
                      | Wraps: (4) TRANSIENT_ERROR(ssh_problem)
                      | Wraps: (5) exit status 255
                      | Error types: (1) *hintdetail.withDetail (2) *withstack.withStack (3) *errutil.withPrefix (4) errors.TransientError (5) *exec.ExitError
                    Wraps: (5) secondary error attachment
                      | _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/replicate/rebalance/3to5/run_1/ssh/ssh_110959.799486877_n6_sudo-systemctl-stop-.log): TRANSIENT_ERROR(ssh_problem): exit status 255
                      | (1) Node 6. Command with error:
                      |   | ``````
                      |   | sudo systemctl stop prometheus || echo 'no prometheus is running'
                      |   | ``````
                      |   | <no output>
                      | Wraps: (2) attached stack trace
                      |   -- stack trace:
                      |   | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).errWithDebug
                      |   |     github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:143
                      |   | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).Run.func1
                      |   |     github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:173
                      |   | runtime.goexit
                      |   |     src/runtime/asm_amd64.s:1695
                      | Wraps: (3) _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/replicate/rebalance/3to5/run_1/ssh/ssh_110959.799486877_n6_sudo-systemctl-stop-.log)
                      | Wraps: (4) TRANSIENT_ERROR(ssh_problem)
                      | Wraps: (5) exit status 255
                      | Error types: (1) *hintdetail.withDetail (2) *withstack.withStack (3) *errutil.withPrefix (4) errors.TransientError (5) *exec.ExitError
                    Wraps: (6) Node 6. Command with error:
                      | ``````
                      | sudo systemctl stop prometheus || echo 'no prometheus is running'
                      | ``````
                      | <no output>
                    Wraps: (7) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).errWithDebug
                      |     github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:143
                      | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).Run.func1
                      |     github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:173
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (8) _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/replicate/rebalance/3to5/run_1/ssh/ssh_110954.747313282_n6_sudo-systemctl-stop-.log)
                    Wraps: (9) TRANSIENT_ERROR(ssh_problem)
                    Wraps: (10) exit status 255
                    Error types: (1) *markers.withMark (2) *withstack.withStack (3) *errutil.withPrefix (4) *secondary.withSecondaryError (5) *secondary.withSecondaryError (6) *hintdetail.withDetail (7) *withstack.withStack (8) *errutil.withPrefix (9) errors.TransientError (10) *exec.ExitError
    Test:           replicate/rebalance/3to5
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/replicate/rebalance/3to5/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ a1cbd782a33a51f7500444373ccdaa81eb2ed971:

test zfs/ycsb/A/nodes=3/cpu=8 failed: VMs preempted during the test run: teamcity-16614829-1724651108-140-n4cpu8-0001 (us-east1-c), teamcity-16614829-1724651108-140-n4cpu8-0003 (us-east1-c)

**Other Failures:**
(cluster.go:2432).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/zfs/ycsb/A/nodes=3/cpu=8/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=zfs
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128143 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ a1cbd782a33a51f7500444373ccdaa81eb2ed971:

test connection_latency/nodes=9/multiregion/password failed: VMs preempted during the test run: teamcity-16628553-1724737597-34-n12cpu4-geo-0009 (europe-west2-b), teamcity-16628553-1724737597-34-n12cpu4-geo-0006 (us-west1-b)

**Other Failures:**
(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/connection_latency.go:85
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/connection_latency.go:96
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/connection_latency.go:149
                                main/pkg/cmd/roachtest/test_runner.go:1223
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    full command output in run_070958.030951865_n8_workload-run-connect.log: COMMAND_PROBLEM: exit status 1
                    (1) attached stack trace
                      -- stack trace:
                      | main.(*clusterImpl).RunE
                      |     main/pkg/cmd/roachtest/cluster.go:2469
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runConnectionLatencyTest.func2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/connection_latency.go:84
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runConnectionLatencyTest
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/connection_latency.go:96
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerConnectionLatencyTest.func3
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/connection_latency.go:149
                      | main.(*testRunner).runTest.func2
                      |     main/pkg/cmd/roachtest/test_runner.go:1223
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) full command output in run_070958.030951865_n8_workload-run-connect.log
                    Wraps: (3) Node 8. Command with error:
                      | ``````
                      | ./workload run connectionlatency 'postgres://roachprod:cockroachdb@35.247.123.243:{pgport:1}?sslmode=require&sslrootcert=certs/ca.crt' 'postgres://roachprod:cockroachdb@35.227.159.203:{pgport:1}?sslmode=require&sslrootcert=certs/ca.crt' 'postgres://roachprod:cockroachdb@34.105.114.155:{pgport:1}?sslmode=require&sslrootcert=certs/ca.crt' --secure --duration 30s --histograms=perf/stats.json --locality us-west1-b
                      | ``````
                      | stdout: <empty>
                      | stderr:I240827 07:09:59.285136 1 workload/cli/run.go:431  [-] 1  creating load generator...
                      | Error: failed to initialize the load generator: failed to connect to ``host=35.227.159.203 user=roachprod database=connectionlatency``: server error (ERROR: server is not accepting clients, try another node (SQLSTATE 57P01))
                    Wraps: (4) COMMAND_PROBLEM
                    Wraps: (5) exit status 1
                    Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *hintdetail.withDetail (4) errors.Cmd (5) *exec.ExitError
    Test:           connection_latency/nodes=9/multiregion/password
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/connection_latency/nodes=9/multiregion/password/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ a1cbd782a33a51f7500444373ccdaa81eb2ed971:

test kv/contention/nodes=4 failed: VMs preempted during the test run: teamcity-16628553-1724737597-77-n5cpu4-0002 (us-central1-a)

**Other Failures:**
(cluster.go:2432).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv/contention/nodes=4/cpu_arch=arm64/run_1

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.4-rc @ a1cbd782a33a51f7500444373ccdaa81eb2ed971:

test tpccbench/nodes=3/cpu=4/mt-shared-process failed: VMs preempted during the test run: teamcity-16628553-1724737597-124-n4cpu4-0001 (us-east1-d)

**Other Failures:**
(cluster.go:2318).Start: one or more parallel execution failure(s): error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=4/mt-shared-process/run_1/ssh/ssh_103913.695230602_n1_init-certs.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=4/mt-shared-process/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #129304 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroachdb / cockroach

roachtest: vm_preemption failed #128948