roachtest: vm_preemption failed

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test allocbench/nodes=7/cpu=8/kv/r=95/access=skew failed: VMs preempted during the test run: teamcity-16298889-1722577605-11-n8cpu8-0002 (us-east1-c), teamcity-16298889-1722577605-11-n8cpu8-0007 (us-east1-c)

**Other Failures:**
(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:389
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:339
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:268
                                main/pkg/cmd/roachtest/test_runner.go:1248
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    monitor failure: context canceled
                    (1) attached stack trace
                      -- stack trace:
                      | main.(*monitorImpl).WaitE
                      |     main/pkg/cmd/roachtest/monitor.go:142
                      | [...repeated from below...]
                    Wraps: (2) monitor failure
                    Wraps: (3) attached stack trace
                      -- stack trace:
                      | main.(*monitorImpl).wait
                      |     main/pkg/cmd/roachtest/monitor.go:237
                      | main.(*monitorImpl).WaitE
                      |     main/pkg/cmd/roachtest/monitor.go:142
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBenchSample
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:388
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBench
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:339
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAllocationBenchSpec.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:268
                      | main.(*testRunner).runTest.func2
                      |     main/pkg/cmd/roachtest/test_runner.go:1248
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (4) context canceled
                      | EOF [owner=test-eng]
                        └─ Wraps: (5) EOF [owner=test-eng]
                        └─ Wraps: (6) context canceled
                    Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *join.joinError (5) registry.ErrorWithOwnership (6) *errors.errorString
    Test:           allocbench/nodes=7/cpu=8/kv/r=95/access=skew
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/access=skew/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-40816

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test cdc/scan/catchup/nodes=5/cpu=16/rows=1G/ranges=100/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-16298889-1722577605-47-n6cpu16-0004 (us-central1-a), teamcity-16298889-1722577605-47-n6cpu16-0006 (us-central1-a), teamcity-16298889-1722577605-47-n6cpu16-0005 (us-central1-a)

**Other Failures:**
(cluster.go:2419).Run: full command output in run_065013.378744154_n6_cockroach-workload-i.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/scan/catchup/nodes=5/cpu=16/rows=1G/ranges=100/protocol=mux/format=json/sink=null/cpu_arch=arm64/run_1/ssh/ssh_065013.431208758_n6_cockroach-workload-i.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/scan/catchup/nodes=5/cpu=16/rows=1G/ranges=100/protocol=mux/format=json/sink=null/cpu_arch=arm64/run_1

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test cdc/webhook-sink failed: VMs preempted during the test run: teamcity-16298889-1722577605-50-n4cpu16-0004 (us-east1-c)

**Other Failures:**
(cluster.go:2419).Run: full command output in run_065605.625598470_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/webhook-sink/run_1/ssh/ssh_065605.701354233_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:379).Close: error shutting down prometheus/grafana: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/webhook-sink/run_1/ssh/ssh_070838.612319899_n4_sudo-rm-rf-tmppromet.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/webhook-sink/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-16298889-1722577605-48-n6cpu16-0002 (us-east1-b), teamcity-16298889-1722577605-48-n6cpu16-0005 (us-east1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100/protocol=mux/format=json/sink=null/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test c2c/tpcc/warehouses=1000/duration=60/cutover=30 failed: VMs preempted during the test run: teamcity-16298889-1722577605-38-n9cpu8-0001 (us-east1-b), teamcity-16298889-1722577605-38-n9cpu8-0006 (us-east1-b), teamcity-16298889-1722577605-38-n9cpu8-0007 (us-east1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/c2c/tpcc/warehouses=1000/duration=60/cutover=30/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/control failed: VMs preempted during the test run: teamcity-16298889-1722577605-53-n7cpu16-0006 (us-east1-c), teamcity-16298889-1722577605-53-n7cpu16-0001 (us-east1-c), teamcity-16298889-1722577605-53-n7cpu16-0003 (us-east1-c), teamcity-16298889-1722577605-53-n7cpu16-0002 (us-east1-c)

**Other Failures:**
(cluster.go:2419).Run: full command output in run_071545.915415759_n7_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/control/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/server=processor/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-16298889-1722577605-54-n7cpu16-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2419).Run: full command output in run_071806.252826565_n7_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/server=processor/protocol=mux/format=json/sink=null/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test cdc/tpcc-1000/sink=kafka failed: VMs preempted during the test run: teamcity-16298889-1722577605-49-n4cpu16-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2419).Run: full command output in run_065921.059190039_n4_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
(cdc.go:379).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/tpcc-1000/sink=kafka/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/server=processor/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-16298889-1722577605-51-n7cpu16-0006 (us-east1-d), teamcity-16298889-1722577605-51-n7cpu16-0007 (us-east1-d), teamcity-16298889-1722577605-51-n7cpu16-0002 (us-east1-d), teamcity-16298889-1722577605-51-n7cpu16-0004 (us-east1-d), teamcity-16298889-1722577605-51-n7cpu16-0003 (us-east1-d), teamcity-16298889-1722577605-51-n7cpu16-0001 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/server=processor/protocol=mux/format=json/sink=null/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test failover/non-system/blackhole-send/lease=expiration failed: VMs preempted during the test run: teamcity-16298889-1722577605-81-n7cpu2-0007 (us-east1-d)

**Other Failures:**
(cluster.go:2305).Start: one or more parallel execution failure(s): error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/failover/non-system/blackhole-send/lease=expiration/run_1/ssh/ssh_075250.136130030_n7_dist-local-certs.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cluster.go:2419).Run: context canceled
test artifacts and logs in: /artifacts/failover/non-system/blackhole-send/lease=expiration/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test decommissionBench/nodes=4/warehouses=1000 failed: VMs preempted during the test run: teamcity-16298889-1722577605-70-n5cpu16-0001 (us-east1-d), teamcity-16298889-1722577605-70-n5cpu16-0003 (us-east1-d)

**Other Failures:**
(decommissionbench.go:727).runDecommissionBench: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/decommissionBench/nodes=4/warehouses=1000/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test failover/partial/lease-leader/lease=expiration failed: VMs preempted during the test run: teamcity-16298889-1722577605-84-n7cpu2-0006 (us-east1-d), teamcity-16298889-1722577605-84-n7cpu2-0002 (us-east1-d), teamcity-16298889-1722577605-84-n7cpu2-0003 (us-east1-d)

**Other Failures:**
(failover.go:1774).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
(cluster.go:2419).Run: context canceled
test artifacts and logs in: /artifacts/failover/partial/lease-leader/lease=expiration/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test indexes/2/nodes=6/multi-region failed: VMs preempted during the test run: teamcity-16298889-1722577605-96-n7cpu16-geo-0001 (us-east1-b)

**Other Failures:**
(cluster.go:2419).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/indexes/2/nodes=6/multi-region/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test tpccbench/nodes=3/cpu=4/backups=true failed: VMs preempted during the test run: teamcity-16298889-1722577605-94-n4cpu4-0003 (us-east1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=4/backups=true/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test ycsb/A/nodes=3/cpu=32/mvcc-range-keys=global failed: VMs preempted during the test run: teamcity-16298889-1722577605-121-n4cpu32-0004 (us-east1-b)

**Other Failures:**
(cluster.go:2419).Run: full command output in run_094505.188911833_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/ycsb/A/nodes=3/cpu=32/mvcc-range-keys=global/run_1/ssh/ssh_094505.215453992_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/A/nodes=3/cpu=32/mvcc-range-keys=global/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_encrypted=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test scrub/index-only/tpcc/w=100 failed: VMs preempted during the test run: teamcity-16298889-1722577605-156-n5cpu4-0004 (us-east1-d), teamcity-16298889-1722577605-156-n5cpu4-0002 (us-east1-d), teamcity-16298889-1722577605-156-n5cpu4-0001 (us-east1-d)

**Other Failures:**
(scrub.go:91).1: pq: server is not accepting clients, try another node
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/scrub/index-only/tpcc/w=100/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test ycsb/E/nodes=3/cpu=32/mvcc-range-keys=global failed: VMs preempted during the test run: teamcity-16298889-1722577605-188-n4cpu32-0004 (us-east1-d)

**Other Failures:**
(cluster.go:2419).Run: full command output in run_105451.275709714_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/ycsb/E/nodes=3/cpu=32/mvcc-range-keys=global/run_1/ssh/ssh_105451.297317583_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/E/nodes=3/cpu=32/mvcc-range-keys=global/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test ycsb/C/nodes=3/cpu=32/uniform failed: VMs preempted during the test run: teamcity-16298889-1722577605-181-n4cpu32-0001 (us-east1-d)

**Other Failures:**
(cluster.go:2419).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/ycsb/C/nodes=3/cpu=32/uniform/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test allocbench/nodes=7/cpu=8/kv/r=0/ops=skew failed: VMs preempted during the test run: teamcity-16313481-1722663992-13-n8cpu8-0005 (us-east1-d), teamcity-16313481-1722663992-13-n8cpu8-0007 (us-east1-d), teamcity-16313481-1722663992-13-n8cpu8-0006 (us-east1-d), teamcity-16313481-1722663992-13-n8cpu8-0008 (us-east1-d), teamcity-16313481-1722663992-13-n8cpu8-0003 (us-east1-d)

**Other Failures:**
(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:389
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:339
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:268
                                main/pkg/cmd/roachtest/test_runner.go:1248
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    monitor failure: context canceled
                    (1) attached stack trace
                      -- stack trace:
                      | main.(*monitorImpl).WaitE
                      |     main/pkg/cmd/roachtest/monitor.go:142
                      | [...repeated from below...]
                    Wraps: (2) monitor failure
                    Wraps: (3) attached stack trace
                      -- stack trace:
                      | main.(*monitorImpl).wait
                      |     main/pkg/cmd/roachtest/monitor.go:237
                      | main.(*monitorImpl).WaitE
                      |     main/pkg/cmd/roachtest/monitor.go:142
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBenchSample
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:388
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBench
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:339
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAllocationBenchSpec.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:268
                      | main.(*testRunner).runTest.func2
                      |     main/pkg/cmd/roachtest/test_runner.go:1248
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (4) context canceled
                      | EOF [owner=test-eng]
                        └─ Wraps: (5) EOF [owner=test-eng]
                        └─ Wraps: (6) context canceled
                    Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *join.joinError (5) registry.ErrorWithOwnership (6) *errors.errorString
    Test:           allocbench/nodes=7/cpu=8/kv/r=0/ops=skew
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/ops=skew/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/control failed: VMs preempted during the test run: teamcity-16313481-1722663992-41-n7cpu16-0006 (us-east1-d), teamcity-16313481-1722663992-41-n7cpu16-0001 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/control/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test kv0/enc=false/nodes=3/cpu=32 failed: VMs preempted during the test run: teamcity-16313481-1722663992-96-n4cpu32-0002 (us-east1-d), teamcity-16313481-1722663992-96-n4cpu32-0003 (us-east1-d)

**Other Failures:**
(cluster.go:2419).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/cpu=32/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test cdc/crdb-chaos failed: VMs preempted during the test run: teamcity-16313481-1722663992-29-n4cpu16-0002 (us-east1-d), teamcity-16313481-1722663992-29-n4cpu16-0003 (us-east1-d)

**Other Failures:**
(test_runner.go:1277).runTest: test timed out (3h0m0s)
test artifacts and logs in: /artifacts/cdc/crdb-chaos/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test zfs/ycsb/A/nodes=3/cpu=8 failed: VMs preempted during the test run: teamcity-16313481-1722663992-142-n4cpu8-0001 (us-east1-d)

**Other Failures:**
(cluster.go:2419).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/zfs/ycsb/A/nodes=3/cpu=8/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=zfs
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test cdc/webhook-sink failed: VMs preempted during the test run: teamcity-16321257-1722750381-29-n4cpu16-0003 (us-east1-d), teamcity-16321257-1722750381-29-n4cpu16-0001 (us-east1-d)

**Other Failures:**
(cluster.go:2419).Run: full command output in run_065856.751076977_n4_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
(cdc.go:379).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/webhook-sink/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test allocbench/nodes=7/cpu=8/kv/r=95/access=skew failed: VMs preempted during the test run: teamcity-16321257-1722750381-12-n8cpu8-0007 (us-east1-d)

**Other Failures:**
(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:389
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:339
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:268
                                main/pkg/cmd/roachtest/test_runner.go:1248
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    monitor failure: context canceled
                    (1) attached stack trace
                      -- stack trace:
                      | main.(*monitorImpl).WaitE
                      |     main/pkg/cmd/roachtest/monitor.go:142
                      | [...repeated from below...]
                    Wraps: (2) monitor failure
                    Wraps: (3) attached stack trace
                      -- stack trace:
                      | main.(*monitorImpl).wait
                      |     main/pkg/cmd/roachtest/monitor.go:237
                      | main.(*monitorImpl).WaitE
                      |     main/pkg/cmd/roachtest/monitor.go:142
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBenchSample
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:388
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBench
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:339
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAllocationBenchSpec.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:268
                      | main.(*testRunner).runTest.func2
                      |     main/pkg/cmd/roachtest/test_runner.go:1248
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (4) context canceled
                      | EOF [owner=test-eng]
                        └─ Wraps: (5) EOF [owner=test-eng]
                        └─ Wraps: (6) context canceled
                    Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *join.joinError (5) registry.ErrorWithOwnership (6) *errors.errorString
    Test:           allocbench/nodes=7/cpu=8/kv/r=95/access=skew
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/access=skew/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test tpcc-nowait/nodes=3/w=1 failed: VMs preempted during the test run: teamcity-16328439-1722836789-30-n4cpu16-0003 (us-central1-a)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpcc-nowait/nodes=3/w=1/cpu_arch=arm64/run_1

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/server=processor/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-16328439-1722836789-36-n7cpu16-0005 (us-east1-d), teamcity-16328439-1722836789-36-n7cpu16-0006 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/server=processor/protocol=mux/format=json/sink=null/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test failover/chaos/read-write failed: VMs preempted during the test run: teamcity-16328439-1722836789-61-n10cpu2-0006 (us-east1-c)

**Other Failures:**
(failover.go:1774).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
(cluster.go:2419).Run: context canceled
(cluster.go:2419).Run: context canceled
(cluster.go:2419).Run: context canceled
(cluster.go:2419).Run: context canceled
test artifacts and logs in: /artifacts/failover/chaos/read-write/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test ycsb/A/nodes=3 failed: VMs preempted during the test run: teamcity-16328439-1722836789-97-n4cpu8-0002 (us-east1-d), teamcity-16328439-1722836789-97-n4cpu8-0001 (us-east1-d)

**Other Failures:**
(cluster.go:2419).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/ycsb/A/nodes=3/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test ycsb/C/nodes=3/cpu=32/uniform failed: VMs preempted during the test run: teamcity-16328439-1722836789-94-n4cpu32-0002 (us-east1-c), teamcity-16328439-1722836789-94-n4cpu32-0003 (us-east1-c)

**Other Failures:**
(cluster.go:2419).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/ycsb/C/nodes=3/cpu=32/uniform/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_encrypted=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test ycsb/F/nodes=3/cpu=32/mvcc-range-keys=global failed: VMs preempted during the test run: teamcity-16328439-1722836789-134-n4cpu32-0004 (us-east1-d), teamcity-16328439-1722836789-134-n4cpu32-0001 (us-east1-d)

**Other Failures:**
(cluster.go:2419).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/ycsb/F/nodes=3/cpu=32/mvcc-range-keys=global/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test zfs/ycsb/A/nodes=3/cpu=8 failed: VMs preempted during the test run: teamcity-16328439-1722836789-136-n4cpu8-0001 (us-east1-d)

**Other Failures:**
(cluster.go:2419).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/zfs/ycsb/A/nodes=3/cpu=8/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=zfs
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-16343484-1722923371-33-n6cpu16-0003 (us-east1-c), teamcity-16343484-1722923371-33-n6cpu16-0004 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100/protocol=mux/format=json/sink=null/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test failover/partial/lease-gateway/lease=expiration failed: VMs preempted during the test run: teamcity-16343484-1722923371-68-n8cpu2-0008 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: full command output in run_084306.885060362_n8_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/failover/partial/lease-gateway/lease=expiration/run_1/ssh/ssh_084306.913458640_n8_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cluster.go:2419).Run: context canceled
test artifacts and logs in: /artifacts/failover/partial/lease-gateway/lease=expiration/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test kv0/enc=true/nodes=1 failed: VMs preempted during the test run: teamcity-16343484-1722923371-100-n2cpu8-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2419).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv0/enc=true/nodes=1/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=true
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test kv95/enc=false/nodes=3/size=4kb failed: VMs preempted during the test run: teamcity-16343484-1722923371-108-n4cpu8-0002 (us-east1-d)

**Other Failures:**
(cluster.go:2419).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/size=4kb/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test kv95/enc=false/nodes=3/cpu=32 failed: VMs preempted during the test run: teamcity-16343484-1722923371-102-n4cpu32-0003 (us-east1-d), teamcity-16343484-1722923371-102-n4cpu32-0002 (us-east1-d)

**Other Failures:**
(cluster.go:2419).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/cpu=32/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test kv95/enc=false/nodes=3/tracing failed: VMs preempted during the test run: teamcity-16343484-1722923371-109-n4cpu8-0004 (us-central1-a)

**Other Failures:**
(cluster.go:2419).Run: full command output in run_101209.637898669_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv95/enc=false/nodes=3/tracing/cpu_arch=arm64/run_1/ssh/ssh_101209.662432278_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/tracing/cpu_arch=arm64/run_1

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test kv0/enc=false/nodes=3/cpu=32 failed: VMs preempted during the test run: teamcity-16343484-1722923371-97-n4cpu32-0003 (us-east1-d), teamcity-16343484-1722923371-97-n4cpu32-0001 (us-east1-d)

**Other Failures:**
(cluster.go:2419).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/cpu=32/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test kv95/enc=false/nodes=4/ssds=8/raid0 failed: VMs preempted during the test run: teamcity-16343484-1722923371-111-n5cpu8-0003 (us-east1-d)

**Other Failures:**
(cluster.go:2419).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=4/ssds=8/raid0/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=8
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test tpcdsvec failed: VMs preempted during the test run: teamcity-16343484-1722923371-114-n3cpu4-0001 (us-east1-d)

**Other Failures:**
(tpcdsvec.go:109).1: pgx conn: failed to connect to ``host=35.227.92.174 user=roachprod database=``: server error (ERROR: server is not accepting clients, try another node (SQLSTATE 57P01))
test artifacts and logs in: /artifacts/tpcdsvec/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test kv95/enc=false/nodes=3/cpu=96 failed: VMs preempted during the test run: teamcity-16343484-1722923371-125-n4cpu96-0004 (us-east1-c)

**Other Failures:**
(cluster.go:2419).Run: full command output in run_105905.707368941_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv95/enc=false/nodes=3/cpu=96/run_1/ssh/ssh_105905.728043140_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/cpu=96/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test replicate/up/1to3 failed: VMs preempted during the test run: teamcity-16343484-1722923371-121-n4cpu4-0001 (us-east1-c)

**Other Failures:**
(allocator.go:57).1: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/replicate/up/1to3/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test tpccbench/nodes=3/cpu=16/mt-shared-process failed: VMs preempted during the test run: teamcity-16343484-1722923371-132-n4cpu16-0003 (us-east1-c), teamcity-16343484-1722923371-132-n4cpu16-0002 (us-east1-c), teamcity-16343484-1722923371-132-n4cpu16-0004 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=16/mt-shared-process/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test tpchvec/direct_scans failed: VMs preempted during the test run: teamcity-16343484-1722923371-145-n3cpu4-0001 (us-central1-a)

**Other Failures:**
(tpchvec.go:574).runTPCHVec: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpchvec/direct_scans/cpu_arch=arm64/run_1/ssh/ssh_113644.927142964_n1_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpchvec/direct_scans/cpu_arch=arm64/run_1

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test scrub/index-only/tpcc/w=100 failed: VMs preempted during the test run: teamcity-16343484-1722923371-124-n5cpu4-0002 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/scrub/index-only/tpcc/w=100/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

test tpccbench/nodes=3/cpu=4/mt-shared-process failed: VMs preempted during the test run: teamcity-16343484-1722923371-134-n4cpu4-0002 (us-east1-b)

**Other Failures:**
(cluster.go:2305).Start: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=4/mt-shared-process/run_1/ssh/ssh_112557.666749177_n2_upload-start-script.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=4/mt-shared-process/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 3d1bcc1df630c10c4065a22fc846daa911b436d6:

test kv95/enc=true/nodes=1 failed: VMs preempted during the test run: teamcity-16356969-1723009620-07-n2cpu8-0002 (us-east1-b), teamcity-16356969-1723009620-07-n2cpu8-0001 (us-east1-b)

**Other Failures:**
(cluster.go:2419).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv95/enc=true/nodes=1/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 3d1bcc1df630c10c4065a22fc846daa911b436d6:

test cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/control failed: VMs preempted during the test run: teamcity-16356969-1723009620-39-n7cpu16-0004 (us-east1-b), teamcity-16356969-1723009620-39-n7cpu16-0001 (us-east1-b), teamcity-16356969-1723009620-39-n7cpu16-0006 (us-east1-b), teamcity-16356969-1723009620-39-n7cpu16-0005 (us-east1-b), teamcity-16356969-1723009620-39-n7cpu16-0002 (us-east1-b)

**Other Failures:**
(cluster.go:2419).Run: full command output in run_081316.006364583_n7_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/control/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 3d1bcc1df630c10c4065a22fc846daa911b436d6:

test allocbench/nodes=7/cpu=8/kv/r=50/ops=skew failed: VMs preempted during the test run: teamcity-16356969-1723009620-10-n8cpu8-0002 (us-east1-b)

**Other Failures:**
(cluster.go:2411).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1/ssh/ssh_082649.948248815_n2_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.2.0-rc @ 3d1bcc1df630c10c4065a22fc846daa911b436d6:

test decommissionBench/nodes=4/warehouses=1000 failed: VMs preempted during the test run: teamcity-16356969-1723009620-50-n5cpu16-0003 (us-central1-a)

**Other Failures:**
(decommissionbench.go:727).runDecommissionBench: monitor failure: context canceled
failed to list cockroach processes: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/decommissionBench/nodes=4/warehouses=1000/cpu_arch=arm64/run_1/ssh/ssh_084547.859414328_n3_list-processes.log): TRANSIENT_ERROR(ssh_problem): exit status 255 [owner=test-eng]
test artifacts and logs in: /artifacts/decommissionBench/nodes=4/warehouses=1000/cpu_arch=arm64/run_1

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127323 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroachdb / cockroach

roachtest: vm_preemption failed #128143