search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.87k stars 3.77k forks source link

roachtest: vm_preemption failed #127323

Open cockroach-teamcity opened 1 month ago

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 984bb6088542c5ef9b9ce8c44af6d980bd7db6f0:

test allocbench/nodes=7/cpu=8/kv/r=50/ops=skew failed: VMs preempted during the test run: teamcity-16105839-1721195315-05-n8cpu8-0005 (us-east1-c), teamcity-16105839-1721195315-05-n8cpu8-0001 (us-east1-c)

**Other Failures:**
(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:394
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:344
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:269
                                main/pkg/cmd/roachtest/test_runner.go:1215
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    monitor failure: EOF [owner=test-eng]
                    (1) attached stack trace
                      -- stack trace:
                      | main.(*monitorImpl).WaitE
                      |     main/pkg/cmd/roachtest/monitor.go:142
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBenchSample
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:393
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBench
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:344
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAllocationBenchSpec.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:269
                      | main.(*testRunner).runTest.func2
                      |     main/pkg/cmd/roachtest/test_runner.go:1215
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) monitor failure
                    Wraps: (3) secondary error attachment
                      | context canceled
                      | (1) context canceled
                      | Error types: (1) *errors.errorString
                    Wraps: (4) EOF [owner=test-eng]
                    Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *secondary.withSecondaryError (4) registry.ErrorWithOwnership
    Test:           allocbench/nodes=7/cpu=8/kv/r=50/ops=skew
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=50/ops=skew/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-40378

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 984bb6088542c5ef9b9ce8c44af6d980bd7db6f0:

test kv95/enc=false/nodes=3/mt-shared-process failed: VMs preempted during the test run: teamcity-16105839-1721195315-104-n4cpu8-0004 (us-east1-b)

**Other Failures:**
(cluster.go:2398).Run: full command output in run_101051.010440837_n4_workload-run-kv-tole.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv95/enc=false/nodes=3/mt-shared-process/run_1/ssh/ssh_101051.034111188_n4_workload-run-kv-tole.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/mt-shared-process/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 984bb6088542c5ef9b9ce8c44af6d980bd7db6f0:

test tpccbench/nodes=3/cpu=4/mt-shared-process failed: VMs preempted during the test run: teamcity-16105839-1721195315-77-n4cpu4-0003 (us-east1-d)

**Other Failures:**
(cluster.go:2335).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=4/mt-shared-process/run_1/ssh/ssh_103142.402189782_n3_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=4/mt-shared-process/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 984bb6088542c5ef9b9ce8c44af6d980bd7db6f0:

test scrub/index-only/tpcc/w=100 failed: VMs preempted during the test run: teamcity-16105839-1721195315-127-n5cpu4-0005 (us-central1-a)

**Other Failures:**
(cluster.go:2398).Run: full command output in run_121545.259461475_n5_cockroach-workload-c.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/scrub/index-only/tpcc/w=100/cpu_arch=arm64/run_1/ssh/ssh_121545.278864234_n5_cockroach-workload-c.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/scrub/index-only/tpcc/w=100/cpu_arch=arm64/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 984bb6088542c5ef9b9ce8c44af6d980bd7db6f0:

test tpcc/headroom/n4cpu16 failed: VMs preempted during the test run: teamcity-16105839-1721195315-130-n4cpu16-0002 (us-east1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpcc/headroom/n4cpu16/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 984bb6088542c5ef9b9ce8c44af6d980bd7db6f0:

test cdc/workload/kv100/nodes=5/cpu=16/ranges=100/server=scheduler/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-16125830-1721281803-30-n7cpu16-0001 (us-east1-d), teamcity-16125830-1721281803-30-n7cpu16-0004 (us-east1-d), teamcity-16125830-1721281803-30-n7cpu16-0002 (us-east1-d), teamcity-16125830-1721281803-30-n7cpu16-0007 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100/server=scheduler/protocol=mux/format=json/sink=null/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 984bb6088542c5ef9b9ce8c44af6d980bd7db6f0:

test allocbench/nodes=7/cpu=8/kv/r=0/access=skew failed: VMs preempted during the test run: teamcity-16125830-1721281803-07-n8cpu8-0007 (us-east1-c)

**Other Failures:**
(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:394
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:344
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:269
                                main/pkg/cmd/roachtest/test_runner.go:1215
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    monitor failure: EOF [owner=test-eng]
                    (1) attached stack trace
                      -- stack trace:
                      | main.(*monitorImpl).WaitE
                      |     main/pkg/cmd/roachtest/monitor.go:142
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBenchSample
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:393
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBench
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:344
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAllocationBenchSpec.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:269
                      | main.(*testRunner).runTest.func2
                      |     main/pkg/cmd/roachtest/test_runner.go:1215
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) monitor failure
                    Wraps: (3) secondary error attachment
                      | context canceled
                      | (1) context canceled
                      | Error types: (1) *errors.errorString
                    Wraps: (4) EOF [owner=test-eng]
                    Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *secondary.withSecondaryError (4) registry.ErrorWithOwnership
    Test:           allocbench/nodes=7/cpu=8/kv/r=0/access=skew
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/access=skew/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 984bb6088542c5ef9b9ce8c44af6d980bd7db6f0:

test failover/partial/lease-gateway/read-only/read-write/lease=expiration failed: VMs preempted during the test run: teamcity-16125830-1721281803-55-n8cpu2-0006 (us-east1-c)

**Other Failures:**
(failover.go:1771).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2398).Run: context canceled
test artifacts and logs in: /artifacts/failover/partial/lease-gateway/read-only/read-write/lease=expiration/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 984bb6088542c5ef9b9ce8c44af6d980bd7db6f0:

test replicate/rebalance/3to5 failed: VMs preempted during the test run: teamcity-16125830-1721281803-90-n6cpu4-0006 (us-east1-d), teamcity-16125830-1721281803-90-n6cpu4-0004 (us-east1-d), teamcity-16125830-1721281803-90-n6cpu4-0001 (us-east1-d), teamcity-16125830-1721281803-90-n6cpu4-0002 (us-east1-d)

**Other Failures:**
(allocator.go:57).1: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/replicate/rebalance/3to5/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 984bb6088542c5ef9b9ce8c44af6d980bd7db6f0:

test schemachange/index/tpcc/w=800 failed: VMs preempted during the test run: teamcity-16125830-1721281803-105-n5cpu16-0002 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/schemachange/index/tpcc/w=800/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 984bb6088542c5ef9b9ce8c44af6d980bd7db6f0:

test tpcc/headroom/isolation-level=read-committed/n4cpu16 failed: VMs preempted during the test run: teamcity-16125830-1721281803-108-n4cpu16-0004 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: full command output in run_103218.482216617_n4_cockroach-workload-r.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpcc/headroom/isolation-level=read-committed/n4cpu16/run_1/ssh/ssh_103218.519432883_n4_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpcc/headroom/isolation-level=read-committed/n4cpu16/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 984bb6088542c5ef9b9ce8c44af6d980bd7db6f0:

test ycsb/A/nodes=3/cpu=32/isolation-level=read-committed failed: VMs preempted during the test run: teamcity-16125830-1721281803-122-n4cpu32-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/A/nodes=3/cpu=32/isolation-level=read-committed/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 984bb6088542c5ef9b9ce8c44af6d980bd7db6f0:

test ycsb/D/nodes=3/cpu=32/isolation-level=read-committed failed: VMs preempted during the test run: teamcity-16125830-1721281803-124-n4cpu32-0001 (us-east1-d), teamcity-16125830-1721281803-124-n4cpu32-0004 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/D/nodes=3/cpu=32/isolation-level=read-committed/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 984bb6088542c5ef9b9ce8c44af6d980bd7db6f0:

test tpccbench/nodes=3/cpu=16/mt-shared-process failed: VMs preempted during the test run: teamcity-16125830-1721281803-113-n4cpu16-0001 (us-east1-d), teamcity-16125830-1721281803-113-n4cpu16-0003 (us-east1-d), teamcity-16125830-1721281803-113-n4cpu16-0004 (us-east1-d)

**Other Failures:**
(cluster.go:2335).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=16/mt-shared-process/run_1/ssh/ssh_111735.312639551_n1_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=16/mt-shared-process/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test cdc/cloud-sink-gcs/rangefeed=true failed: VMs preempted during the test run: teamcity-16146010-1721368161-24-n4cpu16-0004 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: full command output in run_071958.097707922_n4_workload-run-tpcc-wa.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/cloud-sink-gcs/rangefeed=true/run_1/ssh/ssh_071958.142680468_n4_workload-run-tpcc-wa.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:388).Close: error shutting down prometheus/grafana: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/cloud-sink-gcs/rangefeed=true/run_1/ssh/ssh_074528.993893709_n4_sudo-rm-rf-tmppromet.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/cloud-sink-gcs/rangefeed=true/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test cdc/sink-chaos failed: VMs preempted during the test run: teamcity-16146010-1721368161-29-n4cpu16-0001 (us-east1-d), teamcity-16146010-1721368161-29-n4cpu16-0003 (us-east1-d), teamcity-16146010-1721368161-29-n4cpu16-0004 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: full command output in run_073934.304790081_n4_workload-run-tpcc-wa.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/sink-chaos/run_1/ssh/ssh_073934.327402500_n4_workload-run-tpcc-wa.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:388).Close: error shutting down prometheus/grafana: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/sink-chaos/run_1/ssh/ssh_074635.936296729_n4_sudo-rm-rf-tmppromet.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/sink-chaos/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test failover/non-system/pause/read-only/read-write/lease=expiration failed: VMs preempted during the test run: teamcity-16146010-1721368161-63-n7cpu2-0002 (us-east1-b)

**Other Failures:**
(failover.go:1771).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/failover/non-system/pause/read-only/read-write/lease=expiration/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test failover/non-system/disk-stall/read-only/read-write/lease=expiration failed: VMs preempted during the test run: teamcity-16146010-1721368161-66-n7cpu2-0005 (us-east1-c), teamcity-16146010-1721368161-66-n7cpu2-0003 (us-east1-c)

**Other Failures:**
(failover.go:1771).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2398).Run: context canceled
test artifacts and logs in: /artifacts/failover/non-system/disk-stall/read-only/read-write/lease=expiration/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test failover/non-system/disk-stall/read-only/read-write failed: VMs preempted during the test run: teamcity-16146010-1721368161-65-n7cpu2-0005 (us-east1-b)

**Other Failures:**
(failover.go:1771).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2398).Run: context canceled
test artifacts and logs in: /artifacts/failover/non-system/disk-stall/read-only/read-write/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test decommissionBench/nodes=4/warehouses=1000/duration=1h0m0s failed: VMs preempted during the test run: teamcity-16146010-1721368161-50-n5cpu16-0001 (us-east1-b), teamcity-16146010-1721368161-50-n5cpu16-0005 (us-east1-b)

**Other Failures:**
(decommissionbench.go:847).runDecommissionBenchLong: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/decommissionBench/nodes=4/warehouses=1000/duration=1h0m0s/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test failover/partial/lease-gateway/read-only/read-write failed: VMs preempted during the test run: teamcity-16146010-1721368161-67-n8cpu2-0004 (us-east1-d), teamcity-16146010-1721368161-67-n8cpu2-0003 (us-east1-d), teamcity-16146010-1721368161-67-n8cpu2-0005 (us-east1-d)

**Other Failures:**
(failover.go:1771).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2398).Run: context canceled
test artifacts and logs in: /artifacts/failover/partial/lease-gateway/read-only/read-write/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test tpccbench/nodes=3/cpu=16 failed: VMs preempted during the test run: teamcity-16146010-1721368161-25-n4cpu16-0002 (us-east1-b), teamcity-16146010-1721368161-25-n4cpu16-0004 (us-east1-b)

**Other Failures:**
(cluster.go:2284).Start: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=16/run_1/ssh/ssh_093048.625003557_n2_upload-start-script.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=16/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test kv0/enc=false/nodes=3 failed: VMs preempted during the test run: teamcity-16146010-1721368161-99-n4cpu8-0002 (us-central1-a)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/cpu_arch=arm64/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test ycsb/B/nodes=3/cpu=32/mvcc-range-keys=global failed: VMs preempted during the test run: teamcity-16146010-1721368161-130-n4cpu32-0004 (us-east1-b)

**Other Failures:**
(cluster.go:2398).Run: full command output in run_113643.394362029_n4_workload-run-ycsb-in.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/ycsb/B/nodes=3/cpu=32/mvcc-range-keys=global/run_1/ssh/ssh_113643.414324208_n4_workload-run-ycsb-in.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/B/nodes=3/cpu=32/mvcc-range-keys=global/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test ycsb/E/nodes=3/cpu=32/mvcc-range-keys=global failed: VMs preempted during the test run: teamcity-16146010-1721368161-137-n4cpu32-0002 (us-east1-c), teamcity-16146010-1721368161-137-n4cpu32-0003 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/E/nodes=3/cpu=32/mvcc-range-keys=global/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test import/tpch/nodes=8 failed: VMs preempted during the test run: teamcity-16146010-1721368161-77-n8cpu4-0008 (us-east1-c), teamcity-16146010-1721368161-77-n8cpu4-0002 (us-east1-c), teamcity-16146010-1721368161-77-n8cpu4-0007 (us-east1-c), teamcity-16146010-1721368161-77-n8cpu4-0005 (us-east1-c), teamcity-16146010-1721368161-77-n8cpu4-0004 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/import/tpch/nodes=8/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test kv95/enc=false/nodes=3 failed: VMs preempted during the test run: teamcity-16162333-1721454705-89-n4cpu8-0002 (us-east1-d), teamcity-16162333-1721454705-89-n4cpu8-0001 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test kv0/enc=false/nodes=3/cpu=32/size=64kb failed: VMs preempted during the test run: teamcity-16162333-1721454705-84-n4cpu32-0003 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/cpu=32/size=64kb/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test kv0/enc=true/nodes=3 failed: VMs preempted during the test run: teamcity-16162333-1721454705-86-n4cpu8-0004 (us-east1-c), teamcity-16162333-1721454705-86-n4cpu8-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=true/nodes=3/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test kv95/enc=true/nodes=3 failed: VMs preempted during the test run: teamcity-16162333-1721454705-95-n4cpu8-0004 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: full command output in run_105335.889939908_n4_workload-run-kv-tole.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv95/enc=true/nodes=3/run_1/ssh/ssh_105335.912029378_n4_workload-run-kv-tole.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=true/nodes=3/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test kv95/enc=false/nodes=3/tracing failed: VMs preempted during the test run: teamcity-16162333-1721454705-94-n4cpu8-0003 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/tracing/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test ledger/nodes=6/multi-az failed: VMs preempted during the test run: teamcity-16162333-1721454705-104-n7cpu16-geo-0005 (us-central1-c)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ledger/nodes=6/multi-az/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test allocbench/nodes=7/cpu=8/kv/r=0/ops=skew failed: VMs preempted during the test run: teamcity-16171891-1721540954-10-n8cpu8-0002 (us-east1-d)

**Other Failures:**
(cluster.go:2390).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/ops=skew/run_1/ssh/ssh_073147.000053896_n2_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/ops=skew/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test tpccbench/nodes=3/cpu=16 failed: VMs preempted during the test run: teamcity-16171891-1721540954-24-n4cpu16-0003 (us-east1-c), teamcity-16171891-1721540954-24-n4cpu16-0002 (us-east1-c)

**Other Failures:**
(cluster.go:2335).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=16/run_1/ssh/ssh_081801.517030912_n2_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=16/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/control failed: VMs preempted during the test run: teamcity-16171891-1721540954-35-n7cpu16-0004 (us-east1-d), teamcity-16171891-1721540954-35-n7cpu16-0005 (us-east1-d)

**Other Failures:**
(test_runner.go:1239).runTest: test timed out (1h0m0s)
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/control/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test kv50/enc=false/nodes=1/conc=8192 failed: VMs preempted during the test run: teamcity-16171891-1721540954-66-n2cpu8-0002 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: full command output in run_093900.291907156_n2_workload-run-kv-tole.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv50/enc=false/nodes=1/conc=8192/run_1/ssh/ssh_093900.316118993_n2_workload-run-kv-tole.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv50/enc=false/nodes=1/conc=8192/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test ycsb/C/nodes=3/cpu=32/isolation-level=read-committed failed: VMs preempted during the test run: teamcity-16171891-1721540954-78-n4cpu32-0002 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/C/nodes=3/cpu=32/isolation-level=read-committed/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test kv95/enc=false/nodes=1/mt-shared-process failed: VMs preempted during the test run: teamcity-16180453-1721627339-09-n2cpu8-0001 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=1/mt-shared-process/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test kv0/enc=false/nodes=3/cpu=32/size=4kb failed: VMs preempted during the test run: teamcity-16180453-1721627339-102-n4cpu32-0002 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/cpu=32/size=4kb/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test kv0/enc=false/nodes=3/cpu=32/size=64kb failed: VMs preempted during the test run: teamcity-16180453-1721627339-99-n4cpu32-0001 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/cpu=32/size=64kb/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test schemachange/during/tpcc failed: VMs preempted during the test run: teamcity-16180453-1721627339-123-n5cpu16-0002 (us-east1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/schemachange/during/tpcc/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test cdc/scan/catchup-cold/nodes=5/cpu=16/rows=1G/ranges=100/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-16191037-1721713838-29-n6cpu16-0001 (us-east1-b)

**Other Failures:**
(cluster.go:2398).Run: full command output in run_071051.078758693_n6_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/cdc/scan/catchup-cold/nodes=5/cpu=16/rows=1G/ranges=100/protocol=mux/format=json/sink=null/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test allocbench/nodes=7/cpu=8/kv/r=95/access=skew failed: VMs preempted during the test run: teamcity-16191037-1721713838-10-n8cpu8-0003 (us-east1-d), teamcity-16191037-1721713838-10-n8cpu8-0004 (us-east1-d)

**Other Failures:**
(cluster.go:2390).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/access=skew/run_1/ssh/ssh_073130.951860047_n4_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/access=skew/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test tpccbench/nodes=3/cpu=16/enc=true failed: VMs preempted during the test run: teamcity-16191037-1721713838-33-n4cpu16-0001 (us-east1-c), teamcity-16191037-1721713838-33-n4cpu16-0003 (us-east1-c)

**Other Failures:**
(cluster.go:2335).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=16/enc=true/run_1/ssh/ssh_090130.628410956_n3_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=16/enc=true/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test tpccbench/nodes=3/cpu=16 failed: VMs preempted during the test run: teamcity-16191037-1721713838-25-n4cpu16-0003 (us-east1-b)

**Other Failures:**
(cluster.go:2335).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=16/run_1/ssh/ssh_093319.171304515_n3_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=16/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test kv95/enc=false/nodes=4/ssds=8/raid0 failed: VMs preempted during the test run: teamcity-16191037-1721713838-102-n5cpu8-0005 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: full command output in run_112548.182218011_n5_workload-run-kv-tole.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv95/enc=false/nodes=4/ssds=8/raid0/run_1/ssh/ssh_112548.201732927_n5_workload-run-kv-tole.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=4/ssds=8/raid0/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test kv/contention/nodes=4 failed: VMs preempted during the test run: teamcity-16191037-1721713838-91-n5cpu4-0003 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv/contention/nodes=4/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test tpchvec/direct_scans/mt-shared-process failed: VMs preempted during the test run: teamcity-16191037-1721713838-125-n3cpu4-0003 (us-east1-d)

**Other Failures:**
(tpchvec.go:65).performClusterSetup: pq: server is not accepting clients, try another node
test artifacts and logs in: /artifacts/tpchvec/direct_scans/mt-shared-process/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test tpcdsvec failed: VMs preempted during the test run: teamcity-16191037-1721713838-123-n3cpu4-0002 (us-east1-d), teamcity-16191037-1721713838-123-n3cpu4-0003 (us-east1-d)

**Other Failures:**
(tpcdsvec.go:109).1: pgx conn: failed to connect to ``host=34.75.162.149 user=roachprod database=``: server error (ERROR: internal error while retrieving user account memberships: operation "get-user-session" timed out after 10.001s (given timeout 10s): internal error while retrieving user account: get auth info error: interrupted during singleflight acquire-lease:1: context deadline exceeded (SQLSTATE 28000))
test artifacts and logs in: /artifacts/tpcdsvec/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test tpccbench/nodes=3/cpu=4/lease=expiration failed: VMs preempted during the test run: teamcity-16191037-1721713838-122-n4cpu4-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2335).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=4/lease=expiration/run_1/ssh/ssh_121546.478823135_n1_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=4/lease=expiration/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.vm_preemption failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test ycsb/C/nodes=3/cpu=32 failed: VMs preempted during the test run: teamcity-16191037-1721713838-128-n4cpu32-0001 (us-east1-b), teamcity-16191037-1721713838-128-n4cpu32-0004 (us-east1-b)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/C/nodes=3/cpu=32/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126985 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #125727 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124898 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!