search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.97k stars 3.79k forks source link

roachtest: vm_preemption failed #124898

Closed cockroach-teamcity closed 1 day ago

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ 827b522c24d903ec130ea4aa48dd0af6687eaf6d:

test c2c/initialscan/kv0 failed: VMs preempted during the test run: teamcity-15465864-1717134770-21-n9cpu8-0007 (us-east1-b)

**Other Failures:**
(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:787
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1014
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1293
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    monitor failure: EOF [owner=test-eng]
                    (1) attached stack trace
                      -- stack trace:
                      | main.(*monitorImpl).WaitE
                      |     main/pkg/cmd/roachtest/monitor.go:142
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).compareTenantFingerprintsAtTimestamp
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:787
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).main
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1014
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1.1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1293
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) monitor failure
                    Wraps: (3) EOF [owner=test-eng]
                    Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) registry.ErrorWithOwnership
    Test:           c2c/initialscan/kv0
(require.go:1360).NoError: FailNow called
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/initialscan/kv0/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-39129

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ 827b522c24d903ec130ea4aa48dd0af6687eaf6d:

test cdc/initial-scan failed: VMs preempted during the test run: teamcity-15465864-1717134770-27-n4cpu16-0004 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: full command output in run_072619.245450587_n4_workload-run-tpcc-wa.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/initial-scan/run_1/ssh/ssh_072619.278401475_n4_workload-run-tpcc-wa.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:388).Close: error shutting down prometheus/grafana: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/initial-scan/run_1/ssh/ssh_073629.617407897_n4_sudo-rm-rf-tmppromet.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/initial-scan/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ 827b522c24d903ec130ea4aa48dd0af6687eaf6d:

test cdc/ledger failed: VMs preempted during the test run: teamcity-15465864-1717134770-31-n4cpu16-0001 (us-east1-d), teamcity-15465864-1717134770-31-n4cpu16-0002 (us-east1-d), teamcity-15465864-1717134770-31-n4cpu16-0004 (us-east1-d), teamcity-15465864-1717134770-31-n4cpu16-0003 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: full command output in run_073946.860657657_n4_workload-run-ledger-.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/ledger/run_1/ssh/ssh_073946.895215750_n4_workload-run-ledger-.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(ts_util.go:59).mustGetMetrics: roachtestutil.addCookie: unable to extract sessionID: roachtestutil.GetSessionID: failed to authenticate: context canceled
(cdc.go:388).Close: error shutting down prometheus/grafana: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/ledger/run_1/ssh/ssh_074757.875215665_n4_sudo-rm-rf-tmppromet.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/ledger/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ 827b522c24d903ec130ea4aa48dd0af6687eaf6d:

test kv0/enc=false/nodes=3/size=4kb failed: VMs preempted during the test run: teamcity-15465864-1717134770-88-n4cpu8-0002 (us-east1-c), teamcity-15465864-1717134770-88-n4cpu8-0004 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/size=4kb/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ 827b522c24d903ec130ea4aa48dd0af6687eaf6d:

test kv95/enc=false/nodes=4 failed: VMs preempted during the test run: teamcity-15465864-1717134770-91-n5cpu8-0003 (us-east1-c), teamcity-15465864-1717134770-91-n5cpu8-0005 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=4/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ 827b522c24d903ec130ea4aa48dd0af6687eaf6d:

test kv95/enc=false/nodes=3/mt-shared-process failed: VMs preempted during the test run: teamcity-15465864-1717134770-90-n4cpu8-0002 (us-east1-d), teamcity-15465864-1717134770-90-n4cpu8-0004 (us-east1-d), teamcity-15465864-1717134770-90-n4cpu8-0001 (us-east1-d), teamcity-15465864-1717134770-90-n4cpu8-0003 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/mt-shared-process/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ 7d95120d7ad6f1e3f1b0f1c997e1ce0eaada24f9:

test ycsb/B/nodes=3/cpu=32/mvcc-range-keys=global failed: VMs preempted during the test run: teamcity-15479078-1717220948-77-n4cpu32-0004 (us-east1-d), teamcity-15479078-1717220948-77-n4cpu32-0003 (us-east1-d), teamcity-15479078-1717220948-77-n4cpu32-0001 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/B/nodes=3/cpu=32/mvcc-range-keys=global/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ 7d95120d7ad6f1e3f1b0f1c997e1ce0eaada24f9:

test replicate/rebalance/3to5 failed: VMs preempted during the test run: teamcity-15479078-1717220948-91-n6cpu4-0001 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/replicate/rebalance/3to5/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ 7d95120d7ad6f1e3f1b0f1c997e1ce0eaada24f9:

test kv95/enc=false/nodes=3/mt-shared-process failed: VMs preempted during the test run: teamcity-15484213-1717307386-80-n4cpu8-0002 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/mt-shared-process/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ 7d95120d7ad6f1e3f1b0f1c997e1ce0eaada24f9:

test cdc/initial-scan failed: VMs preempted during the test run: teamcity-15491158-1717393677-25-n4cpu16-0003 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: full command output in run_070255.847220535_n4_workload-run-tpcc-wa.log: COMMAND_PROBLEM: exit status 1
(cdc.go:388).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/initial-scan/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ 0415ebc7ded1107eb98a98d93332e59b08f544c5:

test import/tpcc/warehouses=1000/nodes=4 failed: VMs preempted during the test run: teamcity-15504819-1717480196-57-n4cpu4-0003 (us-east1-b)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/import/tpcc/warehouses=1000/nodes=4/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ 0415ebc7ded1107eb98a98d93332e59b08f544c5:

test kv95/enc=false/nodes=4/ssds=8 failed: VMs preempted during the test run: teamcity-15504819-1717480196-84-n5cpu8-0003 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=4/ssds=8/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ 0415ebc7ded1107eb98a98d93332e59b08f544c5:

test kv95/enc=true/nodes=3 failed: VMs preempted during the test run: teamcity-15504819-1717480196-85-n4cpu8-0001 (us-east1-d), teamcity-15504819-1717480196-85-n4cpu8-0004 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=true/nodes=3/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ 0415ebc7ded1107eb98a98d93332e59b08f544c5:

test ycsb/E/nodes=3/cpu=32/isolation-level=read-committed failed: VMs preempted during the test run: teamcity-15504819-1717480196-77-n4cpu32-0003 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/E/nodes=3/cpu=32/isolation-level=read-committed/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ 0415ebc7ded1107eb98a98d93332e59b08f544c5:

test tpccbench/nodes=3/cpu=4 failed: VMs preempted during the test run: teamcity-15504819-1717480196-101-n4cpu4-0003 (us-east1-d), teamcity-15504819-1717480196-101-n4cpu4-0001 (us-east1-d)

**Other Failures:**
(cluster.go:2335).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=4/run_1/ssh/ssh_131904.465579944_n3_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=4/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ fb26da766a8f98eec602de55fdf6c813ee560069:

test cdc/initial-scan failed: VMs preempted during the test run: teamcity-15519805-1717566618-22-n4cpu16-0002 (us-east1-c), teamcity-15519805-1717566618-22-n4cpu16-0004 (us-east1-c), teamcity-15519805-1717566618-22-n4cpu16-0001 (us-east1-c), teamcity-15519805-1717566618-22-n4cpu16-0003 (us-east1-c)

**Other Failures:**
(cluster.go:2284).Start: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/initial-scan/run_1/ssh/ssh_063945.485417975_n3_upload-start-script.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/initial-scan/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ fb26da766a8f98eec602de55fdf6c813ee560069:

test cdc/tpcc-1000/sink=kafka failed: VMs preempted during the test run: teamcity-15519805-1717566618-21-n4cpu16-0003 (us-east1-b)

**Other Failures:**
(cluster.go:2398).Run: full command output in run_070901.591639602_n4_workload-run-tpcc-wa.log: COMMAND_PROBLEM: exit status 1
(cdc.go:388).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/tpcc-1000/sink=kafka/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ fb26da766a8f98eec602de55fdf6c813ee560069:

test cdc/workload/kv0/nodes=5/cpu=16/ranges=100/server=processor/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-15519805-1717566618-32-n7cpu16-0006 (us-east1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100/server=processor/protocol=mux/format=json/sink=null/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ fb26da766a8f98eec602de55fdf6c813ee560069:

test cdc/scan/catchup/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-15519805-1717566618-28-n6cpu16-0002 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/scan/catchup/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ fb26da766a8f98eec602de55fdf6c813ee560069:

test cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/server=processor/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-15519805-1717566618-33-n7cpu16-0003 (us-east1-d), teamcity-15519805-1717566618-33-n7cpu16-0004 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/server=processor/protocol=mux/format=json/sink=null/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ fb26da766a8f98eec602de55fdf6c813ee560069:

test decommissionBench/nodes=6/warehouses=1000/drain-first/while-upreplicating/target=2/multi-region failed: VMs preempted during the test run: teamcity-15519805-1717566618-49-n8cpu16-geo-0005 (us-central1-b)

**Other Failures:**
(decommissionbench.go:721).runDecommissionBench: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/decommissionBench/nodes=6/warehouses=1000/drain-first/while-upreplicating/target=2/multi-region/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ fb26da766a8f98eec602de55fdf6c813ee560069:

test import/tpch/nodes=8 failed: VMs preempted during the test run: teamcity-15519805-1717566618-66-n8cpu4-0006 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/import/tpch/nodes=8/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ fb26da766a8f98eec602de55fdf6c813ee560069:

test tpccbench/nodes=6/cpu=16/multi-az failed: VMs preempted during the test run: teamcity-15519805-1717566618-102-n7cpu16-geo-0006 (us-central1-c), teamcity-15519805-1717566618-102-n7cpu16-geo-0005 (us-central1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpccbench/nodes=6/cpu=16/multi-az/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ fb26da766a8f98eec602de55fdf6c813ee560069:

test tpccbench/nodes=9/cpu=4/multi-region failed: VMs preempted during the test run: teamcity-15519805-1717566618-103-n12cpu4hm-geo-0006 (us-west1-b), teamcity-15519805-1717566618-103-n12cpu4hm-geo-0007 (us-west1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ fb26da766a8f98eec602de55fdf6c813ee560069:

test tpccbench/nodes=3/cpu=4 failed: VMs preempted during the test run: teamcity-15519805-1717566618-99-n4cpu4-0002 (us-east1-d)

**Other Failures:**
(cluster.go:2335).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=4/run_1/ssh/ssh_131602.798717054_n2_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=4/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ fb26da766a8f98eec602de55fdf6c813ee560069:

test failover/non-system/crash failed: VMs preempted during the test run: teamcity-15536048-1717652953-49-n7cpu2-0001 (us-east1-d)

**Other Failures:**
(failover.go:1775).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/failover/non-system/crash/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ fb26da766a8f98eec602de55fdf6c813ee560069:

test tpccbench/nodes=9/cpu=4/multi-region failed: VMs preempted during the test run: teamcity-15536048-1717652953-103-n12cpu4hm-geo-0005 (us-west1-b), teamcity-15536048-1717652953-103-n12cpu4hm-geo-0007 (us-west1-b), teamcity-15536048-1717652953-103-n12cpu4hm-geo-0006 (us-west1-b), teamcity-15536048-1717652953-103-n12cpu4hm-geo-0008 (us-west1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ fb26da766a8f98eec602de55fdf6c813ee560069:

test schemachange/index/tpcc/w=800 failed: VMs preempted during the test run: teamcity-15536048-1717652953-96-n5cpu16-0001 (us-east1-d), teamcity-15536048-1717652953-96-n5cpu16-0003 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/schemachange/index/tpcc/w=800/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ b7a5b158354408939cb3d680aca4305c91b415af:

test failover/non-system/deadlock failed: VMs preempted during the test run: teamcity-15551410-1717739344-57-n7cpu2-0001 (us-east1-c), teamcity-15551410-1717739344-57-n7cpu2-0003 (us-east1-c)

**Other Failures:**
(failover.go:1775).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/failover/non-system/deadlock/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ b7a5b158354408939cb3d680aca4305c91b415af:

test ycsb/E/nodes=3/cpu=32/mvcc-range-keys=global failed: VMs preempted during the test run: teamcity-15551410-1717739344-90-n4cpu32-0002 (us-east1-b)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/E/nodes=3/cpu=32/mvcc-range-keys=global/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ b7a5b158354408939cb3d680aca4305c91b415af:

test tpccbench/nodes=6/cpu=16/multi-az failed: VMs preempted during the test run: teamcity-15551410-1717739344-112-n7cpu16-geo-0006 (us-central1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpccbench/nodes=6/cpu=16/multi-az/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ b7a5b158354408939cb3d680aca4305c91b415af:

test tpccbench/nodes=9/cpu=4/chaos/partition failed: VMs preempted during the test run: teamcity-15574261-1717912233-108-n10cpu4-0002 (us-east1-d), teamcity-15574261-1717912233-108-n10cpu4-0009 (us-east1-d)

**Other Failures:**
(cluster.go:2335).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/run_1/ssh/ssh_130026.353782613_n9_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ b7a5b158354408939cb3d680aca4305c91b415af:

test c2c/MultiRegion/SameRegions/kv0 failed: VMs preempted during the test run: teamcity-15583933-1717998540-20-n9cpu8-geo-0007 (us-west1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/MultiRegion/SameRegions/kv0/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ b7a5b158354408939cb3d680aca4305c91b415af:

test allocbench/nodes=7/cpu=8/kv/r=0/access=skew failed: VMs preempted during the test run: teamcity-15583933-1717998540-10-n8cpu8-0006 (us-east1-d)

**Other Failures:**
(cluster.go:2390).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/access=skew/run_1/ssh/ssh_072927.429918524_n6_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/access=skew/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ b7a5b158354408939cb3d680aca4305c91b415af:

test kv95limited-spans/enc=false/nodes=1/seq/no-load-splitting failed: VMs preempted during the test run: teamcity-15583933-1717998540-09-n2cpu8-0002 (us-east1-c), teamcity-15583933-1717998540-09-n2cpu8-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95limited-spans/enc=false/nodes=1/seq/no-load-splitting/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ b7a5b158354408939cb3d680aca4305c91b415af:

test failover/chaos/read-only failed: VMs preempted during the test run: teamcity-15583933-1717998540-52-n10cpu2-0008 (us-east1-d)

**Other Failures:**
(failover.go:1775).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2398).Run: context canceled
(cluster.go:2398).Run: context canceled
(cluster.go:2398).Run: context canceled
(cluster.go:2398).Run: context canceled
test artifacts and logs in: /artifacts/failover/chaos/read-only/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ b7a5b158354408939cb3d680aca4305c91b415af:

test failover/chaos/read-write failed: VMs preempted during the test run: teamcity-15583933-1717998540-53-n10cpu2-0002 (us-east1-d), teamcity-15583933-1717998540-53-n10cpu2-0001 (us-east1-d), teamcity-15583933-1717998540-53-n10cpu2-0008 (us-east1-d), teamcity-15583933-1717998540-53-n10cpu2-0010 (us-east1-d), teamcity-15583933-1717998540-53-n10cpu2-0006 (us-east1-d)

**Other Failures:**
(failover.go:1775).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2398).Run: context canceled
(cluster.go:2398).Run: context canceled
(cluster.go:2398).Run: context canceled
(cluster.go:2398).Run: context canceled
test artifacts and logs in: /artifacts/failover/chaos/read-write/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ b7a5b158354408939cb3d680aca4305c91b415af:

test tpccbench/nodes=9/cpu=4/multi-region failed: VMs preempted during the test run: teamcity-15583933-1717998540-100-n12cpu4hm-geo-0006 (us-west1-b), teamcity-15583933-1717998540-100-n12cpu4hm-geo-0008 (us-west1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ b7a5b158354408939cb3d680aca4305c91b415af:

test ycsb/F/nodes=3/cpu=32/mvcc-range-keys=global failed: VMs preempted during the test run: teamcity-15583933-1717998540-81-n4cpu32-0003 (us-east1-d), teamcity-15583933-1717998540-81-n4cpu32-0004 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/F/nodes=3/cpu=32/mvcc-range-keys=global/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ 25e4a522b5a77449e43404987337d75f53e66304:

test tpccbench/nodes=3/cpu=16/mt-shared-process failed: VMs preempted during the test run: teamcity-15601173-1718085039-30-n4cpu16-0003 (us-east1-b)

**Other Failures:**
(cluster.go:2335).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=16/mt-shared-process/run_1/ssh/ssh_090552.318186743_n3_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=16/mt-shared-process/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ 25e4a522b5a77449e43404987337d75f53e66304:

test kv95/enc=false/nodes=1/cpu=32/mt-shared-process failed: VMs preempted during the test run: teamcity-15601173-1718085039-78-n2cpu32-0001 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=1/cpu=32/mt-shared-process/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ 25e4a522b5a77449e43404987337d75f53e66304:

test kv0/enc=true/nodes=1 failed: VMs preempted during the test run: teamcity-15601173-1718085039-76-n2cpu8-0002 (us-east1-c), teamcity-15601173-1718085039-76-n2cpu8-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=true/nodes=1/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ 25e4a522b5a77449e43404987337d75f53e66304:

test kv95/enc=false/nodes=3/cpu=32/size=4kb failed: VMs preempted during the test run: teamcity-15601173-1718085039-81-n4cpu32-0003 (us-east1-c), teamcity-15601173-1718085039-81-n4cpu32-0004 (us-east1-c), teamcity-15601173-1718085039-81-n4cpu32-0002 (us-east1-c), teamcity-15601173-1718085039-81-n4cpu32-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/cpu=32/size=4kb/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ 25e4a522b5a77449e43404987337d75f53e66304:

test kv95/enc=false/nodes=3/size=4kb failed: VMs preempted during the test run: teamcity-15601173-1718085039-82-n4cpu8-0002 (us-east1-c), teamcity-15601173-1718085039-82-n4cpu8-0001 (us-east1-c), teamcity-15601173-1718085039-82-n4cpu8-0004 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/size=4kb/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ 25e4a522b5a77449e43404987337d75f53e66304:

test kv95/enc=false/nodes=3/cpu=32 failed: VMs preempted during the test run: teamcity-15601173-1718085039-80-n4cpu32-0004 (us-east1-c), teamcity-15601173-1718085039-80-n4cpu32-0002 (us-east1-c), teamcity-15601173-1718085039-80-n4cpu32-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/cpu=32/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ 25e4a522b5a77449e43404987337d75f53e66304:

test schemachange/index/tpcc/w=800 failed: VMs preempted during the test run: teamcity-15601173-1718085039-98-n5cpu16-0001 (us-east1-c), teamcity-15601173-1718085039-98-n5cpu16-0002 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/schemachange/index/tpcc/w=800/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ 25e4a522b5a77449e43404987337d75f53e66304:

test failover/non-system/blackhole/lease=expiration failed: VMs preempted during the test run: teamcity-15618473-1718171417-53-n7cpu2-0006 (us-east1-d)

**Other Failures:**
(failover.go:1775).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2398).Run: context canceled
test artifacts and logs in: /artifacts/failover/non-system/blackhole/lease=expiration/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ 044088e615f601556c78ce06d8e1a48a2adc6607:

test failover/non-system/blackhole-recv/lease=expiration failed: VMs preempted during the test run: teamcity-15637260-1718257788-48-n7cpu2-0006 (us-east1-c)

**Other Failures:**
(failover.go:1775).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2398).Run: context canceled
test artifacts and logs in: /artifacts/failover/non-system/blackhole-recv/lease=expiration/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ 044088e615f601556c78ce06d8e1a48a2adc6607:

test failover/non-system/blackhole/lease=expiration failed: VMs preempted during the test run: teamcity-15637260-1718257788-49-n7cpu2-0004 (us-east1-c), teamcity-15637260-1718257788-49-n7cpu2-0003 (us-east1-c), teamcity-15637260-1718257788-49-n7cpu2-0002 (us-east1-c), teamcity-15637260-1718257788-49-n7cpu2-0007 (us-east1-c)

**Other Failures:**
(failover.go:1775).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2398).Run: context canceled
test artifacts and logs in: /artifacts/failover/non-system/blackhole/lease=expiration/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ 044088e615f601556c78ce06d8e1a48a2adc6607:

test kv95/enc=false/nodes=3/cpu=32/mt-shared-process failed: VMs preempted during the test run: teamcity-15637260-1718257788-77-n4cpu32-0002 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/cpu=32/mt-shared-process/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.vm_preemption failed with artifacts on release-24.1.1-rc @ 044088e615f601556c78ce06d8e1a48a2adc6607:

test tpccbench/nodes=9/cpu=4/chaos/partition failed: VMs preempted during the test run: teamcity-15637260-1718257788-95-n10cpu4-0009 (us-east1-d)

**Other Failures:**
(cluster.go:2284).Start: failed to find services to register: failed to find 1 open ports: TRANSIENT_ERROR(open_ports): expected 1 ports, got 0
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123543 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #121348 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #118528 roachtest: vm_preemption failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!