search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.18k stars 3.82k forks source link

roachtest: ssh_problem failed #134034

Open cockroach-teamcity opened 3 weeks ago

cockroach-teamcity commented 3 weeks ago

roachtest.ssh_problem failed with artifacts on release-23.1.29-rc @ 74372bc78ad4eb0304d240e6606aee8c34eb9726:

test jepsen/monotonic/parts-start-kill-2 failed: (cluster.go:2139).Run: full command output in run_091139.010698129_n1-5_sh-c-cat-controlleri.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/jepsen/monotonic/parts-start-kill-2/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #133924 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132980 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131724 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131720 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #129915 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-43887

cockroach-teamcity commented 1 week ago

roachtest.ssh_problem failed with artifacts on release-23.1.29-rc @ cb0646dc26328963ca2c89d7a8d105d166836d6d:

test tpccbench/nodes=3/cpu=16/lease=expiration failed: (cluster.go:2037).Start: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=16/lease=expiration/cpu_arch=arm64/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

- #134500 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #134374 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133924 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132980 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131724 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131720 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #129915 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

roachtest.ssh_problem failed with artifacts on release-23.1.29-rc @ cb0646dc26328963ca2c89d7a8d105d166836d6d:

test failover/non-system/blackhole failed: (cluster.go:2139).Run: full command output in run_094612.092236401_n4_sudo-iptables-A-INPU.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
(monitor.go:148).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2139).Run: context canceled
test artifacts and logs in: /artifacts/failover/non-system/blackhole/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #134939 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134500 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #134374 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133924 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132980 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131724 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131720 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #129915 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]

This test on roachdash | Improve this report!