search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.18k stars 3.82k forks source link

roachtest: ssh_problem failed #131724

Open cockroach-teamcity opened 1 month ago

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test backup-restore/small-ranges failed: (test_runner.go:821).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64" failed: error persisted after 2 attempts: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64 ubuntu@35.223.82.63:./cockroach
Warning: Permanently added '35.223.82.63' (ECDSA) to the list of known hosts.
client_loop: send disconnect: Broken pipe
lost connection: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131720 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131148 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.12-rc] - #130639 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.3-rc] - #130478 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.5-rc] - #130477 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.5-rc] - #129915 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129312 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #127328 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126478 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-42663

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test cdc/workload/kv100/nodes=5/cpu=16/ranges=100/server=processor/protocol=mux/format=json/sink=null failed: (test_runner.go:821).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64 ubuntu@34.46.8.254:./cockroach: signal: killed [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131720 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131148 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.12-rc] - #130639 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.3-rc] - #130478 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.5-rc] - #130477 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.5-rc] - #129915 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129312 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #127328 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126478 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test failover/chaos/read-only/lease=expiration failed: (test_runner.go:821).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64" failed: error persisted after 2 attempts: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64 ubuntu@34.16.86.4:./cockroach
Warning: Permanently added '34.16.86.4' (ECDSA) to the list of known hosts.
client_loop: send disconnect: Broken pipe
lost connection: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131720 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131148 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.12-rc] - #130639 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.3-rc] - #130478 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.5-rc] - #130477 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.5-rc] - #129915 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129312 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #127328 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.3-rc] - #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126478 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test tpccbench/nodes=6/cpu=16/multi-az failed: (test_runner.go:821).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 ubuntu@34.44.199.19:./cockroach
ssh: connect to host 34.44.199.19 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131720 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #129915 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test alterpk-bank failed: (test_runner.go:821).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 ubuntu@35.231.246.173:./cockroach
ssh: connect to host 35.231.246.173 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131720 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #129915 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test schemachange/invertedindex failed: (cluster.go:2284).Start: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/schemachange/invertedindex/cpu_arch=arm64/run_1/ssh/ssh_105205.268906055_n1_set-cluster-settings.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/schemachange/invertedindex/cpu_arch=arm64/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131720 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #129915 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test admission-control/snapshot-overload-excise failed: (test_runner.go:821).func4: cluster.PutE: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/_runner-logs/ssh/ssh_055607.412372566_n1_if-hostname-teamcity.log): TRANSIENT_ERROR(ssh_problem): exit status 255 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131720 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #129915 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test rebalance/by-load/replicas/mixed-version failed: (test_runner.go:821).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64 ubuntu@34.41.170.41:./cockroach
ssh: connect to host 34.41.170.41 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131720 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #129915 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test restore/online/workload=true/tpcc/8TB/gce/inc-count=0/nodes=10/cpus=8 failed: (test_runner.go:821).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64 ubuntu@34.135.41.214:./cockroach
ssh: connect to host 34.135.41.214 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131720 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #129915 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test sql-stats/mixed-version failed: (test_runner.go:821).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64 ubuntu@35.184.82.196:./cockroach
ssh: connect to host 35.184.82.196 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131720 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #129915 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/sequential/parts-start-kill-2 failed: (cluster.go:1877).Put: cluster.PutE: put "/tmp/jepsen1769170717/id_rsa.pub" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /tmp/jepsen1769170717/id_rsa.pub ubuntu@34.45.61.52:controller_id_rsa.pub: signal: killed
test artifacts and logs in: /artifacts/jepsen/sequential/parts-start-kill-2/cpu_arch=arm64/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131720 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #129915 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test failover/chaos/read-only/lease=expiration failed: (cluster.go:2397).Run: full command output in run_100559.078198164_n1-10_sudo-aptget-purge-y-.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/failover/chaos/read-only/lease=expiration/cpu_arch=arm64/run_1/ssh/ssh_100559.106817158_n8_sudo-aptget-purge-y-.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cluster.go:2397).Run: context canceled
(cluster.go:2397).Run: context canceled
(cluster.go:2397).Run: context canceled
test artifacts and logs in: /artifacts/failover/chaos/read-only/lease=expiration/cpu_arch=arm64/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131720 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #129915 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/bank/majority-ring-subcritical-skews failed: (cluster.go:1877).Put: cluster.PutE: put "/tmp/jepsen317463294/id_rsa.pub" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /tmp/jepsen317463294/id_rsa.pub ubuntu@34.67.224.20:controller_id_rsa.pub: signal: killed
test artifacts and logs in: /artifacts/jepsen/bank/majority-ring-subcritical-skews/cpu_arch=arm64/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131720 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #129915 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test loqrecovery/half-online/workload=movr/rangeSize=default failed: (cluster.go:2284).Start: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/loqrecovery/half-online/workload=movr/rangeSize=default/cpu_arch=arm64/run_1/ssh/ssh_121928.140174962_n4_upload-start-script.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/loqrecovery/half-online/workload=movr/rangeSize=default/cpu_arch=arm64/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131720 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #129915 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test hibernate-spatial failed: (test_runner.go:821).func4: cluster.PutE: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/_runner-logs/ssh/ssh_085031.319216516_n1_if-hostname-teamcity.log): TRANSIENT_ERROR(ssh_problem): exit status 255 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131720 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #129915 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/control failed: (test_runner.go:821).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64" failed: error persisted after 2 attempts: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64 ubuntu@35.188.198.187:./cockroach
Warning: Permanently added '35.188.198.187' (ECDSA) to the list of known hosts.
client_loop: send disconnect: Broken pipe
lost connection: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131720 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #129915 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test ledger/nodes=6/multi-az failed: (test_runner.go:821).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: error persisted after 2 attempts: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 ubuntu@34.132.189.203:./cockroach
Warning: Permanently added '34.132.189.203' (ECDSA) to the list of known hosts.
Connection reset by 34.132.189.203 port 22
lost connection: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131720 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #129915 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test acceptance/c2c failed: (test_runner.go:821).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64 ubuntu@104.198.144.53:./cockroach
ssh: connect to host 104.198.144.53 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131720 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #129915 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test allocbench/nodes=7/cpu=8/kv/r=0/ops=skew failed: (test_runner.go:821).func4: cluster.PutE: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/_runner-logs/ssh/ssh_055515.343992748_n1_if-hostname-teamcity.log): TRANSIENT_ERROR(ssh_problem): exit status 255 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131720 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #129915 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test cdc/crdb-chaos failed: (cluster.go:2284).Start: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/crdb-chaos/cpu_arch=arm64/run_1/ssh/ssh_061959.905463617_n2_upload-start-script.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/crdb-chaos/cpu_arch=arm64/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131720 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #129915 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test cdc/workload/kv0/nodes=5/cpu=16/ranges=100/control failed: (test_runner.go:821).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64 ubuntu@34.31.29.32:./cockroach
ssh: connect to host 34.31.29.32 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131720 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #129915 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test alterpk-bank failed: (test_runner.go:821).func4: cluster.PutE: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/_runner-logs/ssh/ssh_055545.001899676_n1_if-hostname-teamcity.log): TRANSIENT_ERROR(ssh_problem): exit status 255 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131720 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #129915 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test allocbench/nodes=7/cpu=8/kv/r=0/ops=skew failed: (test_runner.go:821).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64" failed: error persisted after 3 attempts: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64 ubuntu@:./cockroach
ssh: Could not resolve hostname : Name or service not known
lost connection: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131720 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #129915 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test import/mixed-versions failed: (test_runner.go:821).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: error persisted after 2 attempts: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 ubuntu@34.73.206.253:./cockroach
client_loop: send disconnect: Broken pipe
lost connection: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #132980 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131720 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #129915 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test acceptance/rapid-restart failed: (test_runner.go:821).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64 ubuntu@34.57.67.2:./cockroach
ssh: connect to host 34.57.67.2 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #132980 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131720 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #129915 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test acceptance/build-analyze failed: (test_runner.go:821).func4: cluster.PutE: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/_runner-logs/ssh/ssh_055524.863307478_n1_if-hostname-teamcity.log): TRANSIENT_ERROR(ssh_problem): exit status 255 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #132980 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131720 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #129915 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!