search

cockroachdb / cockroach

CockroachDB - the open source, cloud-native distributed SQL database.
https://www.cockroachlabs.com
Other
29.49k stars 3.7k forks source link

roachtest: ssh_problem failed #124905

Open cockroach-teamcity opened 1 month ago

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.1.1-rc @ 827b522c24d903ec130ea4aa48dd0af6687eaf6d:

test failover/non-system/blackhole-send failed: (cluster.go:2398).Run: full command output in run_101904.894066649_n4_sudo-iptables-A-OUTP.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/failover/non-system/blackhole-send/run_1/ssh/ssh_101904.914434616_n4_sudo-iptables-A-OUTP.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2398).Run: context canceled
test artifacts and logs in: /artifacts/failover/non-system/blackhole-send/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124710 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.22-rc] - #124707 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.6-rc] - #123868 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #123538 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #122781 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.5-rc] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-39143

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.1.1-rc @ 827b522c24d903ec130ea4aa48dd0af6687eaf6d:

test tpccbench/nodes=9/cpu=4/multi-region failed: (test_runner.go:797).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 ubuntu@34.83.67.204:./cockroach
ssh: connect to host 34.83.67.204 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124710 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.22-rc] - #124707 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.6-rc] - #123868 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #123538 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #122781 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.5-rc] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

roachtest.ssh_problem failed with artifacts on release-24.1.1-rc @ fb26da766a8f98eec602de55fdf6c813ee560069:

test ledger/nodes=6/multi-az failed: (test_runner.go:797).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 ubuntu@104.154.207.10:./cockroach
ssh: connect to host 104.154.207.10 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124710 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.22-rc] - #124707 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.6-rc] - #123868 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #123538 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #122781 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.5-rc] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.ssh_problem failed with artifacts on release-24.1.1-rc @ 044088e615f601556c78ce06d8e1a48a2adc6607:

test schemachange/index/tpcc/w=100 failed: (test_runner.go:797).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 ubuntu@35.185.30.238:./cockroach
ssh: connect to host 35.185.30.238 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124710 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.22-rc] - #124707 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.6-rc] - #123868 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #123538 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #122781 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.5-rc] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.ssh_problem failed with artifacts on release-24.1.1-rc @ 484f8e2f4d3cb23582416944d79eab72ff42a6a1:

test loqrecovery/workload=movr/rangeSize=2mb failed: (test_runner.go:797).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 ubuntu@35.231.83.215:./cockroach
ssh: connect to host 35.231.83.215 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124707 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.6-rc] - #123868 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!