roachtest: ssh_problem failed

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.1.3-rc @ 984bb6088542c5ef9b9ce8c44af6d980bd7db6f0:

test cdc/workload/kv100/nodes=5/cpu=16/ranges=100/server=scheduler/protocol=mux/format=json/sink=null failed: (test_runner.go:797).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64" failed: error persisted after 2 attempts: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64 ubuntu@34.27.178.164:./cockroach
Warning: Permanently added '34.27.178.164' (ECDSA) to the list of known hosts.
client_loop: send disconnect: Broken pipe
lost connection: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126478 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125985 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124905 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #123868 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-40383

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test scrub/index-only/tpcc/w=100 failed: (test_runner.go:797).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 ubuntu@34.73.4.227:./cockroach
ssh: connect to host 34.73.4.227 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126478 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125985 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124905 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #123868 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.1.3-rc @ 81c0cda194e1a80e21988ad307d76c14a3a5e164:

test allocbench/nodes=7/cpu=8/kv/r=95/access=skew failed: (test_runner.go:797).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 ubuntu@35.196.151.160:./cockroach
Warning: Permanently added '35.196.151.160' (ECDSA) to the list of known hosts.
ubuntu@35.196.151.160: Permission denied (publickey).
lost connection: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126478 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125985 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124905 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #123868 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.1.3-rc @ 9e10212477fde97f55ea4fff01797288c836575c:

test c2c/initialscan/kv0 failed: (test_runner.go:797).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64" failed: error persisted after 2 attempts: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 ubuntu@34.138.167.86:./cockroach
Warning: Permanently added '34.138.167.86' (ECDSA) to the list of known hosts.
client_loop: send disconnect: Broken pipe
lost connection: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126478 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125985 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124905 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #123868 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.1.3-rc @ 9e10212477fde97f55ea4fff01797288c836575c:

test decommissionBench/nodes=4/warehouses=1000 failed: (test_runner.go:797).func4: cluster.PutE: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/_runner-logs/ssh/ssh_080905.726046709_n1_if-hostname-teamcity.log): TRANSIENT_ERROR(ssh_problem): exit status 255 [owner=test-eng]

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128147 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126478 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125985 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124905 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #123868 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.1.3-rc @ 9e10212477fde97f55ea4fff01797288c836575c:

test tpccbench/nodes=3/cpu=16/enc=true failed: (test_runner.go:797).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64 ubuntu@34.67.196.254:./cockroach
ssh: connect to host 34.67.196.254 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128147 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126478 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125985 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124905 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #123868 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.ssh_problem failed with artifacts on release-24.1.3-rc @ 9e10212477fde97f55ea4fff01797288c836575c:

test allocbench/nodes=7/cpu=8/kv/r=0/access=skew failed: (test_runner.go:797).func4: cluster.PutE: put "/go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64" failed: TRANSIENT_ERROR(ssh_problem): ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-arm64 ubuntu@34.41.243.191:./cockroach
ssh: connect to host 34.41.243.191 port 22: Connection timed out
lost connection: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128147 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127118 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126478 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125985 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #124905 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.1-rc] - #123868 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112310 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2]

This test on roachdash | Improve this report!

cockroachdb / cockroach

roachtest: ssh_problem failed #127328