search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.18k stars 3.82k forks source link

roachtest: ssh_problem failed #112310

Open cockroach-teamcity opened 1 year ago

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ bd71370a4afb6cf2a7718a4f20ccba6a9436bdb6:

test cdc/workload/kv100/nodes=5/cpu=16/ranges=100/server=processor/protocol=mux/format=json/sink=null failed due to (cluster.go:2074).Start: one or more parallel execution failure(s): error persisted after 2 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100/server=processor/protocol=mux/format=json/sink=null/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7) See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-12162093/cdc-workload-kv100-nodes-5-cpu-16-ranges-100-server-processor-p/1697205954000/1697206259616)

Same failure on other branches

- #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-32350

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ bd71370a4afb6cf2a7718a4f20ccba6a9436bdb6:

test unoptimized-query-oracle/disable-rules=half/seed-multi-region failed due to (cluster.go:2113).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/unoptimized-query-oracle/disable-rules=half/seed-multi-region/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7) See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-12162093/unoptimized-query-oracle-disable-rules-half-seed-multi-region/1697206570060/1697208699460)

Same failure on other branches

- #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ bd71370a4afb6cf2a7718a4f20ccba6a9436bdb6:

test tpccbench/nodes=3/cpu=16/mt-shared-process failed due to (cluster.go:2074).Start: one or more parallel execution failure(s): error persisted after 2 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=16/mt-shared-process/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7) See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-12162093/tpccbench-nodes-3-cpu-16-mt-shared-process/1697198705607/1697209875947)

Same failure on other branches

- #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ dc24db6b7ed2f53d034ae9b8038ce08fc7d57542:

test tpcc/multiregion/survive=region/chaos=true failed due to (monitor.go:153).Wait: monitor failure: could not stop node :1-3: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/tpcc/multiregion/survive=region/chaos=true/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=true , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7) See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-12302441/tpcc-multiregion-survive-region-chaos-true/1698092483066/1698099067832)

Same failure on other branches

- #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ 348d11425a1184ce4de8a6f8dc85995cd3c653bc:

test import/tpcc/warehouses=4000/geo failed due to (cluster.go:2074).Start: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/import/tpcc/warehouses=4000/geo/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=true , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7) See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-12428970/import-tpcc-warehouses-4000-geo/1698775145323/1698775481004)

Same failure on other branches

- #113515 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc.FROZEN] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ c702128fa39c4aa84e810b898064fd2d66ace765:

test failover/partial/lease-leader failed due to (cluster.go:2239).Run: full command output in run_171217.690767084_n5_sudo-iptables-A-INPU.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2239).Run: context canceled
test artifacts and logs in: /artifacts/failover/partial/lease-leader/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=2 , ROACHTEST_encrypted=false , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7) See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-12647685/failover-partial-lease-leader/1699894409185/1699896112349)

Same failure on other branches

- #113515 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc.FROZEN] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 11 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ 9a13104643153a4233d626e0d398e4b7175674bd:

test cdc/workload/kv100/nodes=5/cpu=16/ranges=100/control failed due to (cluster.go:2046).Start: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100/control/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 9 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ 895000ff72c7a40a075dca3ec4687028ebe468b9:

test failover/non-system/crash/lease=expiration failed: (cluster.go:2187).Start: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/failover/non-system/crash/lease=expiration/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 9 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ 895000ff72c7a40a075dca3ec4687028ebe468b9:

test sysbench/oltp_delete/nodes=3/cpu=32/conc=256 failed: (monitor.go:153).Wait: monitor failure: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/sysbench/oltp_delete/nodes=3/cpu=32/conc=256/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 9 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ 895000ff72c7a40a075dca3ec4687028ebe468b9:

test failover/non-system/blackhole/lease=expiration failed: (cluster.go:2301).Run: full command output in run_162818.375871235_n6_sudo-iptables-F.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2301).Run: context canceled
test artifacts and logs in: /artifacts/failover/non-system/blackhole/lease=expiration/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 9 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ 895000ff72c7a40a075dca3ec4687028ebe468b9:

test failover/non-system/blackhole-send/lease=expiration failed: (cluster.go:2301).Run: full command output in run_162823.829930868_n6_sudo-iptables-A-OUTP.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2301).Run: context canceled
test artifacts and logs in: /artifacts/failover/non-system/blackhole-send/lease=expiration/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 9 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ 895000ff72c7a40a075dca3ec4687028ebe468b9:

test failover/non-system/blackhole-send failed: (cluster.go:2301).Run: full command output in run_162819.394431270_n5_sudo-iptables-F.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2301).Run: context canceled
test artifacts and logs in: /artifacts/failover/non-system/blackhole-send/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 9 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ 895000ff72c7a40a075dca3ec4687028ebe468b9:

test failover/non-system/crash failed: (cluster.go:2187).Start: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/failover/non-system/crash/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 9 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ 895000ff72c7a40a075dca3ec4687028ebe468b9:

test unoptimized-query-oracle/disable-rules=half/rand-tables failed: (cluster.go:2238).Stop: cluster.StopE: one or more parallel execution failure(s): error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/unoptimized-query-oracle/disable-rules=half/rand-tables/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 9 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ 895000ff72c7a40a075dca3ec4687028ebe468b9:

test failover/chaos/read-write failed: (cluster.go:2187).Start: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2301).Run: context canceled
(cluster.go:2301).Run: context canceled
(cluster.go:2301).Run: context canceled
(cluster.go:2301).Run: context canceled
test artifacts and logs in: /artifacts/failover/chaos/read-write/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 9 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ 895000ff72c7a40a075dca3ec4687028ebe468b9:

test failover/chaos/read-write/lease=expiration failed: (cluster.go:2265).Signal: cluster.Signal: one or more parallel execution failure(s): error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2301).Run: context canceled
(cluster.go:2301).Run: context canceled
(cluster.go:2301).Run: context canceled
(cluster.go:2301).Run: context canceled
test artifacts and logs in: /artifacts/failover/chaos/read-write/lease=expiration/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 9 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ 895000ff72c7a40a075dca3ec4687028ebe468b9:

test clearrange/checks=false failed: (cluster.go:2238).Stop: cluster.StopE: one or more parallel execution failure(s): error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/clearrange/checks=false/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 9 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ 895000ff72c7a40a075dca3ec4687028ebe468b9:

test clearrange/checks=true failed: (cluster.go:2238).Stop: cluster.StopE: one or more parallel execution failure(s): error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/clearrange/checks=true/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 9 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ 895000ff72c7a40a075dca3ec4687028ebe468b9:

test schemachange/index/tpcc/w=800 failed: (cluster.go:2301).Run: full command output in run_174030.279875486_n5_cockroach-workload-c.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/schemachange/index/tpcc/w=800/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 9 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ 895000ff72c7a40a075dca3ec4687028ebe468b9:

test schemachange/during/tpcc failed: (cluster.go:2301).Run: full command output in run_175009.659276868_n5_cockroach-workload-c.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/schemachange/during/tpcc/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ 3ec53a441fe4f84d553f1dc206b15d5923e32912:

test kv0/enc=false/nodes=3/cpu=32/splt=0 failed: (cluster.go:2221).Start: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/cpu=32/splt=0/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

- #120494 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.3-rc] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ e91f173f368521c31227b229046b28a007b73d6b:

test jepsen/multi-register/start-kill-2 failed: (jepsen.go:332).func2: full command output in run_142522.861308474_n6_tail-n-100-mntdata1j.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/jepsen/multi-register/start-kill-2/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #120757 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.17-rc] - #120494 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.3-rc] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ e45dd230a094654c32428a7b8f65fad194bef44d:

test cdc/initial-scan failed: (cluster.go:2226).Start: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/cdc/initial-scan/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #120971 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.17-custom-v2] - #120757 roachtest: ssh_problem failed [O-roachtest O-robot X-infra-flake branch-release-23.1.17-rc] - #120494 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.3-rc] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ ebc552858bbc8bd327c112a09d7f0646c5912b84:

test jepsen/bank/majority-ring-start-kill-2 failed: (jepsen.go:429).runJepsen: full command output in run_163829.427716189_n6_bash-e-c-cd-mntdata1.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/jepsen/bank/majority-ring-start-kill-2/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #120494 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.3-rc] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ 57564f6429081f53830536a6fdb26e284c4b3512:

test tpcc/mixed-headroom/n5cpu16 failed: (mixedversion.go:592).Run: mixed-version test failure while running step 38 (restart node 4 with binary version release-23.2): _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/tpcc/mixed-headroom/n5cpu16/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ 57564f6429081f53830536a6fdb26e284c4b3512:

test tpcc/headroom/isolation-level=read-committed/n4cpu16 failed: (cluster.go:2344).Run: full command output in run_212600.777447252_n4_cockroach-workload-c.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/tpcc/headroom/isolation-level=read-committed/n4cpu16/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ 57564f6429081f53830536a6fdb26e284c4b3512:

test tpcc/headroom/n4cpu16 failed: (cluster.go:2344).Run: full command output in run_212818.160461384_n4_cockroach-workload-c.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained under ssh/): SSH_PROBLEM: exit status 255
test artifacts and logs in: /artifacts/tpcc/headroom/n4cpu16/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ 7d5f4ef0d54fe03f947c500c68f9859ec54376e7:

test cdc/kafka-oauth failed: (cdc.go:2055).configureHydraOauth: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/kafka-oauth/run_1/ssh/ssh_105600.795504809_n4_homeubuntuhydra-crea.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:350).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/kafka-oauth/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ 7d5f4ef0d54fe03f947c500c68f9859ec54376e7:

test cdc/kafka-oauth failed: (cdc.go:2055).configureHydraOauth: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/kafka-oauth/run_1/ssh/ssh_103016.424294810_n4_homeubuntuhydra-crea.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:350).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/kafka-oauth/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ 7d5f4ef0d54fe03f947c500c68f9859ec54376e7:

test cdc/kafka-oauth failed: (cdc.go:2055).configureHydraOauth: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/kafka-oauth/run_1/ssh/ssh_104251.421411594_n4_homeubuntuhydra-crea.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:350).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/kafka-oauth/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ 7ff2a502de01a57b490e6eed1f7fd64bc27b2f64:

test cdc/kafka-oauth failed: (cdc.go:2055).configureHydraOauth: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/kafka-oauth/run_1/ssh/ssh_103313.758509847_n4_homeubuntuhydra-crea.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:350).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/kafka-oauth/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ 0a9dc5d57a39356dda4775d0a131e1e4f58b9058:

test cdc/kafka-oauth failed: (cdc.go:2055).configureHydraOauth: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/kafka-oauth/run_1/ssh/ssh_104556.823744262_n4_homeubuntuhydra-crea.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:350).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/kafka-oauth/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ ddf0e4a9fecb2eed4a97e7e0a884af831f702b16:

test cdc/kafka-oauth failed: (cdc.go:2085).configureHydraOauth: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/kafka-oauth/run_1/ssh/ssh_104811.171874438_n4_homeubuntuhydra-crea.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:350).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/kafka-oauth/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ e7576d10aaaa61940cebf8bd8f157ce0e1651e7b:

test cdc/kafka-oauth failed: (cdc.go:2085).configureHydraOauth: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/kafka-oauth/run_1/ssh/ssh_104926.960284774_n4_homeubuntuhydra-crea.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:350).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/kafka-oauth/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ d91ca866b265a837f832bc93b59481847d721f18:

test cdc/kafka-oauth failed: (cdc.go:2085).configureHydraOauth: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/kafka-oauth/run_1/ssh/ssh_105650.922789378_n4_homeubuntuhydra-crea.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:350).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/kafka-oauth/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ d91ca866b265a837f832bc93b59481847d721f18:

test cdc/kafka-oauth failed: (cdc.go:2085).configureHydraOauth: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/kafka-oauth/run_1/ssh/ssh_054245.157651215_n4_homeubuntuhydra-crea.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:350).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/kafka-oauth/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ d91ca866b265a837f832bc93b59481847d721f18:

test cdc/kafka-oauth failed: (cdc.go:2085).configureHydraOauth: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/kafka-oauth/run_1/ssh/ssh_110331.325855354_n4_homeubuntuhydra-crea.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:350).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/kafka-oauth/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ d91ca866b265a837f832bc93b59481847d721f18:

test cdc/kafka-oauth failed: (cdc.go:2085).configureHydraOauth: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/kafka-oauth/run_1/ssh/ssh_105213.238468615_n4_homeubuntuhydra-crea.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:350).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/kafka-oauth/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #122781 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.5-rc] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ a2f7a92877c35dab238153086e817438034da61c:

test cdc/kafka-oauth failed: (cdc.go:2085).configureHydraOauth: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/kafka-oauth/run_1/ssh/ssh_104743.587401298_n4_homeubuntuhydra-crea.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:350).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/kafka-oauth/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #122781 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.5-rc] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ 8a37c9bfbac9da07c899d4617dbbdb93dd7a5bae:

test cdc/kafka-oauth failed: (cdc.go:2085).configureHydraOauth: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/kafka-oauth/run_1/ssh/ssh_110029.162274588_n4_homeubuntuhydra-crea.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:350).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/kafka-oauth/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #122781 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.5-rc] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ 845bc49a72ab2b2b13d5a57b4134f99aa26f1b7b:

test cdc/kafka-oauth failed: (cdc.go:2085).configureHydraOauth: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/kafka-oauth/run_1/ssh/ssh_105404.893867926_n4_homeubuntuhydra-crea.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:350).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/kafka-oauth/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #122781 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.5-rc] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ 845bc49a72ab2b2b13d5a57b4134f99aa26f1b7b:

test failover/non-system/crash failed: (cluster.go:2281).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/failover/non-system/crash/run_1/ssh/ssh_151140.015333242_n4_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/failover/non-system/crash/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #122781 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.5-rc] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ 3bc5df4b5677c248941dfe5042908526655c9282:

test cdc/kafka-oauth failed: (cdc.go:2085).configureHydraOauth: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/kafka-oauth/run_1/ssh/ssh_104731.873237630_n4_homeubuntuhydra-crea.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:350).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/kafka-oauth/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #122781 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.5-rc] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ ba4f669960bf4518c222ef1a31609b442602ab68:

test cdc/kafka-oauth failed: (cdc.go:2085).configureHydraOauth: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/kafka-oauth/run_1/ssh/ssh_104257.688194845_n4_homeubuntuhydra-crea.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:350).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/kafka-oauth/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #122781 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.5-rc] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ ba4f669960bf4518c222ef1a31609b442602ab68:

test cdc/kafka-oauth failed: (cdc.go:2085).configureHydraOauth: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/kafka-oauth/run_1/ssh/ssh_110053.537672763_n4_homeubuntuhydra-crea.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:350).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/kafka-oauth/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #122781 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.5-rc] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 6 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ ba4f669960bf4518c222ef1a31609b442602ab68:

test cdc/kafka-oauth failed: (cdc.go:2085).configureHydraOauth: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/kafka-oauth/run_1/ssh/ssh_104450.952514473_n4_homeubuntuhydra-crea.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:350).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/kafka-oauth/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #122781 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.5-rc] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 6 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ ee678a350315c5268febaa93b860ae7415438667:

test cdc/kafka-oauth failed: (cdc.go:2085).configureHydraOauth: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/kafka-oauth/run_1/ssh/ssh_103258.377431923_n4_homeubuntuhydra-crea.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:350).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/kafka-oauth/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #122781 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.5-rc] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 6 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ b3ff5193e7f24238d25e5e06fc95e256f7741a4b:

test cdc/kafka-oauth failed: (cdc.go:2085).configureHydraOauth: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/kafka-oauth/run_1/ssh/ssh_102712.972200559_n4_homeubuntuhydra-crea.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:350).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/kafka-oauth/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #122781 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.5-rc] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 6 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ 048468b1eeb18667687cc5470acb8f68da77982b:

test cdc/kafka-oauth failed: (cdc.go:2085).configureHydraOauth: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/kafka-oauth/run_1/ssh/ssh_105047.458944832_n4_homeubuntuhydra-crea.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:350).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/kafka-oauth/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #122781 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.5-rc] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 6 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ 109656a0d65dd2e08935581f6876e9a70e110a24:

test cdc/kafka-oauth failed: (cdc.go:2085).configureHydraOauth: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/kafka-oauth/run_1/ssh/ssh_105030.241442031_n4_homeubuntuhydra-crea.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:350).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/kafka-oauth/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123538 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #122781 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.5-rc] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 6 months ago

roachtest.ssh_problem failed with artifacts on release-23.2 @ a36883a03ce0b30af0c025ff2b0de2f3845ad8d7:

test cdc/kafka-oauth failed: (cdc.go:2085).configureHydraOauth: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/kafka-oauth/run_1/ssh/ssh_102625.572849805_n4_homeubuntuhydra-crea.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:350).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/kafka-oauth/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #123538 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.0-rc] - #122781 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.5-rc] - #122095 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #119192 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.1-rc] - #112280 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108264 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1] - #90695 roachtest: ssh_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!