roachtest: ssh_problem failed

cockroachdb / cockroach

CockroachDB - the open source, cloud-native distributed SQL database.

https://www.cockroachlabs.com

Other

29.19k stars 3.65k forks source link

roachtest: ssh_problem failed #90695

Open cockroach-teamcity opened 1 year ago

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ 1b1c8da55be48c174b7b370b305f42622546209f:

test schemachange/random-load failed due to test artifacts and logs in: /artifacts/schemachange/random-load/run_1
(test_impl.go:291).Fatal: output in run_125153.622625054_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-20896

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ 0c1c3e7777b28a30ebe41428fb173f0156e8968c:

test schemachange/random-load failed due to test artifacts and logs in: /artifacts/schemachange/random-load/run_1
(test_impl.go:291).Fatal: output in run_133500.318499865_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ 2d926e68000df659f282d4e4477329867b9a3323:

test schemachange/random-load failed due to test artifacts and logs in: /artifacts/schemachange/random-load/run_1
(test_impl.go:291).Fatal: output in run_140012.557459920_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: SSH_PROBLEM: exit status 255
(test_impl.go:314).Errorf: test timed out (0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

renatolabs commented 1 year ago

Some notes about the failure above:

Error classifiction

The failure was classified as an SSH flake but, in reality, the test timed out. You can see it in the message above, as well as by checking the test logs: the workload ran for 10 hours.

To consider (cc @smg260):

Improve the test timed out message to properly report time spent in the test. This could be done by reusing the default timeout [1] or (even better) displaying the actual duration (for example, by exposing duration [2]).
Improving SSH-flake detection. It seems that when the test runner intentionally stops processes started by roachprod, there's a chance we'll get a 255 error. There's probably a better way to deal with this. One quick and dirty, short term solution would be to not mark a failure an SSH-flake if one of the errors indicates that the test timed out.

[1] https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/test_runner.go#L877 [2] https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/test_impl.go#L373-L375

Actual test failure (workload timeout)

The workload ran for 10h and never finished. We're passing --max-ops 5000, so it technically should terminate when we reach that number of operations (+ the number of concurrent workers which in this case is 20).

The logs generated by the workload are fairly large (280MB). If we grep for a field in the JSON printed when a worker performs its random number of ops, I think we are able to get a count of the actual number of schema change ops the workload ran:

% rg expectedExecErrors run_140012.557459920_n1_workload_run_schemachange.log | wc -l
    5643

This looks suspicious to me. This number shouldn't be > 5019 (5000 maxOps + enough ops for the remaining 19 workers to realize they should stop after maxOps is reached).

@fqazi Thoughts? Anything to be done here?

fqazi commented 1 year ago

Let me dig into this, it looks like we are some how kept generating ops, which is super weird.

fqazi commented 1 year ago

@renatolabs It looks like we ran into an infinite loop inside randParentColumnForFkRelation. I'll get a patch out for it shortly

smg260 commented 1 year ago

Thanks @renatolabs .

Improve the test timed out message to properly report time spent in the test. This could be done by reusing the default timeout [1] or (even better) displaying the actual duration (for example, by exposing duration [2]).

test.log shows the correct actual timeout of 10h (this is the default, and incidentally quite high)

23:59:09 test_runner.go:936: test timed out after 10h0m0s; check __stacks.log and CRDB logs for goroutine dumps

The 0s you see is reported at the time we finally call t.Errorf and it uses the timeout in the TestSpec, which defaults to 0. This should probably use the actual timeout as in the above print.

Improving SSH-flake detection. It seems that when the test runner intentionally stops processes started by roachprod, there's a chance we'll get a 255 error. There's probably a better way to deal with this. One quick and dirty, short term solution would be to not mark a failure an SSH-flake if one of the errors indicates that the test timed out.

Agreed. However this is a symptom of us delaying the call to t.Errorf on timeout until teardown as completed [1], [2]. When the nodes are stopped, the long running ssh command will fail, resulting in a 255 exit code, and the subsequent reporting of that error - but since we delayed reporting the real cause (timeout), this error becomes the primary and gets reported as such. We could certainly look at prioritising the time out error.

[1] - https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/test_runner.go#L933 [2] - https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/test_runner.go#L1051

fqazi commented 1 year ago

There is a secondary bug here inside cockroache's DROP SCHEMA implementation too, which makes this worse. I'll band-aid the infinite loop first since a faster failure is better here.

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:

test jepsen/bank-multitable/majority-ring-start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/bank-multitable/majority-ring-start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_104333.097775667_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.1.137 -n 10.142.0.45 -n 10.142.1.138 -n 10.142.1.127 -n 10.142.1.162 \
    --test bank-multitable --nemesis majority-ring --nemesis2 start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:

test jepsen/bank-multitable/start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/bank-multitable/start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_105547.897432309_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.1.174 -n 10.142.0.224 -n 10.142.1.98 -n 10.142.1.172 -n 10.142.0.217 \
    --test bank-multitable --nemesis start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:

test jepsen/bank/majority-ring-start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/bank/majority-ring-start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_105853.056711258_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.1.166 -n 10.142.1.167 -n 10.142.1.163 -n 10.142.1.171 -n 10.142.1.168 \
    --test bank --nemesis majority-ring --nemesis2 start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:

test jepsen/bank/start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/bank/start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_111113.812210581_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.1.116 -n 10.142.1.114 -n 10.142.1.125 -n 10.142.1.124 -n 10.142.1.126 \
    --test bank --nemesis start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:

test jepsen/g2/majority-ring-start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/g2/majority-ring-start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_113040.476678420_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.1.194 -n 10.142.1.199 -n 10.142.1.193 -n 10.142.1.192 -n 10.142.1.198 \
    --test g2 --nemesis majority-ring --nemesis2 start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:

test jepsen/g2/start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/g2/start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_113653.263800916_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.1.159 -n 10.142.1.157 -n 10.142.1.120 -n 10.142.1.158 -n 10.142.1.155 \
    --test g2 --nemesis start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:

test jepsen/multi-register/majority-ring failed due to test artifacts and logs in: /artifacts/jepsen/multi-register/majority-ring/run_1
(test_impl.go:291).Fatal: output in run_120825.027668848_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.0.152 -n 10.142.0.91 -n 10.142.0.96 -n 10.142.0.145 -n 10.142.0.98 \
    --test multi-register --nemesis majority-ring > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:

test jepsen/multi-register/parts-start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/multi-register/parts-start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_121117.835152541_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.0.95 -n 10.142.0.236 -n 10.142.0.110 -n 10.142.0.154 -n 10.142.0.112 \
    --test multi-register --nemesis parts --nemesis2 start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:

test jepsen/multi-register/split failed due to test artifacts and logs in: /artifacts/jepsen/multi-register/split/run_1
(test_impl.go:291).Fatal: output in run_121247.132219570_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.0.82 -n 10.142.0.86 -n 10.142.0.45 -n 10.142.0.89 -n 10.142.0.70 \
    --test multi-register --nemesis split > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:

test jepsen/monotonic/majority-ring-start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/monotonic/majority-ring-start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_115128.494652605_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.1.96 -n 10.142.0.48 -n 10.142.0.7 -n 10.142.0.9 -n 10.142.1.122 \
    --test monotonic --nemesis majority-ring --nemesis2 start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:

test jepsen/multi-register/majority-ring-start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/multi-register/majority-ring-start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_121653.485694934_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.1.97 -n 10.142.1.99 -n 10.142.1.91 -n 10.142.0.102 -n 10.142.1.98 \
    --test multi-register --nemesis majority-ring --nemesis2 start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:

test jepsen/multi-register/start-stop-2 failed due to test artifacts and logs in: /artifacts/jepsen/multi-register/start-stop-2/run_1
(test_impl.go:291).Fatal: output in run_122045.073594683_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.1.124 -n 10.142.1.149 -n 10.142.1.116 -n 10.142.1.125 -n 10.142.1.133 \
    --test multi-register --nemesis start-stop-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:

test jepsen/multi-register/start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/multi-register/start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_122045.845345901_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.1.87 -n 10.142.0.109 -n 10.142.1.107 -n 10.142.1.89 -n 10.142.1.109 \
    --test multi-register --nemesis start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:

test jepsen/register/majority-ring failed due to test artifacts and logs in: /artifacts/jepsen/register/majority-ring/run_1
(test_impl.go:291).Fatal: output in run_122214.185656775_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.1.52 -n 10.142.1.104 -n 10.142.1.151 -n 10.142.1.160 -n 10.142.1.150 \
    --test register --nemesis majority-ring > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:

test jepsen/multi-register/strobe-skews failed due to test artifacts and logs in: /artifacts/jepsen/multi-register/strobe-skews/run_1
(test_impl.go:291).Fatal: output in run_122222.681493684_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.0.188 -n 10.142.1.42 -n 10.142.0.148 -n 10.142.1.19 -n 10.142.0.177 \
    --test multi-register --nemesis strobe-skews > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:

test jepsen/register/majority-ring-start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/register/majority-ring-start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_122730.701410140_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.0.237 -n 10.142.0.221 -n 10.142.1.46 -n 10.142.1.62 -n 10.142.1.40 \
    --test register --nemesis majority-ring --nemesis2 start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:

test jepsen/register/parts-start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/register/parts-start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_122912.286840348_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.0.222 -n 10.142.0.249 -n 10.142.1.5 -n 10.142.0.46 -n 10.142.0.253 \
    --test register --nemesis parts --nemesis2 start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:

test jepsen/register/split failed due to test artifacts and logs in: /artifacts/jepsen/register/split/run_1
(test_impl.go:291).Fatal: output in run_123006.706704713_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.1.55 -n 10.142.1.119 -n 10.142.1.9 -n 10.142.1.61 -n 10.142.1.21 \
    --test register --nemesis split > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:

test jepsen/register/start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/register/start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_123038.685621713_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.0.62 -n 10.142.0.43 -n 10.142.0.96 -n 10.142.0.169 -n 10.142.0.51 \
    --test register --nemesis start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:

test jepsen/monotonic/start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/monotonic/start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_120815.661539339_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.1.231 -n 10.142.1.228 -n 10.142.0.78 -n 10.142.1.229 -n 10.142.0.2 \
    --test monotonic --nemesis start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:

test jepsen/register/start-stop-2 failed due to test artifacts and logs in: /artifacts/jepsen/register/start-stop-2/run_1
(test_impl.go:291).Fatal: output in run_123126.701248194_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.1.71 -n 10.142.0.196 -n 10.142.0.172 -n 10.142.0.191 -n 10.142.1.128 \
    --test register --nemesis start-stop-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:

test jepsen/register/strobe-skews failed due to test artifacts and logs in: /artifacts/jepsen/register/strobe-skews/run_1
(test_impl.go:291).Fatal: output in run_123604.421276100_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.0.91 -n 10.142.0.95 -n 10.142.0.86 -n 10.142.0.89 -n 10.142.0.57 \
    --test register --nemesis strobe-skews > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:

test jepsen/sequential/majority-ring-start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/sequential/majority-ring-start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_123834.605695740_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.1.94 -n 10.142.1.59 -n 10.142.0.236 -n 10.142.0.61 -n 10.142.0.107 \
    --test sequential --nemesis majority-ring --nemesis2 start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:

test jepsen/sequential/start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/sequential/start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_123905.509212127_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.1.112 -n 10.142.1.100 -n 10.142.1.97 -n 10.142.1.98 -n 10.142.1.96 \
    --test sequential --nemesis start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ cfb5ae9a96e1770daa4aef1615a46e212b561a84:

test jepsen/bank-multitable/majority-ring-start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/bank-multitable/majority-ring-start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_105609.350642108_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.1.113 -n 10.142.1.109 -n 10.142.1.104 -n 10.142.1.116 -n 10.142.1.115 \
    --test bank-multitable --nemesis majority-ring --nemesis2 start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ cfb5ae9a96e1770daa4aef1615a46e212b561a84:

test jepsen/bank-multitable/start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/bank-multitable/start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_110241.885917351_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.0.152 -n 10.142.1.103 -n 10.142.1.96 -n 10.142.1.99 -n 10.142.0.173 \
    --test bank-multitable --nemesis start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ cfb5ae9a96e1770daa4aef1615a46e212b561a84:

test jepsen/bank/majority-ring-start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/bank/majority-ring-start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_112343.141633580_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.1.101 -n 10.142.0.131 -n 10.142.1.93 -n 10.142.1.114 -n 10.142.1.118 \
    --test bank --nemesis majority-ring --nemesis2 start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ cfb5ae9a96e1770daa4aef1615a46e212b561a84:

test jepsen/bank/start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/bank/start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_114113.417706230_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.1.49 -n 10.142.1.7 -n 10.142.0.136 -n 10.142.0.123 -n 10.142.1.86 \
    --test bank --nemesis start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ cfb5ae9a96e1770daa4aef1615a46e212b561a84:

test jepsen/g2/parts-start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/g2/parts-start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_120100.506371579_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.1.123 -n 10.142.1.124 -n 10.142.1.122 -n 10.142.1.95 -n 10.142.1.125 \
    --test g2 --nemesis parts --nemesis2 start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ cfb5ae9a96e1770daa4aef1615a46e212b561a84:

test jepsen/g2/majority-ring-start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/g2/majority-ring-start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_120443.043215478_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.1.153 -n 10.142.1.148 -n 10.142.1.224 -n 10.142.1.155 -n 10.142.1.154 \
    --test g2 --nemesis majority-ring --nemesis2 start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ cfb5ae9a96e1770daa4aef1615a46e212b561a84:

test jepsen/multi-register/majority-ring failed due to test artifacts and logs in: /artifacts/jepsen/multi-register/majority-ring/run_1
(test_impl.go:291).Fatal: output in run_123127.909132960_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.0.22 -n 10.142.0.68 -n 10.142.0.64 -n 10.142.0.69 -n 10.142.0.76 \
    --test multi-register --nemesis majority-ring > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ cfb5ae9a96e1770daa4aef1615a46e212b561a84:

test jepsen/g2/start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/g2/start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_121213.070831897_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.0.31 -n 10.142.1.97 -n 10.142.1.81 -n 10.142.1.149 -n 10.142.0.23 \
    --test g2 --nemesis start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ cfb5ae9a96e1770daa4aef1615a46e212b561a84:

test jepsen/multi-register/start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/multi-register/start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_123748.305461844_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.1.230 -n 10.142.1.232 -n 10.142.1.231 -n 10.142.0.170 -n 10.142.1.99 \
    --test multi-register --nemesis start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ cfb5ae9a96e1770daa4aef1615a46e212b561a84:

test jepsen/multi-register/strobe-skews failed due to test artifacts and logs in: /artifacts/jepsen/multi-register/strobe-skews/run_1
(test_impl.go:291).Fatal: output in run_123937.024923841_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.1.200 -n 10.142.1.177 -n 10.142.1.212 -n 10.142.1.198 -n 10.142.0.38 \
    --test multi-register --nemesis strobe-skews > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ cfb5ae9a96e1770daa4aef1615a46e212b561a84:

test jepsen/multi-register/majority-ring-start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/multi-register/majority-ring-start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_124050.383074603_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.0.107 -n 10.142.1.89 -n 10.142.0.128 -n 10.142.1.118 -n 10.142.0.226 \
    --test multi-register --nemesis majority-ring --nemesis2 start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ cfb5ae9a96e1770daa4aef1615a46e212b561a84:

test jepsen/register/parts-start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/register/parts-start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_124337.428052325_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.1.71 -n 10.142.1.72 -n 10.142.1.114 -n 10.142.0.146 -n 10.142.1.74 \
    --test register --nemesis parts --nemesis2 start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ cfb5ae9a96e1770daa4aef1615a46e212b561a84:

test jepsen/multi-register/parts-start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/multi-register/parts-start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_124347.475207189_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.1.169 -n 10.142.0.136 -n 10.142.1.86 -n 10.142.0.174 -n 10.142.1.176 \
    --test multi-register --nemesis parts --nemesis2 start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ cfb5ae9a96e1770daa4aef1615a46e212b561a84:

test jepsen/multi-register/split failed due to test artifacts and logs in: /artifacts/jepsen/multi-register/split/run_1
(test_impl.go:291).Fatal: output in run_124422.658713782_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.1.197 -n 10.142.1.213 -n 10.142.1.209 -n 10.142.1.179 -n 10.142.1.204 \
    --test multi-register --nemesis split > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ cfb5ae9a96e1770daa4aef1615a46e212b561a84:

test jepsen/multi-register/start-stop-2 failed due to test artifacts and logs in: /artifacts/jepsen/multi-register/start-stop-2/run_1
(test_impl.go:291).Fatal: output in run_124616.099175354_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.1.235 -n 10.142.0.24 -n 10.142.1.215 -n 10.142.1.234 -n 10.142.1.76 \
    --test multi-register --nemesis start-stop-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ cfb5ae9a96e1770daa4aef1615a46e212b561a84:

test jepsen/monotonic/majority-ring-start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/monotonic/majority-ring-start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_122454.467002073_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.1.196 -n 10.142.1.180 -n 10.142.1.133 -n 10.142.1.132 -n 10.142.1.201 \
    --test monotonic --nemesis majority-ring --nemesis2 start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ cfb5ae9a96e1770daa4aef1615a46e212b561a84:

test jepsen/register/majority-ring failed due to test artifacts and logs in: /artifacts/jepsen/register/majority-ring/run_1
(test_impl.go:291).Fatal: output in run_124803.238877536_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.0.175 -n 10.142.0.168 -n 10.142.0.164 -n 10.142.0.195 -n 10.142.0.187 \
    --test register --nemesis majority-ring > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ cfb5ae9a96e1770daa4aef1615a46e212b561a84:

test jepsen/register/majority-ring-start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/register/majority-ring-start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_125016.930878369_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.0.72 -n 10.142.0.11 -n 10.142.0.66 -n 10.142.0.8 -n 10.142.0.159 \
    --test register --nemesis majority-ring --nemesis2 start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.ssh_problem failed with artifacts on master @ cfb5ae9a96e1770daa4aef1615a46e212b561a84:

test jepsen/register/split failed due to test artifacts and logs in: /artifacts/jepsen/register/split/run_1
(test_impl.go:291).Fatal: output in run_125207.738651168_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
    --tarball file://${PWD}/cockroach.tgz \
    --username ${USER} \
    --ssh-private-key ~/.ssh/id_rsa \
    --os ubuntu \
    --time-limit 300 \
    --concurrency 30 \
    --recovery-time 25 \
    --test-count 1 \
    -n 10.142.0.207 -n 10.142.0.203 -n 10.142.1.95 -n 10.142.0.120 -n 10.142.0.45 \
    --test register --nemesis split > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!