roachtest: jepsen/bank-multitable/majority-ring failed

[cockroach-teamcity]

(roachtest).jepsen/bank-multitable/majority-ring failed on master@9304ecd70e9f3ba4cb16b5443a10b4e17d7baee0:

          | main.runJepsen.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/jepsen.go:159
          | main.runJepsen.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/jepsen.go:180
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1373
        Wraps: (2) 2 safe details enclosed
        Wraps: (3) output in run_090351.878_n6_bash
        Wraps: (4) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2068260-1594103442-42-n6cpu4:6 -- bash -e -c "\
          | cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
          |  ~/lein run test \
          |    --tarball file://${PWD}/cockroach.tgz \
          |    --username ${USER} \
          |    --ssh-private-key ~/.ssh/id_rsa \
          |    --os ubuntu \
          |    --time-limit 300 \
          |    --concurrency 30 \
          |    --recovery-time 25 \
          |    --test-count 1 \
          |    -n 10.128.1.83 -n 10.128.1.52 -n 10.128.1.84 -n 10.128.1.50 -n 10.128.1.53 \
          |    --test bank-multitable --nemesis majority-ring \
          | > invoke.log 2>&1 \
          | " returned
          | stderr:
          | Error: SSH_PROBLEM: exit status 255
          | (1) SSH_PROBLEM
          | Wraps: (2) Node 6. Command with error:
          |   | ```
          |   | bash -e -c "\
          |   | cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
          |   |  ~/lein run test \
          |   |    --tarball file://${PWD}/cockroach.tgz \
          |   |    --username ${USER} \
          |   |    --ssh-private-key ~/.ssh/id_rsa \
          |   |    --os ubuntu \
          |   |    --time-limit 300 \
          |   |    --concurrency 30 \
          |   |    --recovery-time 25 \
          |   |    --test-count 1 \
          |   |    -n 10.128.1.83 -n 10.128.1.52 -n 10.128.1.84 -n 10.128.1.50 -n 10.128.1.53 \
          |   |    --test bank-multitable --nemesis majority-ring \
          |   | > invoke.log 2>&1 \
          |   | "
          |   | ```
          | Wraps: (3) exit status 255
          | Error types: (1) errors.SSH (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
        Wraps: (5) exit status 10
        Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *exec.ExitError

More

Artifacts: [/jepsen/bank-multitable/majority-ring](https://teamcity.cockroachdb.com/viewLog.html?buildId=2068260&tab=artifacts#/jepsen/bank-multitable/majority-ring) Related: - #46297 roachtest: jepsen/bank-multitable/majority-ring-start-kill-2 failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-roachtest](https://api.github.com/repos/cockroachdb/cockroach/labels/O-roachtest) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-19.2](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-19.2) [release-blocker](https://api.github.com/repos/cockroachdb/cockroach/labels/release-blocker) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ajepsen%2Fbank-multitable%2Fmajority-ring.%2A&sort=title&restgroup=false&display=lastcommented+project) _{powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)}

[darinpp]

Similar to #51052 Seems like an issue with an import.

[dt]

I didn't think Jepsen tests used IMPORT?

[knz]

This was mis-triaged. The Jepsen command runs an apt-get command on the VMs to install tooling, and the apt-get fails:

Caused by: java.lang.RuntimeException: sudo -S -u root bash -x -c "cd /; env DEBIAN_FRONTEND=noninteractive apt-get install -y --force-yes ntpdate faketime unzip" returned non-zero exit status -1 on 10.128.1.83. STDOUT:

The exact cause of the failure is unclear; it's possible that the repository is presenting an invalid package.

[cockroach-teamcity]

(roachtest).jepsen/bank-multitable/majority-ring failed on master@e9a4f83e3eee59510f97db2c6e0df9b57cf6b944:

          | main.runJepsen.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/jepsen.go:159
          | main.runJepsen.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/jepsen.go:180
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1373
        Wraps: (2) 2 safe details enclosed
        Wraps: (3) output in run_080056.491_n6_bash
        Wraps: (4) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2107908-1595398673-44-n6cpu4:6 -- bash -e -c "\
          | cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
          |  ~/lein run test \
          |    --tarball file://${PWD}/cockroach.tgz \
          |    --username ${USER} \
          |    --ssh-private-key ~/.ssh/id_rsa \
          |    --os ubuntu \
          |    --time-limit 300 \
          |    --concurrency 30 \
          |    --recovery-time 25 \
          |    --test-count 1 \
          |    -n 10.128.0.90 -n 10.128.0.88 -n 10.128.0.114 -n 10.128.0.95 -n 10.128.0.68 \
          |    --test bank-multitable --nemesis majority-ring \
          | > invoke.log 2>&1 \
          | " returned
          | stderr:
          | Error: SSH_PROBLEM: exit status 255
          | (1) SSH_PROBLEM
          | Wraps: (2) Node 6. Command with error:
          |   | ```
          |   | bash -e -c "\
          |   | cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
          |   |  ~/lein run test \
          |   |    --tarball file://${PWD}/cockroach.tgz \
          |   |    --username ${USER} \
          |   |    --ssh-private-key ~/.ssh/id_rsa \
          |   |    --os ubuntu \
          |   |    --time-limit 300 \
          |   |    --concurrency 30 \
          |   |    --recovery-time 25 \
          |   |    --test-count 1 \
          |   |    -n 10.128.0.90 -n 10.128.0.88 -n 10.128.0.114 -n 10.128.0.95 -n 10.128.0.68 \
          |   |    --test bank-multitable --nemesis majority-ring \
          |   | > invoke.log 2>&1 \
          |   | "
          |   | ```
          | Wraps: (3) exit status 255
          | Error types: (1) errors.SSH (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
        Wraps: (5) exit status 10
        Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *exec.ExitError

More

Artifacts: [/jepsen/bank-multitable/majority-ring](https://teamcity.cockroachdb.com/viewLog.html?buildId=2107908&tab=artifacts#/jepsen/bank-multitable/majority-ring) Related: - #46297 roachtest: jepsen/bank-multitable/majority-ring-start-kill-2 failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-roachtest](https://api.github.com/repos/cockroachdb/cockroach/labels/O-roachtest) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-19.2](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-19.2) [release-blocker](https://api.github.com/repos/cockroachdb/cockroach/labels/release-blocker) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ajepsen%2Fbank-multitable%2Fmajority-ring.%2A&sort=title&restgroup=false&display=lastcommented+project) _{powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)}

[knz]

Last failure is different and caused by #51739

[irfansharif]

Fixed by cockroachdb/jepsen#24.