Closed cockroach-teamcity closed 4 years ago
Similar to #51052 Seems like an issue with an import.
I didn't think Jepsen tests used IMPORT?
This was mis-triaged. The Jepsen command runs an apt-get
command on the VMs to install tooling, and the apt-get
fails:
Caused by: java.lang.RuntimeException: sudo -S -u root bash -x -c "cd /; env DEBIAN_FRONTEND=noninteractive apt-get install -y --force-yes ntpdate faketime unzip" returned non-zero exit status -1 on 10.128.1.83. STDOUT:
The exact cause of the failure is unclear; it's possible that the repository is presenting an invalid package.
(roachtest).jepsen/bank-multitable/majority-ring failed on master@e9a4f83e3eee59510f97db2c6e0df9b57cf6b944:
| main.runJepsen.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/jepsen.go:159
| main.runJepsen.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/jepsen.go:180
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1373
Wraps: (2) 2 safe details enclosed
Wraps: (3) output in run_080056.491_n6_bash
Wraps: (4) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2107908-1595398673-44-n6cpu4:6 -- bash -e -c "\
| cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
| ~/lein run test \
| --tarball file://${PWD}/cockroach.tgz \
| --username ${USER} \
| --ssh-private-key ~/.ssh/id_rsa \
| --os ubuntu \
| --time-limit 300 \
| --concurrency 30 \
| --recovery-time 25 \
| --test-count 1 \
| -n 10.128.0.90 -n 10.128.0.88 -n 10.128.0.114 -n 10.128.0.95 -n 10.128.0.68 \
| --test bank-multitable --nemesis majority-ring \
| > invoke.log 2>&1 \
| " returned
| stderr:
| Error: SSH_PROBLEM: exit status 255
| (1) SSH_PROBLEM
| Wraps: (2) Node 6. Command with error:
| | ```
| | bash -e -c "\
| | cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
| | ~/lein run test \
| | --tarball file://${PWD}/cockroach.tgz \
| | --username ${USER} \
| | --ssh-private-key ~/.ssh/id_rsa \
| | --os ubuntu \
| | --time-limit 300 \
| | --concurrency 30 \
| | --recovery-time 25 \
| | --test-count 1 \
| | -n 10.128.0.90 -n 10.128.0.88 -n 10.128.0.114 -n 10.128.0.95 -n 10.128.0.68 \
| | --test bank-multitable --nemesis majority-ring \
| | > invoke.log 2>&1 \
| | "
| | ```
| Wraps: (3) exit status 255
| Error types: (1) errors.SSH (2) *hintdetail.withDetail (3) *exec.ExitError
|
| stdout:
Wraps: (5) exit status 10
Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *exec.ExitError
Artifacts: [/jepsen/bank-multitable/majority-ring](https://teamcity.cockroachdb.com/viewLog.html?buildId=2107908&tab=artifacts#/jepsen/bank-multitable/majority-ring) Related: - #46297 roachtest: jepsen/bank-multitable/majority-ring-start-kill-2 failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-roachtest](https://api.github.com/repos/cockroachdb/cockroach/labels/O-roachtest) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-19.2](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-19.2) [release-blocker](https://api.github.com/repos/cockroachdb/cockroach/labels/release-blocker) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ajepsen%2Fbank-multitable%2Fmajority-ring.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)
Last failure is different and caused by #51739
Fixed by cockroachdb/jepsen#24.
(roachtest).jepsen/bank-multitable/majority-ring failed on master@9304ecd70e9f3ba4cb16b5443a10b4e17d7baee0:
More
Artifacts: [/jepsen/bank-multitable/majority-ring](https://teamcity.cockroachdb.com/viewLog.html?buildId=2068260&tab=artifacts#/jepsen/bank-multitable/majority-ring) Related: - #46297 roachtest: jepsen/bank-multitable/majority-ring-start-kill-2 failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-roachtest](https://api.github.com/repos/cockroachdb/cockroach/labels/O-roachtest) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-19.2](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-19.2) [release-blocker](https://api.github.com/repos/cockroachdb/cockroach/labels/release-blocker) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ajepsen%2Fbank-multitable%2Fmajority-ring.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)