cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.07k stars 3.8k forks source link

roachtest: jepsen-batch2/g2/majority-ring failed #31780

Closed cockroach-teamcity closed 6 years ago

cockroach-teamcity commented 6 years ago

SHA: https://github.com/cockroachdb/cockroach/commits/5a373445c0674f060a4bfe369ad290a0cacccb6c

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stressrace TESTS=jepsen-batch2/g2/majority-ring PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=982644&tab=buildLog

The test failed on master:
    test.go:639,jepsen.go:234,jepsen.go:294: /home/agent/work/.go/bin/roachprod run teamcity-982644-jepsen-batch2:6 -- bash -e -c "\
        cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
         ~/lein run test \
           --tarball file://${PWD}/cockroach.tgz \
           --username ${USER} \
           --ssh-private-key ~/.ssh/id_rsa \
           --os ubuntu \
           --time-limit 300 \
           --concurrency 30 \
           --recovery-time 25 \
           --test-count 1 \
           -n 10.128.0.10 -n 10.128.0.47 -n 10.128.0.51 -n 10.128.0.54 -n 10.128.0.48 \
           --test g2 --nemesis majority-ring \
        > invoke.log 2>&1 \
        " returned:
        stderr:

        stdout:
        Error:  exit status 255
        : exit status 1
bdarnell commented 6 years ago

It's apparently an apt failure, but the logs don't show anything unusual:

Caused by: java.lang.RuntimeException: sudo -S -u root bash -x -c "cd /; env DEBIAN_FRONTEND=noninteractive apt-get install -y --force-yes ntpdate faketime unzip" returned non-zero exit status -1 on 10.128.0.47. STDOUT:
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  libfaketime
Suggested packages:
  zip
The following NEW packages will be installed:
  faketime libfaketime ntpdate unzip
0 upgraded, 4 newly installed, 0 to remove and 6 not upgraded.
Need to get 239 kB of archives.
After this operation, 878 kB of additional disk space will be used.
Get:1 http://us-central1.gce.archive.ubuntu.com/ubuntu xenial-updates/main amd64 ntpdate amd64 1:4.2.8p4+dfsg-3ubuntu5.9 [48.6 kB]
Get:2 http://us-central1.gce.archive.ubuntu.com/ubuntu xenial/universe amd64 libfaketime amd64 0.9.6-4 [22.0 kB]
Get:3 http://us-central1.gce.archive.ubuntu.com/ubuntu xenial/universe amd64 faketime amd64 0.9.6-4 [9,596 B]
Get:4 http://us-central1.gce.archive.ubuntu.com/ubuntu xenial/main amd64 unzip amd64 6.0-20ubuntu1 [158 kB]
Fetched 239 kB in 0s (10.7 MB/s)
Selecting previously unselected package ntpdate.
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 71009 files and directories currently installed.)
Preparing to unpack .../ntpdate_1%3a4.2.8p4+dfsg-3ubuntu5.9_amd64.deb ...
Unpacking ntpdate (1:4.2.8p4+dfsg-3ubuntu5.9) ...
Selecting previously unselected package libfaketime:amd64.
Preparing to unpack .../libfaketime_0.9.6-4_amd64.deb ...
Unpacking libfaketime:amd64 (0.9.6-4) ...
Selecting previously unselected package faketime.
Preparing to unpack .../faketime_0.9.6-4_amd64.deb ...
Unpacking faketime (0.9.6-4) ...
Selecting previously unselected package unzip.
Preparing to unpack .../unzip_6.0-20ubuntu1_amd64.deb ...
Unpacking unzip (6.0-20ubuntu1) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for mime-support (3.59ubuntu1) ...
Setting up ntpdate (1:4.2.8p4+dfsg-3ubuntu5.9) ...
Setting up libfaketime:amd64 (0.9.6-4) ...
Setting up faketime (0.9.6-4) ...
Setting up unzip (6.0-20ubuntu1) ...

STDERR:
+ cd /
+ env DEBIAN_FRONTEND=noninteractive apt-get install -y --force-yes ntpdate faketime unzip

Just a fluke? I don't see anything useful we can do here.