jepsen: apt failures

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/d07351b2d1a3e9b5519aa8bc662db0ceb7b7ef48

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stressrace TESTS=jepsen-batch3/register/majority-ring PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=988969&tab=buildLog

The test failed on 31663:
    test.go:639,cluster.go:1110,jepsen.go:87,jepsen.go:127,jepsen.go:313: /home/agent/work/.go/bin/roachprod run teamcity-988969-jepsen-batch3:1-6 -- sh -c "sudo apt-get -y update > logs/apt-upgrade.log 2>&1" returned:
        stderr:

        stdout:
        teamcity-988969-jepsen-batch3: sh -c "sudo apt-get -y upda...........
           1: 
           2: 
           3: 
        exit status 100
           4: 
           5: 
           6: 
        Error:  exit status 100
        : exit status 1
    test.go:639,cluster.go:1110,jepsen.go:74,asm_amd64.s:573,panic.go:377,test.go:640,cluster.go:1110,jepsen.go:87,jepsen.go:127,jepsen.go:313: test already failed

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/5ef4d2c8621fc5465f73a96221b0bd0bc5cd27aa

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stressrace TESTS=jepsen-batch3/register/majority-ring PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=990073&tab=buildLog

The test failed on master:
    test.go:639,jepsen.go:243,jepsen.go:304: /home/agent/work/.go/bin/roachprod run teamcity-990073-jepsen-batch3:6 -- bash -e -c "\
        cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
         ~/lein run test \
           --tarball file://${PWD}/cockroach.tgz \
           --username ${USER} \
           --ssh-private-key ~/.ssh/id_rsa \
           --os ubuntu \
           --time-limit 300 \
           --concurrency 30 \
           --recovery-time 25 \
           --test-count 1 \
           -n 10.128.0.19 -n 10.128.0.36 -n 10.128.0.18 -n 10.128.0.24 -n 10.128.0.21 \
           --test register --nemesis majority-ring \
        > invoke.log 2>&1 \
        " returned:
        stderr:

        stdout:
        Error:  exit status 255
        : exit status 1

[bdarnell]

Two different apt failures. First one is a failure in apt-get update.

Get:19 http://us-central1.gce.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [869 kB]
Ign:19 http://us-central1.gce.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
Get:21 http://us-central1.gce.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [698 kB]
Get:19 http://us-central1.gce.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [1,123 kB]
Ign:19 http://us-central1.gce.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
Err:19 http://us-central1.gce.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
  Writing more data than expected (1123337 > 1123271) [IP: 35.184.213.5 80]
Get:31 http://security.ubuntu.com/ubuntu xenial-security InRelease [107 kB]
Get:32 http://security.ubuntu.com/ubuntu xenial-security/main Sources [136 kB]
Get:33 http://security.ubuntu.com/ubuntu xenial-security/restricted Sources [2,116 B]
Get:34 http://security.ubuntu.com/ubuntu xenial-security/universe Sources [78.8 kB]
Get:35 http://security.ubuntu.com/ubuntu xenial-security/multiverse Sources [2,088 B]
Get:36 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [573 kB]
Get:37 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [393 kB]
Get:38 http://security.ubuntu.com/ubuntu xenial-security/universe Translation-en [151 kB]
Get:39 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages [3,460 B]
Get:40 http://security.ubuntu.com/ubuntu xenial-security/multiverse Translation-en [1,744 B]
Fetched 24.5 MB in 3s (6,669 kB/s)
Reading package lists...
E: Failed to fetch http://us-central1.gce.archive.ubuntu.com/ubuntu/dists/xenial-updates/main/binary-amd64/Packages  Writing more data than expected (1123337 > 1123271) [IP: 35.184.213.5 80]
E: Some index files failed to download. They have been ignored, or old ones used instead.

Second one is #31780: apt-get install failed, but the logs don't show any evidence of failure.

[tbg]

These are now "caught" by grepping them out in the jepsen nightly script. I'll leave this issue open but move it out of the test-failure label.

[andreimatei]

These are now "caught" by grepping them out in the jepsen nightly script.

That wasn't true, but I'm making it true in #37430

[github-actions[bot]]

We have marked this issue as stale because it has been inactive for 18 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 5 days to keep the issue queue tidy. Thank you for your contribution to CockroachDB!