cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.07k stars 3.8k forks source link

roachtest: jepsen-batch1/bank/majority-ring failed #37823

Closed cockroach-teamcity closed 5 years ago

cockroach-teamcity commented 5 years ago

SHA: https://github.com/cockroachdb/cockroach/commits/db9c1217a6967fcac2d135cf0f24a4265dc76d77

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=jepsen-batch1/bank/majority-ring PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1310296&tab=buildLog

The test failed on branch=master, cloud=gce:
    cluster.go:1516,jepsen.go:88,jepsen.go:136,jepsen.go:324,test.go:1251: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1310296-jepsen-batch1:1-6 -- sh -c "sudo apt-get -y upgrade -o Dpkg::Options::='--force-confold' > logs/apt-upgrade.log 2>&1" returned:
        stderr:

        stdout:
        teamcity-1310296-jepsen-batch1: sh -c "sudo apt-get -y upgr........................................................................
           1: 
           2: 
        ssh verbose log retained in /root/.roachprod/debug/ssh_34.74.98.185_2019-05-26T10:30:53Z: exit status 100
           3: 
           4: 
           5: 
           6: 
        Error:  ssh verbose log retained in /root/.roachprod/debug/ssh_34.74.98.185_2019-05-26T10:30:53Z: exit status 100
        : exit status 1
    cluster.go:1516,jepsen.go:74,asm_amd64.s:522,panic.go:397,test.go:788,test.go:774,cluster.go:1516,jepsen.go:88,jepsen.go:136,jepsen.go:324,test.go:1251: test already failed
tbg commented 5 years ago

31944