cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.07k stars 3.8k forks source link

roachtest: jepsen/1/bank-multitable/strobe-skews failed on master #29071

Closed cockroach-teamcity closed 6 years ago

cockroach-teamcity commented 6 years ago

SHA: https://github.com/cockroachdb/cockroach/commits/9ee43b32c83db6418070f92d9d698c1890d80b9e

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stressrace instead of stress and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stress TESTS=jepsen/1/bank-multitable/strobe-skews PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-stderr=false -maxtime 20m -timeout 10m'

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=860807&tab=buildLog

    test.go:494,jepsen.go:232,jepsen.go:285: /home/agent/work/.go/bin/roachprod run teamcity-860807-jepsen-1:6 -- bash -e -c "\
        cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
         ~/lein run test \
           --tarball file://${PWD}/cockroach.tgz \
           --username ${USER} \
           --ssh-private-key ~/.ssh/id_rsa \
           --os ubuntu \
           --time-limit 300 \
           --concurrency 30 \
           --recovery-time 25 \
           --test-count 1 \
           -n 10.128.0.27 -n 10.128.0.26 -n 10.128.0.25 -n  -n 10.128.0.23 \
           --test bank-multitable --nemesis strobe-skews \
        > invoke.log 2>&1 \
        ": exit status 1
tbg commented 6 years ago

ERROR [2018-08-26 09:51:08,435] clojure-agent-send-off-pool-3 - jepsen.control Error opening SSH session. Verify username, password, and node hostnames are correct.