search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.07k stars 3.8k forks source link

roachtest: jepsen-batch3/multi-register/parts-start-kill-2 failed [hopefully #36431] #37932

Closed cockroach-teamcity closed 5 years ago

cockroach-teamcity commented 5 years ago

SHA: https://github.com/cockroachdb/cockroach/commits/f49f211f8fb2c2aa51182054192ebfcb9c0355f0

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=jepsen-batch3/multi-register/parts-start-kill-2 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1315180&tab=buildLog

The test failed on branch=release-19.1, cloud=gce:
    jepsen.go:254,jepsen.go:324,test.go:1251: exit status 1
tbg commented 5 years ago 
  :failures [2214]},
 :valid? false}

Analysis invalid!
cockroach-teamcity commented 5 years ago

SHA: https://github.com/cockroachdb/cockroach/commits/3f2470f358e638f526a07495bfd11f1805c135a8

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=jepsen-batch3/multi-register/parts-start-kill-2 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1325775&tab=buildLog

The test failed on branch=master, cloud=gce:
    jepsen.go:253,jepsen.go:323,test.go:1248: exit status 1
cockroach-teamcity commented 5 years ago

SHA: https://github.com/cockroachdb/cockroach/commits/de90b5d84c9e5d2ef562ebfa3d4a1bdc0e834e10

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=jepsen-batch3/multi-register/parts-start-kill-2 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1332319&tab=buildLog

The test failed on branch=master, cloud=gce:
    jepsen.go:253,jepsen.go:323,test.go:1248: exit status 1
cockroach-teamcity commented 5 years ago

SHA: https://github.com/cockroachdb/cockroach/commits/4878cb3e960dc26f0946e527e1836f27b3304c97

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=jepsen/multi-register/parts-start-kill-2 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1373334&tab=buildLog

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190704-1373334/jepsen/multi-register/parts-start-kill-2/run_1
    cluster.go:1724,jepsen.go:98,jepsen.go:138,jepsen.go:316,test_runner.go:681: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1562220659-29-n6cpu4:1-6 -- tar --transform s,^,cockroach/, -c -z -f cockroach.tgz cockroach returned:
        stderr:

        stdout:
        teamcity-1562220659-29-n6cpu4: tar --transform s,^,cockroa.........
           1: 
           2: tar: cockroach: Cannot stat: No such file or directory
        tar: Exiting with failure status due to previous errors
        ssh verbose log retained in /root/.roachprod/debug/ssh_35.227.82.203_2019-07-04T14:01:28Z: exit status 2
           3: 
           4: 
           5: tar: cockroach: Cannot stat: No such file or directory
        tar: Exiting with failure status due to previous errors
        ssh verbose log retained in /root/.roachprod/debug/ssh_34.73.100.118_2019-07-04T14:01:28Z: exit status 2
           6: tar: cockroach: Cannot stat: No such file or directory
        tar: Exiting with failure status due to previous errors
        ssh verbose log retained in /root/.roachprod/debug/ssh_35.196.208.190_2019-07-04T14:01:28Z: exit status 2
        Error:  ssh verbose log retained in /root/.roachprod/debug/ssh_35.227.82.203_2019-07-04T14:01:28Z: exit status 2
        : exit status 1
cockroach-teamcity commented 5 years ago

SHA: https://github.com/cockroachdb/cockroach/commits/4878cb3e960dc26f0946e527e1836f27b3304c97

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=jepsen/register/parts-start-kill-2 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1373334&tab=buildLog

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190704-1373334/jepsen/register/parts-start-kill-2/run_1
    cluster.go:1724,jepsen.go:98,jepsen.go:138,jepsen.go:316,test_runner.go:681: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1562220659-29-n6cpu4:1-6 -- tar --transform s,^,cockroach/, -c -z -f cockroach.tgz cockroach returned:
        stderr:

        stdout:
        teamcity-1562220659-29-n6cpu4: tar --transform s,^,cockroa.........
           1: 
           2: tar: cockroach: Cannot stat: No such file or directory
        tar: Exiting with failure status due to previous errors
        ssh verbose log retained in /root/.roachprod/debug/ssh_35.227.82.203_2019-07-04T14:05:09Z: exit status 2
           3: 
           4: 
           5: tar: cockroach: Cannot stat: No such file or directory
        tar: Exiting with failure status due to previous errors
        ssh verbose log retained in /root/.roachprod/debug/ssh_34.73.100.118_2019-07-04T14:05:09Z: exit status 2
           6: tar: cockroach: Cannot stat: No such file or directory
        tar: Exiting with failure status due to previous errors
        ssh verbose log retained in /root/.roachprod/debug/ssh_35.196.208.190_2019-07-04T14:05:09Z: exit status 2
        Error:  ssh verbose log retained in /root/.roachprod/debug/ssh_35.227.82.203_2019-07-04T14:05:09Z: exit status 2
        : exit status 1
cockroach-teamcity commented 5 years ago

SHA: https://github.com/cockroachdb/cockroach/commits/26edea51118a0e16b61748c08068bfa6f76543ca

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=jepsen/register/parts-start-kill-2 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1404886&tab=buildLog

The test failed on branch=provisional_201907241708_v19.2.0-alpha.20190729, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190725-1404886/jepsen/register/parts-start-kill-2/run_1
    jepsen.go:256,jepsen.go:325,test_runner.go:691: exit status 1
cockroach-teamcity commented 5 years ago

SHA: https://github.com/cockroachdb/cockroach/commits/da56c792e968574b8f1d9ef3fdb45d56a530221a

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=jepsen/register/parts-start-kill-2 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1415578&tab=buildLog

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190801-1415578/jepsen/register/parts-start-kill-2/run_1
    jepsen.go:264,jepsen.go:325,test_runner.go:691: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1564640260-59-n6cpu4:6 -- bash -e -c "\
        cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
         ~/lein run test \
           --tarball file://${PWD}/cockroach.tgz \
           --username ${USER} \
           --ssh-private-key ~/.ssh/id_rsa \
           --os ubuntu \
           --time-limit 300 \
           --concurrency 30 \
           --recovery-time 25 \
           --test-count 1 \
           -n 10.128.0.66 -n 10.128.0.64 -n 10.128.0.68 -n 10.128.0.71 -n 10.128.0.67 \
           --test register --nemesis parts --nemesis2 start-kill-2 \
        > invoke.log 2>&1 \
        " returned:
        stderr:

        stdout:
        Error:  exit status 255
        : exit status 1