cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.07k stars 3.8k forks source link

roachtest: jepsen/sequential/start-kill-2 failed #43068

Closed cockroach-teamcity closed 4 years ago

cockroach-teamcity commented 4 years ago

(roachtest).jepsen/sequential/start-kill-2 failed on master@734e357412dadafcd6084b8eab8e251e44e86b4a:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191210-1634867/jepsen/sequential/start-kill-2/run_1
    jepsen.go:266,jepsen.go:324,test_runner.go:700: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1634867-1575963949-41-n6cpu4:6 -- bash -e -c "\
        cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
         ~/lein run test \
           --tarball file://${PWD}/cockroach.tgz \
           --username ${USER} \
           --ssh-private-key ~/.ssh/id_rsa \
           --os ubuntu \
           --time-limit 300 \
           --concurrency 30 \
           --recovery-time 25 \
           --test-count 1 \
           -n 10.128.0.35 -n 10.128.0.88 -n 10.128.0.106 -n 10.128.0.16 -n 10.128.0.96 \
           --test sequential --nemesis start-kill-2 \
        > invoke.log 2>&1 \
        " returned:
        stderr:

        stdout:
        Error:  exit status 255
        : exit status 1
details

Artifacts: [/jepsen/sequential/start-kill-2](https://teamcity.cockroachdb.com/viewLog.html?buildId=1634867&tab=artifacts#/jepsen/sequential/start-kill-2) ``` make stressrace TESTS=jepsen/sequential/start-kill-2 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS=-timeout 5m' 2>&1 ``` powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 4 years ago

(roachtest).jepsen/sequential/start-kill-2 failed on master@beb69e089b6eadc0fde6c92eb533b08d248938fe:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191216-1644933/jepsen/sequential/start-kill-2/run_1
    jepsen.go:266,jepsen.go:324,test_runner.go:700: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1644933-1576481346-44-n6cpu4:6 -- bash -e -c "\
        cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
         ~/lein run test \
           --tarball file://${PWD}/cockroach.tgz \
           --username ${USER} \
           --ssh-private-key ~/.ssh/id_rsa \
           --os ubuntu \
           --time-limit 300 \
           --concurrency 30 \
           --recovery-time 25 \
           --test-count 1 \
           -n 10.128.0.93 -n 10.128.0.97 -n 10.128.0.92 -n 10.128.0.103 -n 10.128.0.5 \
           --test sequential --nemesis start-kill-2 \
        > invoke.log 2>&1 \
        " returned:
        stderr:

        stdout:
        Error:  exit status 255
        : exit status 1
Repro

Artifacts: [/jepsen/sequential/start-kill-2](https://teamcity.cockroachdb.com/viewLog.html?buildId=1644933&tab=artifacts#/jepsen/sequential/start-kill-2) ``` make stressrace TESTS=jepsen/sequential/start-kill-2 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS=-timeout 5m' 2>&1 ``` powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 4 years ago

(roachtest).jepsen/sequential/start-kill-2 failed on master@2739821b911d777fa2a927295d699b559360a802:

            main.runJepsen.func2
                /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/jepsen.go:159
            main.runJepsen.func3
                /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/jepsen.go:180
            runtime.goexit
                /usr/local/go/src/runtime/asm_amd64.s:1357
          - error with embedded safe details: output in %s
            -- arg 1: <string>
          - output in run_101534.892_n6_bash:
          - error with attached stack trace:
            main.execCmd
                /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
            main.(*cluster).RunL
                /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
            main.(*cluster).RunE
                /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
            main.runJepsen.func2
                /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/jepsen.go:159
            main.runJepsen.func3
                /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/jepsen.go:180
            runtime.goexit
                /usr/local/go/src/runtime/asm_amd64.s:1357
          - error with embedded safe details: %s returned:
            stderr:
            %s
            stdout:
            %s
            -- arg 1: <string>
            -- arg 2: <string>
            -- arg 3: <string>
          - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1747374-1581581035-55-n6cpu4:6 -- bash -e -c "\
            cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
             ~/lein run test \
               --tarball file://${PWD}/cockroach.tgz \
               --username ${USER} \
               --ssh-private-key ~/.ssh/id_rsa \
               --os ubuntu \
               --time-limit 300 \
               --concurrency 30 \
               --recovery-time 25 \
               --test-count 1 \
               -n 10.128.0.118 -n 10.128.0.237 -n 10.128.0.199 -n 10.128.0.239 -n 10.128.0.238 \
               --test sequential --nemesis start-kill-2 \
            > invoke.log 2>&1 \
            " returned:
            stderr:
            Error:  exit status 255

            stdout::
          - exit status 1
More

Artifacts: [/jepsen/sequential/start-kill-2](https://teamcity.cockroachdb.com/viewLog.html?buildId=1747374&tab=artifacts#/jepsen/sequential/start-kill-2) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ajepsen%2Fsequential%2Fstart-kill-2.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)