search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.07k stars 3.8k forks source link

roachtest: jepsen/bank-multitable/parts-start-kill-2 failed #78807

Closed cockroach-teamcity closed 2 years ago

cockroach-teamcity commented 2 years ago

roachtest.jepsen/bank-multitable/parts-start-kill-2 failed with artifacts on master @ 29716850b181718594663889ddb5f479fef7a305:

        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1987
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runJepsen.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/jepsen.go:172
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runJepsen.func3
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/jepsen.go:210
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (2) output in run_054706.920030907_n6_bash
        Wraps: (3) bash -e -c "\
          | cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
          |  ~/lein run test \
          |    --tarball file://${PWD}/cockroach.tgz \
          |    --username ${USER} \
          |    --ssh-private-key ~/.ssh/id_rsa \
          |    --os ubuntu \
          |    --time-limit 300 \
          |    --concurrency 30 \
          |    --recovery-time 25 \
          |    --test-count 1 \
          |    -n 10.142.15.226 -n 10.142.15.229 -n 10.142.15.230 -n 10.142.15.228 -n 10.142.0.120 \
          |    --test bank-multitable --nemesis parts --nemesis2 start-kill-2 \
          | > invoke.log 2>&1 \
          | " returned
          | stderr:
          |
          | stdout:
        Wraps: (4) SSH_PROBLEM
        Wraps: (5) Node 6. Command with error:
          | ``````
          | bash -e -c "\
          | cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
          |  ~/lein run test \
          |    --tarball file://${PWD}/cockroach.tgz \
          |    --username ${USER} \
          |    --ssh-private-key ~/.ssh/id_rsa \
          |    --os ubuntu \
          |    --time-limit 300 \
          |    --concurrency 30 \
          |    --recovery-time 25 \
          |    --test-count 1 \
          |    -n 10.142.15.226 -n 10.142.15.229 -n 10.142.15.230 -n 10.142.15.228 -n 10.142.0.120 \
          |    --test bank-multitable --nemesis parts --nemesis2 start-kill-2 \
          | > invoke.log 2>&1 \
          | "
          | ``````
        Wraps: (6) exit status 255
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-14325

cockroach-teamcity commented 2 years ago

roachtest.jepsen/bank-multitable/start-kill-2 failed with artifacts on master @ 29716850b181718594663889ddb5f479fef7a305:

        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1987
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runJepsen.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/jepsen.go:172
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runJepsen.func3
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/jepsen.go:210
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (2) output in run_054811.635657169_n6_bash
        Wraps: (3) bash -e -c "\
          | cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
          |  ~/lein run test \
          |    --tarball file://${PWD}/cockroach.tgz \
          |    --username ${USER} \
          |    --ssh-private-key ~/.ssh/id_rsa \
          |    --os ubuntu \
          |    --time-limit 300 \
          |    --concurrency 30 \
          |    --recovery-time 25 \
          |    --test-count 1 \
          |    -n 10.142.0.166 -n 10.142.0.167 -n 10.142.0.165 -n 10.142.0.161 -n 10.142.0.168 \
          |    --test bank-multitable --nemesis start-kill-2 \
          | > invoke.log 2>&1 \
          | " returned
          | stderr:
          |
          | stdout:
        Wraps: (4) SSH_PROBLEM
        Wraps: (5) Node 6. Command with error:
          | ``````
          | bash -e -c "\
          | cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
          |  ~/lein run test \
          |    --tarball file://${PWD}/cockroach.tgz \
          |    --username ${USER} \
          |    --ssh-private-key ~/.ssh/id_rsa \
          |    --os ubuntu \
          |    --time-limit 300 \
          |    --concurrency 30 \
          |    --recovery-time 25 \
          |    --test-count 1 \
          |    -n 10.142.0.166 -n 10.142.0.167 -n 10.142.0.165 -n 10.142.0.161 -n 10.142.0.168 \
          |    --test bank-multitable --nemesis start-kill-2 \
          | > invoke.log 2>&1 \
          | "
          | ``````
        Wraps: (6) exit status 255
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.jepsen/bank/parts-start-kill-2 failed with artifacts on master @ 29716850b181718594663889ddb5f479fef7a305:

        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1987
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runJepsen.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/jepsen.go:172
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runJepsen.func3
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/jepsen.go:210
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (2) output in run_054824.076259077_n6_bash
        Wraps: (3) bash -e -c "\
          | cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
          |  ~/lein run test \
          |    --tarball file://${PWD}/cockroach.tgz \
          |    --username ${USER} \
          |    --ssh-private-key ~/.ssh/id_rsa \
          |    --os ubuntu \
          |    --time-limit 300 \
          |    --concurrency 30 \
          |    --recovery-time 25 \
          |    --test-count 1 \
          |    -n 10.142.0.85 -n 10.142.0.111 -n 10.142.0.20 -n 10.142.0.137 -n 10.142.0.26 \
          |    --test bank --nemesis parts --nemesis2 start-kill-2 \
          | > invoke.log 2>&1 \
          | " returned
          | stderr:
          |
          | stdout:
        Wraps: (4) SSH_PROBLEM
        Wraps: (5) Node 6. Command with error:
          | ``````
          | bash -e -c "\
          | cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
          |  ~/lein run test \
          |    --tarball file://${PWD}/cockroach.tgz \
          |    --username ${USER} \
          |    --ssh-private-key ~/.ssh/id_rsa \
          |    --os ubuntu \
          |    --time-limit 300 \
          |    --concurrency 30 \
          |    --recovery-time 25 \
          |    --test-count 1 \
          |    -n 10.142.0.85 -n 10.142.0.111 -n 10.142.0.20 -n 10.142.0.137 -n 10.142.0.26 \
          |    --test bank --nemesis parts --nemesis2 start-kill-2 \
          | > invoke.log 2>&1 \
          | "
          | ``````
        Wraps: (6) exit status 255
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.jepsen/bank/start-kill-2 failed with artifacts on master @ 29716850b181718594663889ddb5f479fef7a305:

        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1987
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runJepsen.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/jepsen.go:172
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runJepsen.func3
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/jepsen.go:210
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (2) output in run_054842.240753907_n6_bash
        Wraps: (3) bash -e -c "\
          | cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
          |  ~/lein run test \
          |    --tarball file://${PWD}/cockroach.tgz \
          |    --username ${USER} \
          |    --ssh-private-key ~/.ssh/id_rsa \
          |    --os ubuntu \
          |    --time-limit 300 \
          |    --concurrency 30 \
          |    --recovery-time 25 \
          |    --test-count 1 \
          |    -n 10.142.15.242 -n 10.142.15.246 -n 10.142.15.244 -n 10.142.15.243 -n 10.142.15.241 \
          |    --test bank --nemesis start-kill-2 \
          | > invoke.log 2>&1 \
          | " returned
          | stderr:
          |
          | stdout:
        Wraps: (4) SSH_PROBLEM
        Wraps: (5) Node 6. Command with error:
          | ``````
          | bash -e -c "\
          | cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && \
          |  ~/lein run test \
          |    --tarball file://${PWD}/cockroach.tgz \
          |    --username ${USER} \
          |    --ssh-private-key ~/.ssh/id_rsa \
          |    --os ubuntu \
          |    --time-limit 300 \
          |    --concurrency 30 \
          |    --recovery-time 25 \
          |    --test-count 1 \
          |    -n 10.142.15.242 -n 10.142.15.246 -n 10.142.15.244 -n 10.142.15.243 -n 10.142.15.241 \
          |    --test bank --nemesis start-kill-2 \
          | > invoke.log 2>&1 \
          | "
          | ``````
        Wraps: (6) exit status 255
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!