search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.2k stars 3.82k forks source link

roachtest: cdc/bank failed #52227

Closed cockroach-teamcity closed 4 years ago

cockroach-teamcity commented 4 years ago

(roachtest).cdc/bank failed on master@815725c2df3669fd25d589272167a5e092ba3b2d:

          | main.(*cluster).Run
          |     /go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2124
          | main.kafkaManager.stop
          |     /go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cdc.go:703
          | runtime.gopanic
          |     /usr/local/go/src/runtime/panic.go:969
          | main.(*test).fatalfInner
          |     /go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test.go:277
          | main.(*test).Fatal
          |     /go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test.go:252
          | main.(*monitor).Wait
          |     /go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2556
          | main.runCDCBank
          |     /go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cdc.go:371
          | main.registerCDC.func8
          |     /go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cdc.go:651
          | main.(*testRunner).runTest.func2
          |     /go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:754
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1373
        Wraps: (5) cluster.RunL
        Wraps: (6) context canceled
        Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *withstack.withStack (5) *errutil.withMessage (6) *errors.errorString

    cluster.go:1571,context.go:135,cluster.go:1560,test_runner.go:823: dead node detection: /go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor local --oneshot --ignore-empty-nodes: exit status 1 4: skipped
        2: dead
        3: 1105
        1: 971
        Error: UNCLASSIFIED_PROBLEM: 2: dead
        (1) UNCLASSIFIED_PROBLEM
        Wraps: (2) attached stack trace
          | main.glob..func13
          |     /go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1115
          | main.wrap.func1
          |     /go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:266
          | github.com/spf13/cobra.(*Command).execute
          |     /go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
          | github.com/spf13/cobra.(*Command).ExecuteC
          |     /go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
          | github.com/spf13/cobra.(*Command).Execute
          |     /go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
          | main.main
          |     /go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1808
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:203
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1373
        Wraps: (3) 3 safe details enclosed
        Wraps: (4) 2: dead
        Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString
More

Artifacts: [/cdc/bank](https://teamcity.cockroachdb.com/viewLog.html?buildId=2139245&tab=artifacts#/cdc/bank) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Acdc%2Fbank.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

andreimatei commented 4 years ago

The monitor complained that a node is dead, but I don't see any evidence of that. The node in question logged a few seconds after the monitor detected it to be dead... Don't know what to make of it, but closing.