cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.1k stars 3.81k forks source link

roachtest: cdc/crdb-chaos failed #92802

Closed cockroach-teamcity closed 1 year ago

cockroach-teamcity commented 1 year ago

roachtest.cdc/crdb-chaos failed with artifacts on release-22.1 @ f9730bda77ce8e3ecfa42302be58d81dcd04cd21:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/cdc/crdb-chaos/run_1
    cluster.go:1934,cdc.go:1435,cdc.go:1390,cdc.go:160,cdc.go:759,test_runner.go:883: output in run_070653.064969768_n4_CONFLUENTCURRENTmntdata1confluent: CONFLUENT_CURRENT=/mnt/data1/confluent CONFLUENT_HOME=/mnt/data1/confluent/confluent-6.1.0 KAFKA_OPTS='-Djava.security.auth.login.config=/mnt/data1/confluent/confluent-6.1.0/etc/kafka/server_jaas.conf -Dkafka.logs.dir=logs/kafka' /mnt/data1/confluent/confluent-6.1.0/bin/confluent local services kafka start returned: SSH_PROBLEM: exit status 255
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1968
          | main.(*clusterImpl).Run
          |     main/pkg/cmd/roachtest/cluster.go:1932
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.kafkaManager.restart
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:1435
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.kafkaManager.start
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:1390
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:160
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func5
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:759
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:883
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (2) output in run_070653.064969768_n4_CONFLUENTCURRENTmntdata1confluent
        Wraps: (3) CONFLUENT_CURRENT=/mnt/data1/confluent CONFLUENT_HOME=/mnt/data1/confluent/confluent-6.1.0 KAFKA_OPTS='-Djava.security.auth.login.config=/mnt/data1/confluent/confluent-6.1.0/etc/kafka/server_jaas.conf -Dkafka.logs.dir=logs/kafka' /mnt/data1/confluent/confluent-6.1.0/bin/confluent local services kafka start returned
          | stderr:
          |
          | stdout:
        Wraps: (4) SSH_PROBLEM
        Wraps: (5) Node 4. Command with error:
          | ``````
          | CONFLUENT_CURRENT=/mnt/data1/confluent CONFLUENT_HOME=/mnt/data1/confluent/confluent-6.1.0 KAFKA_OPTS='-Djava.security.auth.login.config=/mnt/data1/confluent/confluent-6.1.0/etc/kafka/server_jaas.conf -Dkafka.logs.dir=logs/kafka' /mnt/data1/confluent/confluent-6.1.0/bin/confluent local services kafka start
          | ``````
        Wraps: (6) exit status 255
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #77815 roachtest: cdc/crdb-chaos failed [C-test-failure O-roachtest O-robot T-cdc branch-master] - #68047 roachtest: cdc/crdb-chaos failed [C-test-failure O-roachtest O-robot T-cdc branch-release-21.1]

/cc @cockroachdb/cdc

This test on roachdash | Improve this report!

Jira issue: CRDB-21973

Epic CRDB-11732

jayshrivastava commented 1 year ago

Closing because it seems like an infra flake - ssh error + no repeated failures.