search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.96k stars 3.79k forks source link

roachtest: cdc/sink-chaos failed #122676

Closed cockroach-teamcity closed 2 months ago

cockroach-teamcity commented 5 months ago

roachtest.cdc/sink-chaos failed with artifacts on release-24.1 @ 7185e6c311160061e0753e9ef1d66e8bb828fc98:

(cdc.go:560).newChangefeed: failed to create changefeed: pq: kafka server: Replication-factor is invalid
(cluster.go:2348).Run: context canceled
(cdc.go:385).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/sink-chaos/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/cdc

This test on roachdash | Improve this report!

Jira issue: CRDB-38021

wenyihu6 commented 5 months ago

Not a release blocker - this looks like the same infra flake as https://github.com/cockroachdb/cockroach/issues/121346. We might want to deflake it by waiting for the new brokers to be properly started before creating topics. 

[2024-04-19 11:00:15,235] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)
[2024-04-19 11:00:15,256] INFO [Admin Manager on Broker 0]: Error processing create topic request CreatableTopic(name='item', numPartitions=1, replicationFactor=1, assignments=[], configs=[]) (kafka.server.ZkAdminManager)
org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 1 larger than available brokers: 0.
[2024-04-19 11:00:15,257] INFO [Admin Manager on Broker 0]: Error processing create topic request CreatableTopic(name='district', numPartitions=1, replicationFactor=1, assignments=[], configs=[]) (kafka.server.ZkAdminManager)
org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 1 larger than available brokers: 0.
[2024-04-19 11:00:15,258] INFO [Admin Manager on Broker 0]: Error processing create topic request CreatableTopic(name='order_line', numPartitions=1, replicationFactor=1, assignments=[], configs=[]) (kafka.server.ZkAdminManager)
org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 1 larger than available brokers: 0.
[2024-04-19 11:00:15,258] INFO [Admin Manager on Broker 0]: Error processing create topic request CreatableTopic(name='history', numPartitions=1, replicationFactor=1, assignments=[], configs=[]) (kafka.server.ZkAdminManager)
org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 1 larger than available brokers: 0.
[2024-04-19 11:00:15,258] INFO [Admin Manager on Broker 0]: Error processing create topic request CreatableTopic(name='warehouse', numPartitions=1, replicationFactor=1, assignments=[], configs=[]) (kafka.server.ZkAdminManager)
org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 1 larger than available brokers: 0.
[2024-04-19 11:00:15,258] INFO [Admin Manager on Broker 0]: Error processing create topic request CreatableTopic(name='stock', numPartitions=1, replicationFactor=1, assignments=[], configs=[]) (kafka.server.ZkAdminManager)
org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 1 larger than available brokers: 0.
[2024-04-19 11:00:15,258] INFO [Admin Manager on Broker 0]: Error processing create topic request CreatableTopic(name='new_order', numPartitions=1, replicationFactor=1, assignments=[], configs=[]) (kafka.server.ZkAdminManager)
org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 1 larger than available brokers: 0.
[2024-04-19 11:00:15,258] INFO [Admin Manager on Broker 0]: Error processing create topic request CreatableTopic(name='order', numPartitions=1, replicationFactor=1, assignments=[], configs=[]) (kafka.server.ZkAdminManager)
org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 1 larger than available brokers: 0.
[2024-04-19 11:00:15,259] INFO [Admin Manager on Broker 0]: Error processing create topic request CreatableTopic(name='customer', numPartitions=1, replicationFactor=1, assignments=[], configs=[]) (kafka.server.ZkAdminManager)
org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 1 larger than available brokers: 0.
[2024-04-19 11:00:15,299] INFO [BrokerToControllerChannelManager broker=0 name=forwarding]: Recorded new controller, from now on will use node teamcity-14898299-1713505946-26-n4cpu16-0004.c.cockroach-ephemeral.internal:9092 (id: 0 rack: null) (kafka.server.BrokerToControllerRequestThread)
[2024-04-19 11:00:15,333] INFO [BrokerToControllerChannelManager broker=0 name=alterPartition]: Recorded new controller, from now on will use node teamcity-14898299-1713505946-26-n4cpu16-0004.c.cockroach-ephemeral.internal:9092 (id: 0 rack: null) (kafka.server.BrokerToControllerRequestThread)
andyyang890 commented 2 months ago

Closing since this was an infra flake and we haven't seen it reoccur in a few months