brimdata / zync

Kafka connector to sync Zed lakes to and from Kafka topics
BSD 3-Clause "New" or "Revised" License
17 stars 3 forks source link

add -kafka.replicas flag to "zync from-kafka" #110

Closed nwt closed 1 year ago

nwt commented 1 year ago

When the -kafka.replicas flag is set to a positive integer, "zync from-kafka" interprets it as a replication factor and attempts to create each topic from which it will consume with one partition and the given replication factor. Existing topics are ignored, including those with a different number of partitions or replication factor.

Creating missing topics can reduce CPU utilization in zync and the Kafka broker caused by the Kafka client package, which otherwise repeatedly updates topic metadata to poll for creation of those topics.