search

brimdata / zync

Kafka connector to sync Zed lakes to and from Kafka topics
BSD 3-Clause "New" or "Revised" License
17 stars 3 forks source link

reduce "zync from-kafka" concurrency and memory usage #108

Closed nwt closed 1 year ago

nwt commented 1 year ago

"zync from-kafka" creates a goroutine and Kafka client per Kafka topic, leading to substantial CPU and memory usage when the number of topics is large. Reduce both by creating a single goroutine and Kafka client for all topics.