brimdata / zync

Kafka connector to sync Zed lakes to and from Kafka topics
BSD 3-Clause "New" or "Revised" License
17 stars 3 forks source link

improve "zync from-kafka" startup behavior #99

Closed nwt closed 1 year ago

nwt commented 1 year ago

"zync from-kafka" does slow per-pool and per-topic configuration steps sequentially in the main goroutine, so when configured with many pools, it can be slow to start synchronizing some of them, particularly when CPU-bound. Move those configuration steps into goroutines so they complete sooner.