confluentinc / confluent-kafka-go

Confluent's Apache Kafka Golang client
Apache License 2.0
4.63k stars 658 forks source link

How to implement redundant kafka logic #493

Closed evgeniyasti closed 7 months ago

evgeniyasti commented 4 years ago

Hello.

My team is implementing Kafka redundancy. I need to be able to switch between instances if one instance goes down.

Is there a feature in the library for this or I need to handle it programmatically?

Thanks

mhowlett commented 4 years ago

If you just mean failover between broker instances within a single cluster, that is standard behavior - recommend reading up on Kafka fundamentals. There's no real reason to have redundant clusters in the same data center. If you are looking for cross-dc redundancy, here's the current state of affairs: https://www.confluent.io/blog/multi-region-data-replication/ . The confluent go client supports fetch-from-follower.

milindl commented 7 months ago

Closing this with Matt's answer. Please reopen if needed.