helpshift / ekaf

A minimal, high-performance Kafka client in Erlang.
https://engineering.helpshift.com
Other
165 stars 50 forks source link

Forcefully disconnect from old brokers, if not present in new metadata #11

Open bosky101 opened 9 years ago

bosky101 commented 9 years ago

When doing something like a migration of kafka clusters - ekaf does not need to be restarted.

doing a dns change of ekaf_bootstrap_broker will suffice and ekaf picks this up automatically and starts sending to new brokers without any downtime. however if the older brokers are still up - it continues sending to them.

eg: suppose metadata1 comprises of broker1,broker2 and metadata2 comprises of broker2, broker3

ekaf currently does not stop workers of broker1 which it should broker1 is not in metadata2 although the connection is still up.

on the other hand this is interesting because, ekaf is able to send to 2 kafka clusters. but still marking this as a bug that should be fixed.

~B