kafka output zone awareness

[danielmotaleite]

Describe the enhancement: Right now filebeat outputs to all kafka nodes and partitions, even if in a different zone. With a new option, kafka output should look to it own zone and find kafkas in the same zone and use then, only fallback to the other zone kafkas if the first one is unavailable/inexistent.

Describe a specific use case for the enhancement or feature: This will save costs, having one zone filebeat talking to their own kafka node will reduce cross-zone traffic. Kafka will then rebalance their partitions and replicas between zones. Later logstash could do the same zone awareness for consuming events. Check this example: https://amplitude.engineering/reducing-costs-with-az-awareness-efc92bc7113a

[danielmotaleite]

Of course, the same issue can be applied while talking to logstash or elasticsearch directly, so maybe this could be renamed beat output zone awareness?

[hamishforbes]

Has there been any progress on this? I'm interested in this as a feature too

FWIW segmentio/kafka-go supports rack-aware balancing, it may be easier to switch out sarama than re-implement the functionality

[botelastic[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[hamishforbes]

This would still be a useful feature, not stale!

[botelastic[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[danielmotaleite]

please reopen, this is still a useful feature

[akamac]

/reopen

[Voodoolay]

/reopen