search

linkedin / Burrow

Kafka Consumer Lag Checking
Apache License 2.0
3.73k stars 797 forks source link

Query: Multiple Burrow instances #785

Open shivam1040 opened 1 year ago

shivam1040 commented 1 year ago

What would happen if I have a cluster of burrow pointing to the one broker? Ideally the burrow consumer should get rebalanced but are the metrics duplicated or something I should take into consideration?

Rahulreddy1020 commented 1 year ago

Since each Burrow consumer will be fetching the same data from the broker and duplicating the metrics can lead to performance issue and wasting your resources. This can lead to wasted resources and performance issues. ideally we should point one Burrow consumer to each broker.

shivam1040 commented 1 year ago

@Rahulreddy1020 thanks for the response. I guessed something like that when metrics were abruptly higher than the actual. On an ending note, what would you suggest for Burrow configs for a heavy duty Kafka cluster, say 70+ brokers with 4000+ topics