Yelp / kafka-utils

Apache License 2.0
313 stars 127 forks source link

Allow using Kafka AdminClient for kafka-consumer-manager commands #252

Closed flavray closed 4 years ago

flavray commented 4 years ago

Reading the __consumer_offsets topic at every run is slow and can add unwanted load on a Kafka cluster. The AdminClient allows to get the list of consumer groups and the list of topics from a group id in constance time.

This is currently a feature flag to keep backwards compatibility, but will later be enabled by default.

--use-admin-client

flavray commented 4 years ago

Example runs:

$ time kafka-consumer-manager --cluster-type <> --cluster-name <> offset_get <redacted> --use-admin-client
Cluster name: <>, consumer group: <redacted>
Topic Name: <redacted>  Total Distance: 12067145
    Partition ID: 0
        High Watermark: 12067145
        Low Watermark: 0
        Current Offset: 0
        Offset Distance: 12067145
        Percentage Distance: 100.0%
Topic Name: <redacted>  Total Distance: 1850
    Partition ID: 0
        High Watermark: 197266485
        Low Watermark: 0
        Current Offset: 197264635
        Offset Distance: 1850
        Percentage Distance: 0.0%

real    0m5.371s
user    0m4.232s
sys 0m0.200s

$ time kafka-consumer-manager --cluster-type <> --cluster-name <> offset_get <redacted>
Cluster name: <>, consumer group: <redacted>
Topic Name: <redacted> Total Distance: 421
    Partition ID: 0
        High Watermark: 197269431
        Low Watermark: 0
        Current Offset: 197269010
        Offset Distance: 421
        Percentage Distance: 0.0%

real    4m15.218s
user    3m59.092s
sys 0m11.048s
flavray commented 4 years ago

Ha. I had forgotten about OffsetWriter. The build should be fixed on the next run :)