search

segmentio / kafka-go

Kafka library in Go
MIT License
7.53k stars 780 forks source link

Is there some way to get current lag of the current consumer group? #862

Open jiangjinggit opened 2 years ago

jiangjinggit commented 2 years ago

Is there some way to get current lag of the current consumer group?I want to get the lag of each partition under the consumption group.Our project relies heavily on this.Confluent Kafka go supports it.I think it is a very normal demand,thank you.

achille-roussel commented 2 years ago

Hello @jiangjinggit

We usually recommend using an out-of-band system like Burrow to report consumer group lag.

Would this be an acceptable solution to your problem? Or are there reasons why you would prefer to have partition lag reported in the consumer application?

Jrmy2402 commented 2 years ago

Hello, I think it could be great to have the possibility of having the lag with the lib Kafka-go so that we can monitor the lag without another system. Because now for making this, we have to need to use another lib like Sarama or confluent-Kafka-go

achille-roussel commented 2 years ago

Hello @Jrmy2402, thanks for contributing to the discussion.

You can retrieve the list of committed consumer group offsets, and partition offsets through these two API calls:

These are the building blocks upon which lag computation can be done, being the delta between the committed offset of a partition and its last offset.

I know this is not a fully packaged solution, I just wanted to point out the existing building blocks that are available to solve this problem.

If you think kafka-go should provide more support to solve these issues, would you have a suggestion of what that API would ideally look like to best suit your applications?

jiangjinggit commented 2 years ago

Thank you for your answer,I've used Sarama to get the lag,Thanks again!

achille-roussel commented 2 years ago

Is this solution satisfying to you? In which case I will close the ticket, otherwise please let me know what alternative you would prefer.

ys93239 commented 2 years ago

Thank you for your answer,I've used Sarama to get the lag,Thanks again!

@jiangjinggit Did you have to move over to Sarama entirely or were you able to use it with segmentio somehow?

jiangjinggit commented 2 years ago

I just use Sarama for lag acquisition, and I still use segmentio for other functions

achille-roussel commented 2 years ago

Hello @jiangjinggit,

would you be able to share more details about how you're using Sarama for lag computation?

prakharrke commented 1 year ago

Upon calling these API, I get an error "fmt.WrapError" Unexpected EOF. Could you please help me understand that better?

sansmoraxz commented 1 year ago

You can retrieve the list of committed consumer group offsets, and partition offsets through these two API calls:

These are the building blocks upon which lag computation can be done, being the delta between the committed offset of a partition and its last offset.

I know this is not a fully packaged solution, I just wanted to point out the existing building blocks that are available to solve this problem.

If you think kafka-go should provide more support to solve these issues, would you have a suggestion of what that API would ideally look like to best suit your applications?

It would be nice if reader had a direct implementation of this. Like as a map for each topic and partition when backed by a consumer group.

zouyaowen commented 1 year ago

Is there any API that can get consumer-group through topic?The API Client.ListGroups just get all consumer group,but I don't know the consumer relation between consumer group and topic?Any suggestions?

sciclon2 commented 3 weeks ago

You coud use this : https://github.com/sciclon2/kafka-lag-go