KMinion is a feature-rich Prometheus exporter for Apache Kafka written in Go. It is lightweight and highly configurable so that it will meet your requirements.
MIT License
601
stars
123
forks
source link
How to fetch Earliest_offset Timestamp Message for each partition of kafka Topic #220
I'm Currently working on a project to analyse record in kafka topic.I have found Kminion to be a very useful tool, providing various metrics and functionalities to monitor my cluster. However, I've encountered a challenge while trying to get the timestamp associated with the first(earliest / beggining) offset in each partition of the topic.
To give you more context, In my cluster, there are retention policies configuration, which determine the deletion of old messages. After an exploration of the kminion doc and code, and even an attempt to use the ListOffsets method, i haven't found a way to perform this particulat task.
My goal is to get the timestamp of the first message in each partition, as it important for me to know how much time is left befor these messages become unavailable for consumption, given the retention policies defined in my cluster.
I've also considered using a kafka consumer to do this, but i'd like to avoid this approach due to some constraints in my use case. Instead, i'm looking for a way to retrieve the timestamp via the kafka-API without the need for a consumer.
So, if you have any bright ideas, clever suggestions, or other awesome Golang modules that allow me to grab this timestamp witout the need for a consumer group, I'm all ears !
Greetings,
I'm Currently working on a project to analyse record in kafka topic.I have found Kminion to be a very useful tool, providing various metrics and functionalities to monitor my cluster. However, I've encountered a challenge while trying to get the timestamp associated with the first(earliest / beggining) offset in each partition of the topic.
To give you more context, In my cluster, there are retention policies configuration, which determine the deletion of old messages. After an exploration of the kminion doc and code, and even an attempt to use the ListOffsets method, i haven't found a way to perform this particulat task.
My goal is to get the timestamp of the first message in each partition, as it important for me to know how much time is left befor these messages become unavailable for consumption, given the retention policies defined in my cluster.
I've also considered using a kafka consumer to do this, but i'd like to avoid this approach due to some constraints in my use case. Instead, i'm looking for a way to retrieve the timestamp via the kafka-API without the need for a consumer.
So, if you have any bright ideas, clever suggestions, or other awesome Golang modules that allow me to grab this timestamp witout the need for a consumer group, I'm all ears !