linkedin / li-apache-kafka-clients

li-apache-kafka-clients is a wrapper library for the Apache Kafka vanilla clients. It provides additional features such as large message support and auditing to the Java producer and consumer in the open source Apache Kafka.
BSD 2-Clause "Simplified" License
133 stars 52 forks source link

Add new LiKafkaConsumerImpl metric - _offsetInvalidOrOutRangeCounter to estimate the potential data loss #185

Closed zsong-li closed 4 years ago

zsong-li commented 4 years ago

Added new metric( _offsetInvalidOrOutRangeCounter) into LiKafkaConsumerImpl to count number of OffsetOutOfRange exceptions under following cases with liclosest reset strategy.

**1. fetchedOffset < Log Start Offset (LSO)

  1. fetchedOffset <= Log End Offset (LEO)**

This metric can be used as indicator to estimate the potential data loss upon the LiKafkaConsumer client reseting the offset to "liclosest"