Kafka consumer offset becomes undefined after few hours of inactivity.

ashoksahoo commented 7 years ago

After few hours of inactivity(No messages to consume), the consumer offset in Kafka becomes undefined.

~/kafka_2.11-0.10.0.0$ bin/kafka-consumer-groups.sh --bootstrap-server kafka1:9092 --describe --group metrics-workflow --new-consumer

After few hours of inactivity

GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER metrics-workflow workflow 0 unknown 123202 unknown consumer276741495101558201/13.0.11.7 metrics-workflow workflow 2 unknown 122358 unknown consumer276741495101558201/13.0.11.7 metrics-workflow workflow 4 unknown 123927 unknown consumer276741495101558201/13.0.11.7 metrics-workflow workflow 1 unknown 123880 unknown consumer160561495101534393/13.0.11.8 metrics-workflow workflow 3 unknown 125532 unknown consumer160561495101534393_/13.0.11.8

After adding some messages to the Queue.

GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER metrics-workflow workflow 0 123225 123233 8 consumer276741495101558201/13.0.11.7 metrics-workflow workflow 2 122371 122380 9 consumer276741495101558201/13.0.11.7 metrics-workflow workflow 4 123950 123958 8 consumer276741495101558201/13.0.11.7 metrics-workflow workflow 1 123893 123897 4 consumer160561495101534393/13.0.11.8 metrics-workflow workflow 3 125553 125564 11 consumer160561495101534393_/13.0.11.8

I am using no-kafka on all the consumer nodes.

oleksiyk commented 7 years ago

I don't think its something with the library. Just make sure your code doesn't make commitOffset without offset field or set to undefined.

ashoksahoo commented 7 years ago

if(commitOffsets && commitOffsets.length>0) {
            consumer.commitOffset(commitOffsets)
                .then(function () {
                    logger.debug("kafka commit at "+JSON.stringify(commitOffsets));
                    resolve();
                })
                .catch(function (error) {
                    logger.error("failure in kafka commit " + error);
                    reject();
                });
        }else{
            logger.error("empty commit object");
            reject("empty commit object")
        }

This should Avoid Situations like this, but this is reproducing across every environment, when there is no activity in the topic.

I checked the logs, I found no one is committing anything undefined/falsey/ambiguous.

[DEBUG] KafkaConsumerInterface - kafka commit at [{"topic":"workflow","partition":2,"offset":122371}]
[2017-05-26 10:14:01.985] [DEBUG] KafkaConsumerInterface - kafka commit at [{"topic":"workflow","partition":4,"offset":123950}]
[2017-05-26 10:14:01.986] [DEBUG] KafkaConsumerInterface - kafka commit at [{"topic":"workflow","partition":0,"offset":123225}]
[2017-05-26 10:14:08.210] [DEBUG] KafkaConsumerInterface - kafka commit at [{"topic":"workflow","partition":4,"offset":123958}]
[2017-05-26 10:14:08.211] [DEBUG] KafkaConsumerInterface - kafka commit at [{"topic":"workflow","partition":2,"offset":122380}]
[2017-05-26 10:14:08.215] [DEBUG] KafkaConsumerInterface - kafka commit at [{"topic":"workflow","partition":0,"offset":123233}]

oleksiyk commented 7 years ago

no-kafka never makes commits itself, so this is definitely not an issue with the library.

oleksiyk / kafka

Kafka consumer offset becomes undefined after few hours of inactivity. #175