Consume kafka message unstable in logstash 2.3.4

[cangchen8180]

hi guy, now i have a knotty problem.

logstash cann't consume kafka message continuously.like this

sometimes be worse, like this

I not find out abnormal figure after trace gc times and heap size. And no error message in logstash.log.

message flow

Kafka-> Logstash -> ES

version

logstash	2.3.4
kafka	0.8.2.1

I have three logstash, and one topic have 20 partitions. so i config consumer_threads is 7.

logstash config

kafka {
        zk_connect => "oss-mon-kafka-bjc-001:2181,oss-mon-kafka-bjc-002:2181,oss-mon-kafka-bjc-003:2181"
        group_id => "logstash_app_consumer"
        topic_id => "app_netty_log_topic"
        reset_beginning => false
        consumer_threads => 7
        decorate_events => false  
        type => "appnettyjava"
        codec => plain
    }
}

So please give me some tip !

[gquintana]

You should:

• Reduce the number of consumer_threads (one thread can consume several partitions)
• Enable Log4J logging to get Kafka consumer logs (just had -Dlog4j.configuration=... in Java options)
• Check Kafka server logs

What I suspect: with Kafka <=0.10.0 (Logstash input plugin <=5.x), as soon as Logstash gets overwhelmed, Kafka thinks the consumer is dead are reallocates partitions to another node which in turn gets saturated.

[ssozonoff]

I see this issue is still open, we are seeing something similar.

Logstash 5.4.1 and logstash-input-kafka-6.3.2

Our Logstash just freezes, nothing in the logs to indicate the problem and Kafka side it seems like the consumer have just disappeared. Restarting Logstash and we are away again for a random amount of time.

[zhsh87]

logstash 5.2.0
kafka 0.10.1 i found logstash consumer the kafka topic, when i want to use kafka-consumer-groups.sh to check the consumer suitation at kafka server , but kafka-consumer-groups.sh didn't found the consumer group info at the zookeeper . Did this different than before? I am sure when used logstash 2.4.0 , it can be found consumer group at the zookeeper.