search

logstash-plugins / logstash-input-kafka

Kafka input for Logstash
Apache License 2.0
139 stars 122 forks source link

Kafka input plug-in skip reading from single partition from assigned topic #338

Open patelprakashp opened 3 years ago

patelprakashp commented 3 years ago

Logstash information:

Please include the following information:

  1. Logstash version : 6.8.14
  2. Logstash installation source : Docker
  3. How is Logstash being run : Docker
  4. How was the Logstash Plugin installed: Default
  5. Kafka-Input Plug-in : 9.1.0
  6. Kafka client : 2.3.0

I have logstash running on Kubernetes and consuming events from 100+ topics.Each topic have 3+ partitions. Thus Single consumer group subscribed to 100+ topics and consumes events from 500+ partitions. Below are the Kafka input configuration.I am running 10 instances of logstash on kubernetes so total 100 consumer threads consuming from 500 partitions.

auto_offset_reset => "earliest" enable_auto_commit => "true" group_id => "test_logstash" consumer_threads => "10" max_poll_records => "500" heartbeat_interval_ms => "9000" session_timeout_ms => "30000" fetch_max_bytes => "10485760" max_partition_fetch_bytes => "524288" client_id => "test_logstash" decorate_events => true partition_assignment_strategy => "org.apache.kafka.clients.consumer.RoundRobinAssignor"

Kafka consumer stops reading data from single partition although its subscribed and assigned to consumer group.