Multiple rebalanced because coordinator is dead

[duylong]

Hi,

I have multiples disconnect since update to Logstash 6.2.2, no issue with Logstash 5.6.5. Logstash notice a lost coordinator while there are no problems:

[INFO ][org.apache.kafka.common.utils.AppInfoParser] Kafka version : 1.0.0
[INFO ][org.apache.kafka.common.utils.AppInfoParser] Kafka commitId : aaa7af6d4a11b29d

[INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator] [Consumer clientId=logstash-1, groupId=logstash] Successfully joined group with generation 866
[INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator] [Consumer clientId=logstash-2, groupId=logstash] Successfully joined group with generation 866
[INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator] [Consumer clientId=logstash-0, groupId=logstash] Successfully joined group with generation 866
[INFO ][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator] [Consumer clientId=logstash-0, groupId=logstash] Setting newly assigned partitions [MY_TOPIC-0, MY_TOPIC-1]
[INFO ][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator] [Consumer clientId=logstash-1, groupId=logstash] Setting newly assigned partitions [MY_TOPIC-6, MY_TOPIC-7]
[INFO ][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator] [Consumer clientId=logstash-2, groupId=logstash] Setting newly assigned partitions [MY_TOPIC-8]
...
[INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator] [Consumer clientId=logstash-0, groupId=logstash] Marking the coordinator kafka3:9092 (id: 2147483645 rack: null) dead
[INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator] [Consumer clientId=logstash-1, groupId=logstash] Marking the coordinator kafka3:9092 (id: 2147483645 rack: null) dead
[INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator] [Consumer clientId=logstash-2, groupId=logstash] Marking the coordinator kafka3:9092 (id: 2147483645 rack: null) dead
[INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator] [Consumer clientId=logstash-2, groupId=logstash] Discovered coordinator kafka2:9092 (id: 2147483646 rack: null)
[INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator] [Consumer clientId=logstash-0, groupId=logstash] Discovered coordinator kafka2:9092 (id: 2147483646 rack: null)
[INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator] [Consumer clientId=logstash-1, groupId=logstash] Discovered coordinator kafka2:9092 (id: 2147483646 rack: null)
...
[ERROR][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator] [Consumer clientId=logstash-2, groupId=logstash] Offset commit failed on partition MY_TOPIC-8 at offset 2461009149: The coordinator is not aware of this member.
[WARN ][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator] [Consumer clientId=logstash-2, groupId=logstash] Asynchronous auto-commit of offsets {MY_TOPIC-8=OffsetAndMetadata{offset=2461009149, metadata=''}} failed: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.

The messages are looped. No problem with my second Logstash that has not been updated.

Kafka logs:

 INFO [GroupCoordinator 1]: Preparing to rebalance group logstash with old generation 865 (__consumer_offsets-49) (kafka.coordinator.group.GroupCoordinator)
 INFO [GroupCoordinator 1]: Stabilized group logstash generation 866 (__consumer_offsets-49) (kafka.coordinator.group.GroupCoordinator)
 INFO [GroupCoordinator 1]: Assignment received from leader for group logstash for generation 866 (kafka.coordinator.group.GroupCoordinator)
 INFO [GroupCoordinator 1]: Member logstash-0-0deec638-4b37-49ab-aae9-aaabbbccc in group logstash has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
...

• Kafka version 1.0
• RHEL7
• Config:

  input {
      kafka {
              type => "syslog"
              codec => "json"
              bootstrap_servers => "kafka:9092"
              topics => [ "MY_TOPIC" ]
              auto_offset_reset => "latest"
              group_id => "logstash"
              enable_metric => "false"
              consumer_threads => 3
      }
  }
  ...

  Somebody have also this issue ?

[sshantveer]

I have a similar issue. But I am seeing this on Logstash 5.6.7 with Kafka broker 0.11.0.2

input { kafka { codec => json bootstrap_servers => "${KAFKA_CONNECT}" group_id => "logstash-indexer" topics => "logstash" consumer_threads => "16" request_timeout_ms => "305000" } }

The message I am seeing is -

[WARN ][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator] Auto offset commit failed for group logstash-indexer-onprem: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured session.timeout.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.

[duylong]

My temporaly solution, downgrade to Logstash 5.6.x and update the kafka plugin.