logstash-plugins / logstash-integration-kafka

Kafka Integration for Logstash, providing Input and Output Plugins
Apache License 2.0
32 stars 60 forks source link

Add `static membership` Kafka feature to clients #134

Closed andsel closed 1 year ago

andsel commented 1 year ago

The static membership concept is proposed and explained in KIP-345.

When a client of a consumer group reconnects to a cluster it triggers a re-balance, to assign partition to consumers and this consumer is assigned an id named member.id which is randomly generated. If a client is member of a consumer groups, and reconnects frequently is mean it triggers frequent re-balances. To avoid this re-assignment of consumer to partitions during re-balancing, a static member.id could be passed by the consumer, to uniquely identify it inside the consumer group and in stored on both sides (broker and consumer). In this way the static group.instance.id avoids re-balances in case of continuous disconnections and re-connections. Also with Static Membership however, re-balances could happen when a new member joins the group or a leader re-joins, for a full list of cases where re-balances happen also with static membership, please check KIP-345

andsel commented 1 year ago

What happens if group.instance.id is not static?

From paragraph Join Group Logic Change in KIP-345

The edge case is that if we could have members with the same `group.instance.id` (for example mis-configured 
instances with a valid `member.id` but added a used `group.instance.id` on runtime). 
When `group.instance.id` has duplicates, we could refuse join request from members with an outdated 
`member.id`...[omissis]...In an edge case where the client hits this exception in the response, 
it is suggesting that some other consumer takes its spot. The client should immediately fail itself 
to inform end user that there is a configuration bug which is generating duplicate consumers 
with same identity

So in that case the client would fail the joining request.