This PR changes the leader election logic so it relies on Connect's consumer group. Previously, the leader (coordinator) was selected based on the task that was assigned partition 0 of the first topic, however this required defining a static list of topics in the topics property and did not work when using topics.regex.
This PR also uses the Connect consumer group to determine the membership count, which is used to check when a commit is ready to proceed. Previously this was determined by comparing the total partition count for the topics defined, which also required a static list of topics.
This PR changes the leader election logic so it relies on Connect's consumer group. Previously, the leader (coordinator) was selected based on the task that was assigned partition 0 of the first topic, however this required defining a static list of topics in the
topicsproperty and did not work when usingtopics.regex.This PR also uses the Connect consumer group to determine the membership count, which is used to check when a commit is ready to proceed. Previously this was determined by comparing the total partition count for the topics defined, which also required a static list of topics.