When Kafka consumer does not have access to topic, it gets org.apache.kafka.common.errors.TopicAuthorizationException with Not authorized to access topics: [topicA, topicB]. As this is a perm issue, there are high chances that it will not be resolved right away, so the subsequent poll for the same topic may get the same error. This can cause the issue if the consumer was subscribed to more topics' partitions which consumer has access for and no issues.
This PR will add the topics' partitions into the pause list if the consumer does not have access. The auto-pause is for 5 minutes(config driven); same as producer side pause.
before each poll, task checks if there are any partitions to be paused or resumed. This logic is already there, which will take care or resuming pause created due to consumer side access issue.
Add leader busy metrics to know how busy leader is
When Kafka consumer does not have access to topic, it gets
org.apache.kafka.common.errors.TopicAuthorizationExceptionwithNot authorized to access topics: [topicA, topicB]. As this is a perm issue, there are high chances that it will not be resolved right away, so the subsequent poll for the same topic may get the same error. This can cause the issue if the consumer was subscribed to more topics' partitions which consumer has access for and no issues.This PR will add the topics' partitions into the pause list if the consumer does not have access. The auto-pause is for 5 minutes(config driven); same as producer side pause.
before each poll, task checks if there are any partitions to be paused or resumed. This logic is already there, which will take care or resuming pause created due to consumer side access issue.