redpanda-data / kminion

KMinion is a feature-rich Prometheus exporter for Apache Kafka written in Go. It is lightweight and highly configurable so that it will meet your requirements.
MIT License
601 stars 123 forks source link

Collect consumer groups data can't be disabled #243

Open timchenko-a opened 5 months ago

timchenko-a commented 5 months ago

Hi, I'd like to use Kminion only for E2E monitoring. So my config looks like this:

minion:
  consumerGroups:
    enabled: false
  topics:
    ignoredTopics: [ ".*" ]
  logDirs:
    enabled: false
  endToEnd:
    enabled: true

Based on the info from documentation, I assume it shouldn't try to get consumer groups from brokers. But it actually trying: Kminion log example:

{"level":"warn","ts":"2024-02-08T12:56:07.910Z","logger":"main.minion_service","msg":"failed to fetch consumer group offsets, inner kafka error","consumer_group":"SOME_NON_KMINION_GROUP_NAME","error":"failed to request group offsets for group 'SOME_NON_KMINION_GROUP_NAME': GROUP_AUTHORIZATION_FAILED: Not authorized to access group: Group authorization failed."}

Kafka auth log:

{"short_message":"Principal = User:kminion is Denied Operation = Describe from host = EDITED on resource = Group:LITERAL:SOME_NON_KMINION_GROUP_NAME","full_message":"Principal = User:kminion is Denied Operation = Describe from host = EDITED on resource = Group:LITERAL:SOME_NON_KMINION_GROUP_NAME","timestamp":1.707396472924E9,"level":6,"facility":"logstash-gelf","LoggerName":"kafka.authorizer.logger","SourceSimpleClassName":"SimpleAclAuthorizer$BaseAuthorizer","SourceClassName":"kafka.security.auth.SimpleAclAuthorizer$BaseAuthorizer","Time":"2024-02-08 12:47:52,924","Severity":"INFO","SourceLineNumber":66,"Thread":"data-plane-kafka-request-handler-0","SourceMethodName":"logAuditMessage"}

I suspect this is due to collectConsumerGroupLags function not having enabled check, like e.g. in collectConsumerGroups . I've also tried using ignoredGroups: [ ".*" ], but since offset request happens earlier this doesn't have any effect.

timchenko-a commented 3 months ago

Hi @weeco , if this is actually a bug, not a feature, I can try working on this :)