search

Aiven-Open / klaw

Klaw, the latest OS tool by Aiven, helps enterprises cope with Apache Kafka(r) topics, schema registry and connectors governance by introducing roles/authorizations to users of various teams of an org.
https://www.klaw-project.io/
Apache License 2.0
150 stars 60 forks source link

Topic events cannot be read #2671

Open yvessavoy opened 1 month ago

yvessavoy commented 1 month ago

What happened?

Klaw always shows the message "No messages in topic" even if there are messages (I verified that with the CLI).

I am not sure where the issue now lies. Is it an ACL problem or some issue in Klaw. Also, I'm working with the strimzi-operator and it doesn't allow me to create a user with the name "KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup", maybe that is the problem? Any advice is appreciated!

What did you expect to happen?

All events in the topic are displayed

What else do we need to know?

I set the klaw.topiccontents.consumergroup.id-property to the value "kwgenericconsumergroup". I also created an ACL like this: ./kafka-acls.bat --bootstrap-server localhost:9092 --topic ewz-sap-test --add --group kwgenericconsumergroup --operation read --allow-principal User:*

The cluster-api-log (set to debug) shows the following messages:

2024-10-24T12:16:06.357Z DEBUG 7 --- [nio-9343-exec-6] o.s.security.web.FilterChainProxy        : Securing GET /topics/getTopicContents/ewz-lori-kafka-kafka-bootstrap.ewz-lori-kafka-dev.svc:9092/PLAINTEXT/notdefined/ewz-sap-test/custom/partitionId/0/selectedNumberOfOffsets/3/DEV_CLUSTER1/rangeOffsets/-1/-1
2024-10-24T12:16:06.360Z DEBUG 7 --- [nio-9343-exec-6] o.s.security.web.FilterChainProxy        : Secured GET /topics/getTopicContents/ewz-lori-kafka-kafka-bootstrap.ewz-lori-kafka-dev.svc:9092/PLAINTEXT/notdefined/ewz-sap-test/custom/partitionId/0/selectedNumberOfOffsets/3/DEV_CLUSTER1/rangeOffsets/-1/-1
2024-10-24T12:16:06.360Z DEBUG 7 --- [nio-9343-exec-6] o.s.web.servlet.DispatcherServlet        : GET "/topics/getTopicContents/ewz-lori-kafka-kafka-bootstrap.ewz-lori-kafka-dev.svc:9092/PLAINTEXT/notdefined/ewz-sap-test/custom/partitionId/0/selectedNumberOfOffsets/3/DEV_CLUSTER1/rangeOffsets/-1/-1", parameters={}
2024-10-24T12:16:06.361Z DEBUG 7 --- [nio-9343-exec-6] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to io.aiven.klaw.clusterapi.controller.TopicContentsController#getTopicContents(String, String, String, String, String, Integer, Integer, String, Integer, Integer)
2024-10-24T12:16:06.362Z DEBUG 7 --- [nio-9343-exec-6] i.a.k.c.services.TopicContentsService    : readEvents bootStrapServers ewz-lori-kafka-kafka-bootstrap.ewz-lori-kafka-dev.svc:9092, protocol PLAINTEXT,  consumerGroupId notdefined, topicName ewz-sap-test, offsetPosition custom,  readMessagesType OFFSET_ID clusterIdentification DEV_CLUSTER1 selectedPartitionId 0 selectedNumberOfOffsets 3 rangeOffsetsStart -1 rangeOffsetsEnd -1
2024-10-24T12:16:06.362Z  INFO 7 --- [nio-9343-exec-6] o.a.k.clients.consumer.ConsumerConfig    : ConsumerConfig values:
    allow.auto.create.topics = true
    auto.commit.interval.ms = 5000
    auto.include.jmx.reporter = true
    auto.offset.reset = latest
    bootstrap.servers = [ewz-lori-kafka-kafka-bootstrap.ewz-lori-kafka-dev.svc:9092]
    check.crcs = true
    client.dns.lookup = use_all_dns_ips
    client.id = KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup
    client.rack =
    connections.max.idle.ms = 540000
    default.api.timeout.ms = 60000
    enable.auto.commit = false
    enable.metrics.push = true
    exclude.internal.topics = true
    fetch.max.bytes = 52428800
    fetch.max.wait.ms = 500
    fetch.min.bytes = 1
    group.id = kwgenericconsumergroup
    group.instance.id = null
    group.protocol = classic
    group.remote.assignor = null
    heartbeat.interval.ms = 3000
    interceptor.classes = []
    internal.leave.group.on.close = true
    internal.throw.on.fetch.stable.offset.unsupported = false
    isolation.level = read_uncommitted
    key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
    max.partition.fetch.bytes = 1048576
    max.poll.interval.ms = 300000
    max.poll.records = 500
    metadata.max.age.ms = 300000
    metric.reporters = []
    metrics.num.samples = 2
    metrics.recording.level = INFO
    metrics.sample.window.ms = 30000
    partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor, class org.apache.kafka.clients.consumer.CooperativeStickyAssignor]
    receive.buffer.bytes = 65536
    reconnect.backoff.max.ms = 1000
    reconnect.backoff.ms = 50
    request.timeout.ms = 30000
    retry.backoff.max.ms = 1000
    retry.backoff.ms = 100
    sasl.client.callback.handler.class = null
    sasl.jaas.config = null
    sasl.kerberos.kinit.cmd = /usr/bin/kinit
    sasl.kerberos.min.time.before.relogin = 60000
    sasl.kerberos.service.name = null
    sasl.kerberos.ticket.renew.jitter = 0.05
    sasl.kerberos.ticket.renew.window.factor = 0.8
    sasl.login.callback.handler.class = null
    sasl.login.class = null
    sasl.login.connect.timeout.ms = null
    sasl.login.read.timeout.ms = null
    sasl.login.refresh.buffer.seconds = 300
    sasl.login.refresh.min.period.seconds = 60
    sasl.login.refresh.window.factor = 0.8
    sasl.login.refresh.window.jitter = 0.05
    sasl.login.retry.backoff.max.ms = 10000
    sasl.login.retry.backoff.ms = 100
    sasl.mechanism = GSSAPI
    sasl.oauthbearer.clock.skew.seconds = 30
    sasl.oauthbearer.expected.audience = null
    sasl.oauthbearer.expected.issuer = null
    sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000
    sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000
    sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100
    sasl.oauthbearer.jwks.endpoint.url = null
    sasl.oauthbearer.scope.claim.name = scope
    sasl.oauthbearer.sub.claim.name = sub
    sasl.oauthbearer.token.endpoint.url = null
    security.protocol = PLAINTEXT
    security.providers = null
    send.buffer.bytes = 131072
    session.timeout.ms = 45000
    socket.connection.setup.timeout.max.ms = 30000
    socket.connection.setup.timeout.ms = 10000
    ssl.cipher.suites = null
    ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
    ssl.endpoint.identification.algorithm = https
    ssl.engine.factory.class = null
    ssl.key.password = null
    ssl.keymanager.algorithm = SunX509
    ssl.keystore.certificate.chain = null
    ssl.keystore.key = null
    ssl.keystore.location = null
    ssl.keystore.password = null
    ssl.keystore.type = JKS
    ssl.protocol = TLSv1.3
    ssl.provider = null
    ssl.secure.random.implementation = null
    ssl.trustmanager.algorithm = PKIX
    ssl.truststore.certificates = null
    ssl.truststore.location = null
    ssl.truststore.password = null
    ssl.truststore.type = JKS
    value.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
2024-10-24T12:16:06.362Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.c.internals.LegacyKafkaConsumer  : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Initializing the Kafka consumer
2024-10-24T12:16:06.363Z  INFO 7 --- [nio-9343-exec-6] o.a.k.c.t.i.KafkaMetricsCollector        : initializing Kafka metrics collector
2024-10-24T12:16:06.459Z  INFO 7 --- [nio-9343-exec-6] o.a.kafka.common.utils.AppInfoParser     : Kafka version: 3.7.0
2024-10-24T12:16:06.459Z  INFO 7 --- [nio-9343-exec-6] o.a.kafka.common.utils.AppInfoParser     : Kafka commitId: 2ae524ed625438c5
2024-10-24T12:16:06.459Z  INFO 7 --- [nio-9343-exec-6] o.a.kafka.common.utils.AppInfoParser     : Kafka startTimeMs: 1729772166459
2024-10-24T12:16:06.460Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.c.internals.LegacyKafkaConsumer  : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Kafka consumer initialized
2024-10-24T12:16:06.460Z  INFO 7 --- [nio-9343-exec-6] o.a.k.c.c.internals.LegacyKafkaConsumer  : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Subscribed to topic(s): ewz-sap-test
2024-10-24T12:16:06.460Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Sending FindCoordinator request to broker ewz-lori-kafka-kafka-bootstrap.ewz-lori-kafka-dev.svc:9092 (id: -1 rack: null)
2024-10-24T12:16:06.460Z DEBUG 7 --- [nio-9343-exec-6] org.apache.kafka.clients.ClientUtils     : Resolved host ewz-lori-kafka-kafka-bootstrap.ewz-lori-kafka-dev.svc as 10.201.5.208
2024-10-24T12:16:06.460Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.clients.ClusterConnectionStates    : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Resolved host ewz-lori-kafka-kafka-bootstrap.ewz-lori-kafka-dev.svc to addresses [ewz-lori-kafka-kafka-bootstrap.ewz-lori-kafka-dev.svc/10.201.5.208]
2024-10-24T12:16:06.460Z DEBUG 7 --- [nio-9343-exec-6] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Initiating connection to node ewz-lori-kafka-kafka-bootstrap.ewz-lori-kafka-dev.svc:9092 (id: -1 rack: null) using address ewz-lori-kafka-kafka-bootstrap.ewz-lori-kafka-dev.svc/10.201.5.208
2024-10-24T12:16:06.461Z DEBUG 7 --- [nio-9343-exec-6] o.apache.kafka.common.network.Selector   : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -1
2024-10-24T12:16:06.461Z DEBUG 7 --- [nio-9343-exec-6] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Completed connection to node -1. Fetching API versions.
2024-10-24T12:16:06.461Z DEBUG 7 --- [nio-9343-exec-6] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Initiating API versions fetch from node -1.
2024-10-24T12:16:06.461Z DEBUG 7 --- [nio-9343-exec-6] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Sending API_VERSIONS request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, correlationId=1, headerVersion=2) and timeout 30000 to node -1: ApiVersionsRequestData(clientSoftwareName='apache-kafka-java', clientSoftwareVersion='3.7.0')
2024-10-24T12:16:06.462Z DEBUG 7 --- [nio-9343-exec-6] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Received API_VERSIONS response from node -1 for request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, correlationId=1, headerVersion=2): ApiVersionsResponseData(errorCode=0, apiKeys=[ApiVersion(apiKey=0, minVersion=0, maxVersion=10), ApiVersion(apiKey=1, minVersion=0, maxVersion=16), ApiVersion(apiKey=2, minVersion=0, maxVersion=8), ApiVersion(apiKey=3, minVersion=0, maxVersion=12), ApiVersion(apiKey=8, minVersion=0, maxVersion=9), ApiVersion(apiKey=9, minVersion=0, maxVersion=9), ApiVersion(apiKey=10, minVersion=0, maxVersion=4), ApiVersion(apiKey=11, minVersion=0, maxVersion=9), ApiVersion(apiKey=12, minVersion=0, maxVersion=4), ApiVersion(apiKey=13, minVersion=0, maxVersion=5), ApiVersion(apiKey=14, minVersion=0, maxVersion=5), ApiVersion(apiKey=15, minVersion=0, maxVersion=5), ApiVersion(apiKey=16, minVersion=0, maxVersion=4), ApiVersion(apiKey=17, minVersion=0, maxVersion=1), ApiVersion(apiKey=18, minVersion=0, maxVersion=3), ApiVersion(apiKey=19, minVersion=0, maxVersion=7), ApiVersion(apiKey=20, minVersion=0, maxVersion=6), ApiVersion(apiKey=21, minVersion=0, maxVersion=2), ApiVersion(apiKey=22, minVersion=0, maxVersion=4), ApiVersion(apiKey=23, minVersion=0, maxVersion=4), ApiVersion(apiKey=24, minVersion=0, maxVersion=4), ApiVersion(apiKey=25, minVersion=0, maxVersion=3), ApiVersion(apiKey=26, minVersion=0, maxVersion=3), ApiVersion(apiKey=27, minVersion=0, maxVersion=1), ApiVersion(apiKey=28, minVersion=0, maxVersion=3), ApiVersion(apiKey=29, minVersion=0, maxVersion=3), ApiVersion(apiKey=30, minVersion=0, maxVersion=3), ApiVersion(apiKey=31, minVersion=0, maxVersion=3), ApiVersion(apiKey=32, minVersion=0, maxVersion=4), ApiVersion(apiKey=33, minVersion=0, maxVersion=2), ApiVersion(apiKey=34, minVersion=0, maxVersion=2), ApiVersion(apiKey=35, minVersion=0, maxVersion=4), ApiVersion(apiKey=36, minVersion=0, maxVersion=2), ApiVersion(apiKey=37, minVersion=0, maxVersion=3), ApiVersion(apiKey=38, minVersion=0, maxVersion=3), ApiVersion(apiKey=39, minVersion=0, maxVersion=2), ApiVersion(apiKey=40, minVersion=0, maxVersion=2), ApiVersion(apiKey=41, minVersion=0, maxVersion=3), ApiVersion(apiKey=42, minVersion=0, maxVersion=2), ApiVersion(apiKey=43, minVersion=0, maxVersion=2), ApiVersion(apiKey=44, minVersion=0, maxVersion=1), ApiVersion(apiKey=45, minVersion=0, maxVersion=0), ApiVersion(apiKey=46, minVersion=0, maxVersion=0), ApiVersion(apiKey=47, minVersion=0, maxVersion=0), ApiVersion(apiKey=48, minVersion=0, maxVersion=1), ApiVersion(apiKey=49, minVersion=0, maxVersion=1), ApiVersion(apiKey=50, minVersion=0, maxVersion=0), ApiVersion(apiKey=51, minVersion=0, maxVersion=0), ApiVersion(apiKey=55, minVersion=0, maxVersion=1), ApiVersion(apiKey=57, minVersion=0, maxVersion=1), ApiVersion(apiKey=60, minVersion=0, maxVersion=1), ApiVersion(apiKey=61, minVersion=0, maxVersion=0), ApiVersion(apiKey=64, minVersion=0, maxVersion=0), ApiVersion(apiKey=65, minVersion=0, maxVersion=0), ApiVersion(apiKey=66, minVersion=0, maxVersion=0), ApiVersion(apiKey=68, minVersion=0, maxVersion=0), ApiVersion(apiKey=74, minVersion=0, maxVersion=0)], throttleTimeMs=0, supportedFeatures=[SupportedFeatureKey(name='metadata.version', minVersion=1, maxVersion=19)], finalizedFeaturesEpoch=7365019, finalizedFeatures=[FinalizedFeatureKey(name='metadata.version', maxVersionLevel=19, minVersionLevel=19)], zkMigrationReady=false)
2024-10-24T12:16:06.462Z DEBUG 7 --- [nio-9343-exec-6] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Node -1 has finalized features epoch: 7365019, finalized features: [FinalizedFeatureKey(name='metadata.version', maxVersionLevel=19, minVersionLevel=19)], supported features: [SupportedFeatureKey(name='metadata.version', minVersion=1, maxVersion=19)], ZK migration ready: false, API versions: (Produce(0): 0 to 10 [usable: 10], Fetch(1): 0 to 16 [usable: 16], ListOffsets(2): 0 to 8 [usable: 8], Metadata(3): 0 to 12 [usable: 12], LeaderAndIsr(4): UNSUPPORTED, StopReplica(5): UNSUPPORTED, UpdateMetadata(6): UNSUPPORTED, ControlledShutdown(7): UNSUPPORTED, OffsetCommit(8): 0 to 9 [usable: 9], OffsetFetch(9): 0 to 9 [usable: 9], FindCoordinator(10): 0 to 4 [usable: 4], JoinGroup(11): 0 to 9 [usable: 9], Heartbeat(12): 0 to 4 [usable: 4], LeaveGroup(13): 0 to 5 [usable: 5], SyncGroup(14): 0 to 5 [usable: 5], DescribeGroups(15): 0 to 5 [usable: 5], ListGroups(16): 0 to 4 [usable: 4], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 3 [usable: 3], CreateTopics(19): 0 to 7 [usable: 7], DeleteTopics(20): 0 to 6 [usable: 6], DeleteRecords(21): 0 to 2 [usable: 2], InitProducerId(22): 0 to 4 [usable: 4], OffsetForLeaderEpoch(23): 0 to 4 [usable: 4], AddPartitionsToTxn(24): 0 to 4 [usable: 4], AddOffsetsToTxn(25): 0 to 3 [usable: 3], EndTxn(26): 0 to 3 [usable: 3], WriteTxnMarkers(27): 0 to 1 [usable: 1], TxnOffsetCommit(28): 0 to 3 [usable: 3], DescribeAcls(29): 0 to 3 [usable: 3], CreateAcls(30): 0 to 3 [usable: 3], DeleteAcls(31): 0 to 3 [usable: 3], DescribeConfigs(32): 0 to 4 [usable: 4], AlterConfigs(33): 0 to 2 [usable: 2], AlterReplicaLogDirs(34): 0 to 2 [usable: 2], DescribeLogDirs(35): 0 to 4 [usable: 4], SaslAuthenticate(36): 0 to 2 [usable: 2], CreatePartitions(37): 0 to 3 [usable: 3], CreateDelegationToken(38): 0 to 3 [usable: 3], RenewDelegationToken(39): 0 to 2 [usable: 2], ExpireDelegationToken(40): 0 to 2 [usable: 2], DescribeDelegationToken(41): 0 to 3 [usable: 3], DeleteGroups(42): 0 to 2 [usable: 2], ElectLeaders(43): 0 to 2 [usable: 2], IncrementalAlterConfigs(44): 0 to 1 [usable: 1], AlterPartitionReassignments(45): 0 [usable: 0], ListPartitionReassignments(46): 0 [usable: 0], OffsetDelete(47): 0 [usable: 0], DescribeClientQuotas(48): 0 to 1 [usable: 1], AlterClientQuotas(49): 0 to 1 [usable: 1], DescribeUserScramCredentials(50): 0 [usable: 0], AlterUserScramCredentials(51): 0 [usable: 0], DescribeQuorum(55): 0 to 1 [usable: 1], AlterPartition(56): UNSUPPORTED, UpdateFeatures(57): 0 to 1 [usable: 1], Envelope(58): UNSUPPORTED, DescribeCluster(60): 0 to 1 [usable: 1], DescribeProducers(61): 0 [usable: 0], UnregisterBroker(64): 0 [usable: 0], DescribeTransactions(65): 0 [usable: 0], ListTransactions(66): 0 [usable: 0], AllocateProducerIds(67): UNSUPPORTED, ConsumerGroupHeartbeat(68): 0 [usable: 0], ConsumerGroupDescribe(69): UNSUPPORTED, GetTelemetrySubscriptions(71): UNSUPPORTED, PushTelemetry(72): UNSUPPORTED, ListClientMetricsResources(74): 0 [usable: 0]).
2024-10-24T12:16:06.462Z DEBUG 7 --- [nio-9343-exec-6] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Sending metadata request MetadataRequestData(topics=[MetadataRequestTopic(topicId=AAAAAAAAAAAAAAAAAAAAAA, name='ewz-sap-test')], allowAutoTopicCreation=true, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) to node ewz-lori-kafka-kafka-bootstrap.ewz-lori-kafka-dev.svc:9092 (id: -1 rack: null)
2024-10-24T12:16:06.462Z DEBUG 7 --- [nio-9343-exec-6] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Sending METADATA request with header RequestHeader(apiKey=METADATA, apiVersion=12, clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, correlationId=2, headerVersion=2) and timeout 30000 to node -1: MetadataRequestData(topics=[MetadataRequestTopic(topicId=AAAAAAAAAAAAAAAAAAAAAA, name='ewz-sap-test')], allowAutoTopicCreation=true, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false)
2024-10-24T12:16:06.462Z DEBUG 7 --- [nio-9343-exec-6] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Give up sending telemetry request since no node is available
2024-10-24T12:16:06.463Z DEBUG 7 --- [nio-9343-exec-6] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Sending FIND_COORDINATOR request with header RequestHeader(apiKey=FIND_COORDINATOR, apiVersion=4, clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, correlationId=0, headerVersion=2) and timeout 30000 to node -1: FindCoordinatorRequestData(key='', keyType=0, coordinatorKeys=[kwgenericconsumergroup])
2024-10-24T12:16:06.463Z DEBUG 7 --- [nio-9343-exec-6] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Give up sending telemetry request since no node is available
2024-10-24T12:16:06.463Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.t.i.ClientTelemetryReporter      : Creating telemetry subscription request with client instance id AAAAAAAAAAAAAAAAAAAAAA
2024-10-24T12:16:06.463Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.t.i.ClientTelemetryReporter      : Setting telemetry state from SUBSCRIPTION_NEEDED to SUBSCRIPTION_IN_PROGRESS
2024-10-24T12:16:06.463Z DEBUG 7 --- [nio-9343-exec-6] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Version mismatch when attempting to send GetTelemetrySubscriptionsRequestData(clientInstanceId=AAAAAAAAAAAAAAAAAAAAAA) with correlation id 3 to -1
org.apache.kafka.common.errors.UnsupportedVersionException: The node does not support GET_TELEMETRY_SUBSCRIPTIONS
2024-10-24T12:16:06.463Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.t.i.ClientTelemetryReporter      : The broker generated an error for the get telemetry network API request
org.apache.kafka.common.errors.UnsupportedVersionException: The node does not support GET_TELEMETRY_SUBSCRIPTIONS
2024-10-24T12:16:06.463Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.t.i.ClientTelemetryReporter      : Updating intervalMs: 300000, lastRequestMs: 1729772166463
2024-10-24T12:16:06.463Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.t.i.ClientTelemetryReporter      : Setting telemetry state from SUBSCRIPTION_IN_PROGRESS to SUBSCRIPTION_NEEDED
2024-10-24T12:16:06.464Z DEBUG 7 --- [nio-9343-exec-6] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Received METADATA response from node -1 for request with header RequestHeader(apiKey=METADATA, apiVersion=12, clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, correlationId=2, headerVersion=2): MetadataResponseData(throttleTimeMs=0, brokers=[MetadataResponseBroker(nodeId=0, host='ewz-lori-kafka-brokers-0.ewz-lori-kafka-kafka-brokers.ewz-lori-kafka-dev.svc', port=9092, rack=null), MetadataResponseBroker(nodeId=1, host='ewz-lori-kafka-brokers-1.ewz-lori-kafka-kafka-brokers.ewz-lori-kafka-dev.svc', port=9092, rack=null), MetadataResponseBroker(nodeId=2, host='ewz-lori-kafka-brokers-2.ewz-lori-kafka-kafka-brokers.ewz-lori-kafka-dev.svc', port=9092, rack=null)], clusterId='UGCc1dchTtysb1GvPz0c7Q', controllerId=0, topics=[MetadataResponseTopic(errorCode=0, name='ewz-sap-test', topicId=c5la-z45R8-IvlaY_DFmqQ, isInternal=false, partitions=[MetadataResponsePartition(errorCode=0, partitionIndex=0, leaderId=1, leaderEpoch=22, replicaNodes=[1], isrNodes=[1], offlineReplicas=[]), MetadataResponsePartition(errorCode=0, partitionIndex=1, leaderId=0, leaderEpoch=14, replicaNodes=[0], isrNodes=[0], offlineReplicas=[]), MetadataResponsePartition(errorCode=0, partitionIndex=2, leaderId=1, leaderEpoch=10, replicaNodes=[1], isrNodes=[1], offlineReplicas=[])], topicAuthorizedOperations=-2147483648)], clusterAuthorizedOperations=-2147483648)
2024-10-24T12:16:06.464Z DEBUG 7 --- [nio-9343-exec-6] org.apache.kafka.clients.Metadata        : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Setting the last seen epoch of partition ewz-sap-test-0 to 22 since the last known epoch was undefined.
2024-10-24T12:16:06.464Z DEBUG 7 --- [nio-9343-exec-6] org.apache.kafka.clients.Metadata        : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Setting the last seen epoch of partition ewz-sap-test-1 to 14 since the last known epoch was undefined.
2024-10-24T12:16:06.464Z DEBUG 7 --- [nio-9343-exec-6] org.apache.kafka.clients.Metadata        : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Setting the last seen epoch of partition ewz-sap-test-2 to 10 since the last known epoch was undefined.
2024-10-24T12:16:06.464Z  INFO 7 --- [nio-9343-exec-6] org.apache.kafka.clients.Metadata        : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Cluster ID: UGCc1dchTtysb1GvPz0c7Q
2024-10-24T12:16:06.464Z DEBUG 7 --- [nio-9343-exec-6] org.apache.kafka.clients.Metadata        : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Updated cluster metadata updateVersion 2 to MetadataCache{clusterId='UGCc1dchTtysb1GvPz0c7Q', nodes={0=ewz-lori-kafka-brokers-0.ewz-lori-kafka-kafka-brokers.ewz-lori-kafka-dev.svc:9092 (id: 0 rack: null), 1=ewz-lori-kafka-brokers-1.ewz-lori-kafka-kafka-brokers.ewz-lori-kafka-dev.svc:9092 (id: 1 rack: null), 2=ewz-lori-kafka-brokers-2.ewz-lori-kafka-kafka-brokers.ewz-lori-kafka-dev.svc:9092 (id: 2 rack: null)}, partitions=[PartitionMetadata(error=NONE, partition=ewz-sap-test-0, leader=Optional[1], leaderEpoch=Optional[22], replicas=1, isr=1, offlineReplicas=), PartitionMetadata(error=NONE, partition=ewz-sap-test-1, leader=Optional[0], leaderEpoch=Optional[14], replicas=0, isr=0, offlineReplicas=), PartitionMetadata(error=NONE, partition=ewz-sap-test-2, leader=Optional[1], leaderEpoch=Optional[10], replicas=1, isr=1, offlineReplicas=)], controller=ewz-lori-kafka-brokers-0.ewz-lori-kafka-kafka-brokers.ewz-lori-kafka-dev.svc:9092 (id: 0 rack: null)}
2024-10-24T12:16:06.464Z DEBUG 7 --- [nio-9343-exec-6] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Received FIND_COORDINATOR response from node -1 for request with header RequestHeader(apiKey=FIND_COORDINATOR, apiVersion=4, clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, correlationId=0, headerVersion=2): FindCoordinatorResponseData(throttleTimeMs=0, errorCode=0, errorMessage='', nodeId=0, host='', port=0, coordinators=[Coordinator(key='kwgenericconsumergroup', nodeId=1, host='ewz-lori-kafka-brokers-1.ewz-lori-kafka-kafka-brokers.ewz-lori-kafka-dev.svc', port=9092, errorCode=0, errorMessage='')])
2024-10-24T12:16:06.464Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Received FindCoordinator response ClientResponse(receivedTimeMs=1729772166464, latencyMs=4, disconnected=false, timedOut=false, requestHeader=RequestHeader(apiKey=FIND_COORDINATOR, apiVersion=4, clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, correlationId=0, headerVersion=2), responseBody=FindCoordinatorResponseData(throttleTimeMs=0, errorCode=0, errorMessage='', nodeId=0, host='', port=0, coordinators=[Coordinator(key='kwgenericconsumergroup', nodeId=1, host='ewz-lori-kafka-brokers-1.ewz-lori-kafka-kafka-brokers.ewz-lori-kafka-dev.svc', port=9092, errorCode=0, errorMessage='')]))
2024-10-24T12:16:06.464Z  INFO 7 --- [nio-9343-exec-6] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Discovered group coordinator ewz-lori-kafka-brokers-1.ewz-lori-kafka-kafka-brokers.ewz-lori-kafka-dev.svc:9092 (id: 2147483646 rack: null)
2024-10-24T12:16:06.464Z DEBUG 7 --- [nio-9343-exec-6] org.apache.kafka.clients.ClientUtils     : Resolved host ewz-lori-kafka-brokers-1.ewz-lori-kafka-kafka-brokers.ewz-lori-kafka-dev.svc as 10.200.24.90
2024-10-24T12:16:06.464Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.clients.ClusterConnectionStates    : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Resolved host ewz-lori-kafka-brokers-1.ewz-lori-kafka-kafka-brokers.ewz-lori-kafka-dev.svc to addresses [ewz-lori-kafka-brokers-1.ewz-lori-kafka-kafka-brokers.ewz-lori-kafka-dev.svc/10.200.24.90]
2024-10-24T12:16:06.464Z DEBUG 7 --- [nio-9343-exec-6] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Initiating connection to node ewz-lori-kafka-brokers-1.ewz-lori-kafka-kafka-brokers.ewz-lori-kafka-dev.svc:9092 (id: 2147483646 rack: null) using address ewz-lori-kafka-brokers-1.ewz-lori-kafka-kafka-brokers.ewz-lori-kafka-dev.svc/10.200.24.90
2024-10-24T12:16:06.465Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Executing onJoinPrepare with generation -1 and memberId
2024-10-24T12:16:06.465Z  INFO 7 --- [nio-9343-exec-6] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] (Re-)joining group
2024-10-24T12:16:06.465Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Joining group with current subscription: [ewz-sap-test]
2024-10-24T12:16:06.465Z DEBUG 7 --- [icconsumergroup] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Heartbeat thread started
2024-10-24T12:16:06.465Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Sending JoinGroup (JoinGroupRequestData(groupId='kwgenericconsumergroup', sessionTimeoutMs=45000, rebalanceTimeoutMs=300000, memberId='', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='range', metadata=[0, 3, 0, 0, 0, 1, 0, 12, 101, 119, 122, 45, 115, 97, 112, 45, 116, 101, 115, 116, -1, -1, -1, -1, 0, 0, 0, 0, -1, -1, -1, -1, -1, -1]), JoinGroupRequestProtocol(name='cooperative-sticky', metadata=[0, 3, 0, 0, 0, 1, 0, 12, 101, 119, 122, 45, 115, 97, 112, 45, 116, 101, 115, 116, 0, 0, 0, 4, -1, -1, -1, -1, 0, 0, 0, 0, -1, -1, -1, -1, -1, -1])], reason='')) to coordinator ewz-lori-kafka-brokers-1.ewz-lori-kafka-kafka-brokers.ewz-lori-kafka-dev.svc:9092 (id: 2147483646 rack: null)
2024-10-24T12:16:06.465Z DEBUG 7 --- [nio-9343-exec-6] o.apache.kafka.common.network.Selector   : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node 2147483646
2024-10-24T12:16:06.465Z DEBUG 7 --- [nio-9343-exec-6] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Completed connection to node 2147483646. Fetching API versions.
2024-10-24T12:16:06.465Z DEBUG 7 --- [nio-9343-exec-6] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Initiating API versions fetch from node 2147483646.
2024-10-24T12:16:06.465Z DEBUG 7 --- [nio-9343-exec-6] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Sending API_VERSIONS request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, correlationId=5, headerVersion=2) and timeout 30000 to node 2147483646: ApiVersionsRequestData(clientSoftwareName='apache-kafka-java', clientSoftwareVersion='3.7.0')
2024-10-24T12:16:06.559Z DEBUG 7 --- [nio-9343-exec-6] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Received API_VERSIONS response from node 2147483646 for request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, correlationId=5, headerVersion=2): ApiVersionsResponseData(errorCode=0, apiKeys=[ApiVersion(apiKey=0, minVersion=0, maxVersion=10), ApiVersion(apiKey=1, minVersion=0, maxVersion=16), ApiVersion(apiKey=2, minVersion=0, maxVersion=8), ApiVersion(apiKey=3, minVersion=0, maxVersion=12), ApiVersion(apiKey=8, minVersion=0, maxVersion=9), ApiVersion(apiKey=9, minVersion=0, maxVersion=9), ApiVersion(apiKey=10, minVersion=0, maxVersion=4), ApiVersion(apiKey=11, minVersion=0, maxVersion=9), ApiVersion(apiKey=12, minVersion=0, maxVersion=4), ApiVersion(apiKey=13, minVersion=0, maxVersion=5), ApiVersion(apiKey=14, minVersion=0, maxVersion=5), ApiVersion(apiKey=15, minVersion=0, maxVersion=5), ApiVersion(apiKey=16, minVersion=0, maxVersion=4), ApiVersion(apiKey=17, minVersion=0, maxVersion=1), ApiVersion(apiKey=18, minVersion=0, maxVersion=3), ApiVersion(apiKey=19, minVersion=0, maxVersion=7), ApiVersion(apiKey=20, minVersion=0, maxVersion=6), ApiVersion(apiKey=21, minVersion=0, maxVersion=2), ApiVersion(apiKey=22, minVersion=0, maxVersion=4), ApiVersion(apiKey=23, minVersion=0, maxVersion=4), ApiVersion(apiKey=24, minVersion=0, maxVersion=4), ApiVersion(apiKey=25, minVersion=0, maxVersion=3), ApiVersion(apiKey=26, minVersion=0, maxVersion=3), ApiVersion(apiKey=27, minVersion=0, maxVersion=1), ApiVersion(apiKey=28, minVersion=0, maxVersion=3), ApiVersion(apiKey=29, minVersion=0, maxVersion=3), ApiVersion(apiKey=30, minVersion=0, maxVersion=3), ApiVersion(apiKey=31, minVersion=0, maxVersion=3), ApiVersion(apiKey=32, minVersion=0, maxVersion=4), ApiVersion(apiKey=33, minVersion=0, maxVersion=2), ApiVersion(apiKey=34, minVersion=0, maxVersion=2), ApiVersion(apiKey=35, minVersion=0, maxVersion=4), ApiVersion(apiKey=36, minVersion=0, maxVersion=2), ApiVersion(apiKey=37, minVersion=0, maxVersion=3), ApiVersion(apiKey=38, minVersion=0, maxVersion=3), ApiVersion(apiKey=39, minVersion=0, maxVersion=2), ApiVersion(apiKey=40, minVersion=0, maxVersion=2), ApiVersion(apiKey=41, minVersion=0, maxVersion=3), ApiVersion(apiKey=42, minVersion=0, maxVersion=2), ApiVersion(apiKey=43, minVersion=0, maxVersion=2), ApiVersion(apiKey=44, minVersion=0, maxVersion=1), ApiVersion(apiKey=45, minVersion=0, maxVersion=0), ApiVersion(apiKey=46, minVersion=0, maxVersion=0), ApiVersion(apiKey=47, minVersion=0, maxVersion=0), ApiVersion(apiKey=48, minVersion=0, maxVersion=1), ApiVersion(apiKey=49, minVersion=0, maxVersion=1), ApiVersion(apiKey=50, minVersion=0, maxVersion=0), ApiVersion(apiKey=51, minVersion=0, maxVersion=0), ApiVersion(apiKey=55, minVersion=0, maxVersion=1), ApiVersion(apiKey=57, minVersion=0, maxVersion=1), ApiVersion(apiKey=60, minVersion=0, maxVersion=1), ApiVersion(apiKey=61, minVersion=0, maxVersion=0), ApiVersion(apiKey=64, minVersion=0, maxVersion=0), ApiVersion(apiKey=65, minVersion=0, maxVersion=0), ApiVersion(apiKey=66, minVersion=0, maxVersion=0), ApiVersion(apiKey=68, minVersion=0, maxVersion=0), ApiVersion(apiKey=74, minVersion=0, maxVersion=0)], throttleTimeMs=0, supportedFeatures=[SupportedFeatureKey(name='metadata.version', minVersion=1, maxVersion=19)], finalizedFeaturesEpoch=7365020, finalizedFeatures=[FinalizedFeatureKey(name='metadata.version', maxVersionLevel=19, minVersionLevel=19)], zkMigrationReady=false)
2024-10-24T12:16:06.560Z DEBUG 7 --- [nio-9343-exec-6] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Node 2147483646 has finalized features epoch: 7365020, finalized features: [FinalizedFeatureKey(name='metadata.version', maxVersionLevel=19, minVersionLevel=19)], supported features: [SupportedFeatureKey(name='metadata.version', minVersion=1, maxVersion=19)], ZK migration ready: false, API versions: (Produce(0): 0 to 10 [usable: 10], Fetch(1): 0 to 16 [usable: 16], ListOffsets(2): 0 to 8 [usable: 8], Metadata(3): 0 to 12 [usable: 12], LeaderAndIsr(4): UNSUPPORTED, StopReplica(5): UNSUPPORTED, UpdateMetadata(6): UNSUPPORTED, ControlledShutdown(7): UNSUPPORTED, OffsetCommit(8): 0 to 9 [usable: 9], OffsetFetch(9): 0 to 9 [usable: 9], FindCoordinator(10): 0 to 4 [usable: 4], JoinGroup(11): 0 to 9 [usable: 9], Heartbeat(12): 0 to 4 [usable: 4], LeaveGroup(13): 0 to 5 [usable: 5], SyncGroup(14): 0 to 5 [usable: 5], DescribeGroups(15): 0 to 5 [usable: 5], ListGroups(16): 0 to 4 [usable: 4], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 3 [usable: 3], CreateTopics(19): 0 to 7 [usable: 7], DeleteTopics(20): 0 to 6 [usable: 6], DeleteRecords(21): 0 to 2 [usable: 2], InitProducerId(22): 0 to 4 [usable: 4], OffsetForLeaderEpoch(23): 0 to 4 [usable: 4], AddPartitionsToTxn(24): 0 to 4 [usable: 4], AddOffsetsToTxn(25): 0 to 3 [usable: 3], EndTxn(26): 0 to 3 [usable: 3], WriteTxnMarkers(27): 0 to 1 [usable: 1], TxnOffsetCommit(28): 0 to 3 [usable: 3], DescribeAcls(29): 0 to 3 [usable: 3], CreateAcls(30): 0 to 3 [usable: 3], DeleteAcls(31): 0 to 3 [usable: 3], DescribeConfigs(32): 0 to 4 [usable: 4], AlterConfigs(33): 0 to 2 [usable: 2], AlterReplicaLogDirs(34): 0 to 2 [usable: 2], DescribeLogDirs(35): 0 to 4 [usable: 4], SaslAuthenticate(36): 0 to 2 [usable: 2], CreatePartitions(37): 0 to 3 [usable: 3], CreateDelegationToken(38): 0 to 3 [usable: 3], RenewDelegationToken(39): 0 to 2 [usable: 2], ExpireDelegationToken(40): 0 to 2 [usable: 2], DescribeDelegationToken(41): 0 to 3 [usable: 3], DeleteGroups(42): 0 to 2 [usable: 2], ElectLeaders(43): 0 to 2 [usable: 2], IncrementalAlterConfigs(44): 0 to 1 [usable: 1], AlterPartitionReassignments(45): 0 [usable: 0], ListPartitionReassignments(46): 0 [usable: 0], OffsetDelete(47): 0 [usable: 0], DescribeClientQuotas(48): 0 to 1 [usable: 1], AlterClientQuotas(49): 0 to 1 [usable: 1], DescribeUserScramCredentials(50): 0 [usable: 0], AlterUserScramCredentials(51): 0 [usable: 0], DescribeQuorum(55): 0 to 1 [usable: 1], AlterPartition(56): UNSUPPORTED, UpdateFeatures(57): 0 to 1 [usable: 1], Envelope(58): UNSUPPORTED, DescribeCluster(60): 0 to 1 [usable: 1], DescribeProducers(61): 0 [usable: 0], UnregisterBroker(64): 0 [usable: 0], DescribeTransactions(65): 0 [usable: 0], ListTransactions(66): 0 [usable: 0], AllocateProducerIds(67): UNSUPPORTED, ConsumerGroupHeartbeat(68): 0 [usable: 0], ConsumerGroupDescribe(69): UNSUPPORTED, GetTelemetrySubscriptions(71): UNSUPPORTED, PushTelemetry(72): UNSUPPORTED, ListClientMetricsResources(74): 0 [usable: 0]).
2024-10-24T12:16:06.560Z DEBUG 7 --- [nio-9343-exec-6] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=9, clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, correlationId=4, headerVersion=2) and timeout 305000 to node 2147483646: JoinGroupRequestData(groupId='kwgenericconsumergroup', sessionTimeoutMs=45000, rebalanceTimeoutMs=300000, memberId='', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='range', metadata=[0, 3, 0, 0, 0, 1, 0, 12, 101, 119, 122, 45, 115, 97, 112, 45, 116, 101, 115, 116, -1, -1, -1, -1, 0, 0, 0, 0, -1, -1, -1, -1, -1, -1]), JoinGroupRequestProtocol(name='cooperative-sticky', metadata=[0, 3, 0, 0, 0, 1, 0, 12, 101, 119, 122, 45, 115, 97, 112, 45, 116, 101, 115, 116, 0, 0, 0, 4, -1, -1, -1, -1, 0, 0, 0, 0, -1, -1, -1, -1, -1, -1])], reason='')
2024-10-24T12:16:06.562Z DEBUG 7 --- [nio-9343-exec-6] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Received JOIN_GROUP response from node 2147483646 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=9, clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, correlationId=4, headerVersion=2): JoinGroupResponseData(throttleTimeMs=0, errorCode=79, generationId=-1, protocolType=null, protocolName=null, leader='', skipAssignment=false, memberId='KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup-2d2a4ce9-a07a-4476-bf70-d0a050178684', members=[])
2024-10-24T12:16:06.562Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] JoinGroup failed due to non-fatal error: MEMBER_ID_REQUIRED. Will set the member id as KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup-2d2a4ce9-a07a-4476-bf70-d0a050178684 and then rejoin. Sent generation was Generation{generationId=-1, memberId='', protocol='null'}
2024-10-24T12:16:06.562Z  INFO 7 --- [nio-9343-exec-6] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Request joining group due to: need to re-join with the given member-id: KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup-2d2a4ce9-a07a-4476-bf70-d0a050178684
2024-10-24T12:16:06.562Z  INFO 7 --- [nio-9343-exec-6] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] (Re-)joining group
2024-10-24T12:16:06.562Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Joining group with current subscription: [ewz-sap-test]
2024-10-24T12:16:06.562Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Sending JoinGroup (JoinGroupRequestData(groupId='kwgenericconsumergroup', sessionTimeoutMs=45000, rebalanceTimeoutMs=300000, memberId='KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup-2d2a4ce9-a07a-4476-bf70-d0a050178684', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='range', metadata=[0, 3, 0, 0, 0, 1, 0, 12, 101, 119, 122, 45, 115, 97, 112, 45, 116, 101, 115, 116, -1, -1, -1, -1, 0, 0, 0, 0, -1, -1, -1, -1, -1, -1]), JoinGroupRequestProtocol(name='cooperative-sticky', metadata=[0, 3, 0, 0, 0, 1, 0, 12, 101, 119, 122, 45, 115, 97, 112, 45, 116, 101, 115, 116, 0, 0, 0, 4, -1, -1, -1, -1, 0, 0, 0, 0, -1, -1, -1, -1, -1, -1])], reason='need to re-join with the given member-id: KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup-2d2a4ce9-a07a-4476-bf70-d0a050178684')) to coordinator ewz-lori-kafka-brokers-1.ewz-lori-kafka-kafka-brokers.ewz-lori-kafka-dev.svc:9092 (id: 2147483646 rack: null)
2024-10-24T12:16:06.562Z DEBUG 7 --- [nio-9343-exec-6] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=9, clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, correlationId=6, headerVersion=2) and timeout 305000 to node 2147483646: JoinGroupRequestData(groupId='kwgenericconsumergroup', sessionTimeoutMs=45000, rebalanceTimeoutMs=300000, memberId='KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup-2d2a4ce9-a07a-4476-bf70-d0a050178684', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='range', metadata=[0, 3, 0, 0, 0, 1, 0, 12, 101, 119, 122, 45, 115, 97, 112, 45, 116, 101, 115, 116, -1, -1, -1, -1, 0, 0, 0, 0, -1, -1, -1, -1, -1, -1]), JoinGroupRequestProtocol(name='cooperative-sticky', metadata=[0, 3, 0, 0, 0, 1, 0, 12, 101, 119, 122, 45, 115, 97, 112, 45, 116, 101, 115, 116, 0, 0, 0, 4, -1, -1, -1, -1, 0, 0, 0, 0, -1, -1, -1, -1, -1, -1])], reason='need to re-join with the given member-id: KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup-2d2a4ce9-a07a-4476-bf70-d0a050178684')
2024-10-24T12:16:08.461Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.t.i.ClientTelemetryReporter      : Initiate close of ClientTelemetryReporter
2024-10-24T12:16:08.461Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.t.i.ClientTelemetryReporter      : initiate close for client telemetry, check if terminal push required. Timeout 30000 ms.
2024-10-24T12:16:08.461Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.t.i.ClientTelemetryReporter      : Subscription not yet loaded, ignoring terminal push
2024-10-24T12:16:08.461Z DEBUG 7 --- [icconsumergroup] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Heartbeat thread has closed
2024-10-24T12:16:08.461Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Executing onLeavePrepare with generation Generation{generationId=-1, memberId='KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup-2d2a4ce9-a07a-4476-bf70-d0a050178684', protocol='null'}
2024-10-24T12:16:08.461Z  INFO 7 --- [nio-9343-exec-6] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Member KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup-2d2a4ce9-a07a-4476-bf70-d0a050178684 sending LeaveGroup request to coordinator ewz-lori-kafka-brokers-1.ewz-lori-kafka-kafka-brokers.ewz-lori-kafka-dev.svc:9092 (id: 2147483646 rack: null) due to the consumer is being closed
2024-10-24T12:16:08.461Z DEBUG 7 --- [nio-9343-exec-6] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Sending LEAVE_GROUP request with header RequestHeader(apiKey=LEAVE_GROUP, apiVersion=5, clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, correlationId=7, headerVersion=2) and timeout 30000 to node 2147483646: LeaveGroupRequestData(groupId='kwgenericconsumergroup', memberId='', members=[MemberIdentity(memberId='KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup-2d2a4ce9-a07a-4476-bf70-d0a050178684', groupInstanceId=null, reason='the consumer is being closed')])
2024-10-24T12:16:08.462Z  INFO 7 --- [nio-9343-exec-6] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Resetting generation and member id due to: consumer pro-actively leaving the group
2024-10-24T12:16:08.462Z  INFO 7 --- [nio-9343-exec-6] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Request joining group due to: consumer pro-actively leaving the group
2024-10-24T12:16:09.564Z DEBUG 7 --- [nio-9343-exec-6] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Received JOIN_GROUP response from node 2147483646 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=9, clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, correlationId=6, headerVersion=2): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=5, protocolType='consumer', protocolName='range', leader='KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup-2d2a4ce9-a07a-4476-bf70-d0a050178684', skipAssignment=false, memberId='KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup-2d2a4ce9-a07a-4476-bf70-d0a050178684', members=[JoinGroupResponseMember(memberId='KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup-2d2a4ce9-a07a-4476-bf70-d0a050178684', groupInstanceId=null, metadata=[0, 3, 0, 0, 0, 1, 0, 12, 101, 119, 122, 45, 115, 97, 112, 45, 116, 101, 115, 116, -1, -1, -1, -1, 0, 0, 0, 0, -1, -1, -1, -1, -1, -1])])
2024-10-24T12:16:09.564Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=5, protocolType='consumer', protocolName='range', leader='KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup-2d2a4ce9-a07a-4476-bf70-d0a050178684', skipAssignment=false, memberId='KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup-2d2a4ce9-a07a-4476-bf70-d0a050178684', members=[JoinGroupResponseMember(memberId='KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup-2d2a4ce9-a07a-4476-bf70-d0a050178684', groupInstanceId=null, metadata=[0, 3, 0, 0, 0, 1, 0, 12, 101, 119, 122, 45, 115, 97, 112, 45, 116, 101, 115, 116, -1, -1, -1, -1, 0, 0, 0, 0, -1, -1, -1, -1, -1, -1])])
2024-10-24T12:16:09.565Z DEBUG 7 --- [nio-9343-exec-6] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Received LEAVE_GROUP response from node 2147483646 for request with header RequestHeader(apiKey=LEAVE_GROUP, apiVersion=5, clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, correlationId=7, headerVersion=2): LeaveGroupResponseData(throttleTimeMs=0, errorCode=0, members=[MemberResponse(memberId='KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup-2d2a4ce9-a07a-4476-bf70-d0a050178684', groupInstanceId=null, errorCode=0)])
2024-10-24T12:16:09.565Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] LeaveGroup response with Generation{generationId=-1, memberId='KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup-2d2a4ce9-a07a-4476-bf70-d0a050178684', protocol='null'} returned successfully: ClientResponse(receivedTimeMs=1729772169565, latencyMs=1104, disconnected=false, timedOut=false, requestHeader=RequestHeader(apiKey=LEAVE_GROUP, apiVersion=5, clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, correlationId=7, headerVersion=2), responseBody=LeaveGroupResponseData(throttleTimeMs=0, errorCode=0, members=[MemberResponse(memberId='KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup-2d2a4ce9-a07a-4476-bf70-d0a050178684', groupInstanceId=null, errorCode=0)]))
2024-10-24T12:16:09.566Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.t.i.KafkaMetricsCollector        : removing kafka metric : MetricName [name=last-poll-seconds-ago, group=consumer-metrics, description=The number of seconds since the last poll() invocation., tags={client-id=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup}]
2024-10-24T12:16:09.566Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.t.i.KafkaMetricsCollector        : removing kafka metric : MetricName [name=time-between-poll-avg, group=consumer-metrics, description=The average delay between invocations of poll() in milliseconds., tags={client-id=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup}]
2024-10-24T12:16:09.566Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.t.i.KafkaMetricsCollector        : removing kafka metric : MetricName [name=time-between-poll-max, group=consumer-metrics, description=The max delay between invocations of poll() in milliseconds., tags={client-id=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup}]
2024-10-24T12:16:09.566Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.t.i.KafkaMetricsCollector        : removing kafka metric : MetricName [name=poll-idle-ratio-avg, group=consumer-metrics, description=The average fraction of time the consumer's poll() is idle as opposed to waiting for the user code to process records., tags={client-id=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup}]
2024-10-24T12:16:09.566Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.t.i.KafkaMetricsCollector        : removing kafka metric : MetricName [name=commit-sync-time-ns-total, group=consumer-metrics, description=The total time the consumer has spent in commitSync in nanoseconds, tags={client-id=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup}]
2024-10-24T12:16:09.566Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.t.i.KafkaMetricsCollector        : removing kafka metric : MetricName [name=committed-time-ns-total, group=consumer-metrics, description=The total time the consumer has spent in committed in nanoseconds, tags={client-id=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup}]
2024-10-24T12:16:09.566Z  INFO 7 --- [nio-9343-exec-6] o.apache.kafka.common.metrics.Metrics    : Metrics scheduler closed
2024-10-24T12:16:09.566Z  INFO 7 --- [nio-9343-exec-6] o.apache.kafka.common.metrics.Metrics    : Closing reporter org.apache.kafka.common.metrics.JmxReporter
2024-10-24T12:16:09.566Z  INFO 7 --- [nio-9343-exec-6] o.apache.kafka.common.metrics.Metrics    : Closing reporter org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter
2024-10-24T12:16:09.566Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.t.i.ClientTelemetryReporter      : Stopping ClientTelemetryReporter
2024-10-24T12:16:09.566Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.t.i.ClientTelemetryReporter      : close telemetry sender for client telemetry reporter instance
2024-10-24T12:16:09.566Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.t.i.ClientTelemetryReporter      : Setting telemetry state from SUBSCRIPTION_NEEDED to TERMINATED
2024-10-24T12:16:09.566Z  INFO 7 --- [nio-9343-exec-6] o.apache.kafka.common.metrics.Metrics    : Metrics reporters closed
2024-10-24T12:16:09.569Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.t.i.ClientTelemetryReporter      : close telemetry sender for client telemetry reporter instance
2024-10-24T12:16:09.569Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.t.i.ClientTelemetryReporter      : Ignoring subsequent close
2024-10-24T12:16:09.569Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.t.i.ClientTelemetryReporter      : Stopping ClientTelemetryReporter
2024-10-24T12:16:09.569Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.t.i.ClientTelemetryReporter      : close telemetry sender for client telemetry reporter instance
2024-10-24T12:16:09.569Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.t.i.ClientTelemetryReporter      : Ignoring subsequent close
2024-10-24T12:16:09.569Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.t.i.KafkaMetricsCollector        : removing kafka metric : MetricName [name=version, group=app-info, description=Metric indicating version, tags={client-id=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup}]
2024-10-24T12:16:09.569Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.t.i.KafkaMetricsCollector        : removing kafka metric : MetricName [name=commit-id, group=app-info, description=Metric indicating commit-id, tags={client-id=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup}]
2024-10-24T12:16:09.569Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.t.i.KafkaMetricsCollector        : removing kafka metric : MetricName [name=start-time-ms, group=app-info, description=Metric indicating start-time-ms, tags={client-id=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup}]
2024-10-24T12:16:09.569Z  INFO 7 --- [nio-9343-exec-6] o.a.kafka.common.utils.AppInfoParser     : App info kafka.consumer for KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup unregistered
2024-10-24T12:16:09.569Z DEBUG 7 --- [nio-9343-exec-6] o.a.k.c.c.internals.LegacyKafkaConsumer  : [Consumer clientId=KLAW_TMP_GROUP_CLIENTkwgenericconsumergroup, groupId=kwgenericconsumergroup] Kafka consumer has been closed
2024-10-24T12:16:09.570Z DEBUG 7 --- [nio-9343-exec-6] o.s.w.s.m.m.a.HttpEntityMethodProcessor  : Using 'application/json', given [application/json] and supported [application/json]
2024-10-24T12:16:09.570Z DEBUG 7 --- [nio-9343-exec-6] o.s.w.s.m.m.a.HttpEntityMethodProcessor  : Writing [{}]
2024-10-24T12:16:09.571Z DEBUG 7 --- [nio-9343-exec-6] o.s.web.servlet.DispatcherServlet        : Completed 200 OK
muralibasani commented 4 weeks ago

@yvessavoy can you check authorizer logs if that's an acl issue ? and do you know if klaw host has read access through this consumer group kwgenericconsumergroup , if not can you add and acl for this ? kwgenericconsumergroup consumer group is configured in cluster api by default

aindriu-aiven commented 3 weeks ago

Hey @yvessavoy we just wanted to check in again to see if you managed to resolve this, or if you still need some further assistance?