"Error reading field 'topics': Error reading array of size 740943, only 210 bytes available" on Kafka 2.4.0 with Kafka Manager 3.0.0.2 and 3.0.0.4

[yhk6190]

After upgrading Kafka manager 2.0.0.2 to 3.0.0.2(and also 3.0.0.4), WARN messages occur in CMAK application log so frequently. In my cluster there are many topics(235+) and partitions(about 2500+), it might be bring WARN messages out or something other reasons.

Logs are full of messages like these,

2020-03-09 17:27:43,023 - [WARN] - from kafka.manager.actor.cluster.KafkaManagedOffsetCache in KafkaManagedOffsetCache
Failed to process a message from offset topic on cluster MYCLUSTER!
org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'topics': Error reading array of size 740943, only 210 bytes available
        at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:110)
        at org.apache.kafka.clients.consumer.internals.ConsumerProtocol.deserializeSubscriptionV1(ConsumerProtocol.java:190)
        at org.apache.kafka.clients.consumer.internals.ConsumerProtocol.deserializeSubscription(ConsumerProtocol.java:223)
        at kafka.manager.utils.two40.GroupMetadataManager$.$anonfun$readGroupMessageValue$1(GroupMetadataManager.scala:769)
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
        at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
        at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
        at scala.collection.TraversableLike.map(TraversableLike.scala:238)
        at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
        at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
        at kafka.manager.utils.two40.GroupMetadataManager$.readGroupMessageValue(GroupMetadataManager.scala:759)
        at kafka.manager.actor.cluster.KafkaManagedOffsetCache.$anonfun$run$3(KafkaStateActor.scala:391)
        at kafka.manager.actor.cluster.KafkaManagedOffsetCache.$anonfun$run$3$adapted(KafkaStateActor.scala:335)
        at scala.util.Success.foreach(Try.scala:253)
        at kafka.manager.actor.cluster.KafkaManagedOffsetCache.run(KafkaStateActor.scala:335)
        at java.base/java.lang.Thread.run(Thread.java:834)

[heaje]

@yhk6190 - I'm seeing this same thing after upgrading from 2.0.0.2 to 3.0.0.4. Were you able to find any resolution to this issue? In my case, I'm running Kafka 2.3.1.

[alexandreklayton]

I have the same problem.

• cmak-3.0.0.5
• kafka_2.10-0.10.1.0
• Operating System: CentOS Linux 7 (Core) CPE OS Name: cpe:/o:centos:centos:7 Kernel: Linux 3.10.0-957.1.3.el7.x86_64 Architecture: x86-64
• ERRO: 2020-08-04 15:18:56,195 - [WARN] k.m.a.c.KafkaManagedOffsetCache - Failed to process a message from offset topic on cluster IIDR-HLG! org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'topics': Error reading array of size 1927284, only 536 bytes available at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:110) at org.apache.kafka.clients.consumer.internals.ConsumerProtocol.deserializeSubscriptionV1(ConsumerProtocol.java:190) at org.apache.kafka.clients.consumer.internals.ConsumerProtocol.deserializeSubscription(ConsumerProtocol.java:223) at kafka.manager.utils.two40.GroupMetadataManager$.$anonfun$readGroupMessageValue$1(GroupMetadataManager.scala:769) at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238) at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198) at scala.collection.TraversableLike.map(TraversableLike.scala:238) at scala.collection.TraversableLike.map$(TraversableLike.scala:231)

[DevarajKR]

Any resolution for this ? i am also getting same error with kafka-manager-1.3.3.22 version

Failed to process a message from offset topic on cluster kafka2-dev! org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'members': Error reading array of size 383, only 161 bytes available at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:76) ~[org.apache.kafka.kafka-clients-1.1.0.jar:na] at kafka.manager.utils.one10.GroupMetadataManager$.readGroupMessageValue(GroupMetadataManager.scala:608) ~[kafka-manager.kafka-manager-1.3.3.22-sans-externalized.jar:na]

[bachmanity1]

This is probably caused by log.message.format.version incompatibility between Kafka cluster & CMAK.