search

yahoo / CMAK

CMAK is a tool for managing Apache Kafka clusters
Apache License 2.0
11.81k stars 2.5k forks source link

"Error reading array of size 361, only 186 bytes available" on Kafka 2.1.1 with Kafka Manager 1.3.3.21 #619

Open jthompson6 opened 5 years ago

jthompson6 commented 5 years ago

Our logs are full of messages like these, 


org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'members': Error reading array of size 361, only 186 bytes available
    at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:76) ~[org.apache.kafka.kafka-clients-1.1.0.jar:na]
    at kafka.manager.utils.one10.GroupMetadataManager$.readGroupMessageValue(GroupMetadataManager.scala:608) ~[kafka-manager.kafka-manager-1.3.3.21-sans-externalized.jar:na]
    at kafka.manager.actor.cluster.KafkaManagedOffsetCache$$anonfun$run$4.apply(KafkaStateActor.scala:378) [kafka-manager.kafka-manager-1.3.3.21-sans-externalized.jar:na]
    at kafka.manager.actor.cluster.KafkaManagedOffsetCache$$anonfun$run$4.apply(KafkaStateActor.scala:326) [kafka-manager.kafka-manager-1.3.3.21-sans-externalized.jar:na]
    at scala.util.Success.foreach(Try.scala:236) [org.scala-lang.scala-library-2.11.12.jar:na]
    at kafka.manager.actor.cluster.KafkaManagedOffsetCache.run(KafkaStateActor.scala:326) [kafka-manager.kafka-manager-1.3.3.21-sans-externalized.jar:na]```

and then the Kafka Manager API endpoints start failing. Restarting seems to temporarily fix the problem, but we are seeing it die at least once a day since we upgraded from Kafka 1.1 to 2.1.1.
jthompson6 commented 5 years ago

The version that we are running already has this fix: https://github.com/yahoo/kafka-manager/pull/593/files

hagaibarel commented 5 years ago

Same here, using kafka-mangaer 1.3.3.23