search

linkedin / cruise-control

Cruise-control is the first of its kind to fully automate the dynamic workload rebalance and self-healing of a Kafka cluster. It provides great value to Kafka users by simplifying the operation of Kafka clusters.
https://github.com/linkedin/cruise-control/tags
BSD 2-Clause "Simplified" License
2.74k stars 587 forks source link

CruiseControlMetric type 55 does not exist. #574

Closed saritago closed 5 years ago

saritago commented 5 years ago

Seeing below error in the cc logs. I deleted all the content of topic __CruiseControlMetrics and tried to create a new CC instance and yet i see same error. 

[2019-02-26 08:35:33,610] ERROR Received exception. (com.linkedin.kafka.cruisecontrol.monitor.sampling.MetricFetcher)
org.apache.kafka.common.errors.SerializationException: Error deserializing key/value for partition __CruiseControlMetrics-2 at offset 359418308. If needed, please seek past the record to continue consumption.
Caused by: java.lang.RuntimeException: Error occurred when deserialize Cruise Control metrics.
    at com.linkedin.kafka.cruisecontrol.metricsreporter.metric.MetricSerde.deserialize(MetricSerde.java:47)
    at com.linkedin.kafka.cruisecontrol.metricsreporter.metric.MetricSerde.deserialize(MetricSerde.java:14)
    at org.apache.kafka.common.serialization.ExtendedDeserializer$Wrapper.deserialize(ExtendedDeserializer.java:65)
    at org.apache.kafka.common.serialization.ExtendedDeserializer$Wrapper.deserialize(ExtendedDeserializer.java:55)
    at org.apache.kafka.clients.consumer.internals.Fetcher.parseRecord(Fetcher.java:918)
    at org.apache.kafka.clients.consumer.internals.Fetcher.access$2600(Fetcher.java:93)
    at org.apache.kafka.clients.consumer.internals.Fetcher$PartitionRecords.fetchRecords(Fetcher.java:1095)
    at org.apache.kafka.clients.consumer.internals.Fetcher$PartitionRecords.access$1200(Fetcher.java:944)
    at org.apache.kafka.clients.consumer.internals.Fetcher.fetchRecords(Fetcher.java:567)
    at org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:528)
    at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1086)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043)
    at com.linkedin.kafka.cruisecontrol.monitor.sampling.CruiseControlMetricsReporterSampler.getSamples(CruiseControlMetricsReporterSampler.java:98)
    at com.linkedin.kafka.cruisecontrol.monitor.sampling.SamplingFetcher.fetchSamples(SamplingFetcher.java:105)
    at com.linkedin.kafka.cruisecontrol.monitor.sampling.SamplingFetcher.fetchMetricsForAssignedPartitions(SamplingFetcher.java:85)
    at com.linkedin.kafka.cruisecontrol.monitor.sampling.MetricFetcher.call(MetricFetcher.java:24)
    at com.linkedin.kafka.cruisecontrol.monitor.sampling.MetricFetcher.call(MetricFetcher.java:16)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: CruiseControlMetric type 55 does not exist.
    at com.linkedin.kafka.cruisecontrol.metricsreporter.metric.RawMetricType.forId(RawMetricType.java:112)
    at com.linkedin.kafka.cruisecontrol.metricsreporter.metric.BrokerMetric.fromBuffer(BrokerMetric.java:49)
    at com.linkedin.kafka.cruisecontrol.metricsreporter.metric.MetricSerde.fromBytes(MetricSerde.java:30)
    at com.linkedin.kafka.cruisecontrol.metricsreporter.metric.MetricSerde.deserialize(MetricSerde.java:45)
    ... 20 more
efeg commented 5 years ago

@saritago What is the version of your (1) CC instance and (2) CC metrics reporter used by Kafka brokers? I suspect that you are using an old CC instance version; hence, metrics reporter generates new metrics, but the CC instance does not have the min. required version to process the reported metrics.

saritago commented 5 years ago

Yes, the cruise control client version I was using was of older version. I updated it and it worked fine. But I keep hitting one or the other warnings/ issues which I post either in gitter or under issues..

On Thu, 28 Feb, 2019, 11:24 PM Adem Efe Gencer, notifications@github.com wrote:

@saritago https://github.com/saritago What is the version of your (1) CC instance and (2) CC metrics reporter used by Kafka brokers? I suspect that you are using an old CC instance version; hence, metrics reporter generates new metrics, but the CC instance does not have the min. required version to process the reported metrics.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/linkedin/cruise-control/issues/574#issuecomment-468371511, or mute the thread https://github.com/notifications/unsubscribe-auth/AoTqAi2lpgNv5EOGP8qfV62h31k4qnPEks5vSBfbgaJpZM4bRuSg .

efeg commented 5 years ago

@saritago glad to hear that the issue is resolved! Please continue letting us know if you hit further issues.