search

linkedin / cruise-control

Cruise-control is the first of its kind to fully automate the dynamic workload rebalance and self-healing of a Kafka cluster. It provides great value to Kafka users by simplifying the operation of Kafka clusters.
https://github.com/linkedin/cruise-control/tags
BSD 2-Clause "Simplified" License
2.76k stars 589 forks source link

__CruiseControlMetrics topic is not populating any data #1978

Open rivlinpereira opened 1 year ago

rivlinpereira commented 1 year ago

Hi,

I am running MSK 2.6.2 and cruise control is working fine for me, but this topic __CruiseControlMetrics which i also had to manually create on the cluster shows size 0, I dont see any obvious errors in the logs. Any idea what the issue might be? I am running cruise control version 2.5.112

I only see __KafkaCruiseControlPartitionMetricSamples or __KafkaCruiseControlPartitionMetricSamples topics getting populated and not __CruiseControlMetrics

In cruise control logs I see things like these, no errors - 

[2023-03-07 23:58:33,701] INFO Kicking off metric sampling for time range [1678233393701, 1678233513701], duration 120000 ms with timeout 120000 ms. (com.linkedin.kafka.cruisecontrol.monitor.sampling.MetricFetcherManager)
[2023-03-07 23:58:33,823] INFO Added 2640 metric values. Skipped 0 invalid query results. (com.linkedin.kafka.cruisecontrol.monitor.sampling.prometheus.PrometheusMetricSampler)
[2023-03-07 23:58:33,824] INFO Generated 189 partition metric samples and 4 broker metric samples for timestamp 1678233513000. (com.linkedin.kafka.cruisecontrol.monitor.sampling.CruiseControlMetricsProcessor)
[2023-03-07 23:58:33,824] INFO Collected 189 partition metric samples for 189 partitions. Total partition assigned: 189. (com.linkedin.kafka.cruisecontrol.monitor.sampling.SamplingFetcher)
[2023-03-07 23:58:33,825] INFO Collected 4 broker metric samples for 4 brokers. (com.linkedin.kafka.cruisecontrol.monitor.sampling.SamplingFetcher)
[2023-03-07 23:58:33,829] INFO Finished sampling in 128 ms. (com.linkedin.kafka.cruisecontrol.monitor.sampling.MetricFetcherManager)
wonkyto commented 7 months ago

We've noticed the same behaviour running on MSK Kafka v3.6.0. Cruise Control appears to be running normally (or at least to my inexperienced use), but we never see any content being written into the topic __CruiseControlMetrics.