PrometheusMetricSampler query window time range

[rkferreira]

Hi,

During the setup of Cruise Control using "PrometheusMetricSampler" I see on issue collecting data from AWS MSK cluster.

CC VERSION="2.5.42"

DefaultPrometheusQuerySupplier.java proposes "BROKER_CPU_UTIL" query using time range window of 1 minute, but MSK reports CPU metrics as of 5 minutes window.

The result is you always get discarded metrics due to missing BROKER_CPU_UTIL metric:

#SamplingUtils.java
    if (brokerLoad == null || !brokerLoad.brokerMetricAvailable(BROKER_CPU_UTIL)) {
      // Broker load or its BROKER_CPU_UTIL metric is not available.
      LOG.debug("{}partition {} because {} metric for broker {} is unavailable.", SKIP_BUILDING_SAMPLE_PREFIX,
                tpDotNotHandled, BROKER_CPU_UTIL, leaderId);
      return true;

My current fix:

--- cruise-control/src/main/java/com/linkedin/kafka/cruisecontrol/monitor/sampling/prometheus/DefaultPrometheusQuerySupplier.java   2021-10-04 15:27:09.000000000 -0300
+++ cruise-control/src/main/java/com/linkedin/kafka/cruisecontrol/monitor/sampling/prometheus/DefaultPrometheusQuerySupplier_aws.java   2021-10-19 22:18:19.000000000 -0300
@@ -23,7 +23,7 @@
     static {
         // broker metrics
         TYPE_TO_QUERY.put(BROKER_CPU_UTIL,
-            "1 - avg by (instance) (irate(node_cpu_seconds_total{mode=\"idle\"}[1m]))");
+            "1 - avg by (instance) (irate(node_cpu_seconds_total{mode=\"idle\"}[5m]))");
         TYPE_TO_QUERY.put(ALL_TOPIC_BYTES_IN,
             "kafka_server_BrokerTopicMetrics_OneMinuteRate{name=\"BytesInPerSec\",topic=\"\"}");
         TYPE_TO_QUERY.put(ALL_TOPIC_BYTES_OUT,

An improvement would be a configurable prometheus window.

Thanks, Rodrigo Kellermann Ferreira

[efeg]

Hi @rkferreira Thanks for reporting this issue! Would you like to contribute your fix and/or the proposed improvement?

[mohitpali]

The reason why the metrics are getting discarded is because your scraping interval is > 30 seconds and 1 - avg by (instance) (irate(node_cpu_seconds_total{mode=\"idle\"}[1m])) is an Prometheus iRate query requires at least 2 data points to calculate the rate.

I however agree that there should be a provision to change the duration based on your scraping interval.