search

linkedin / cruise-control

Cruise-control is the first of its kind to fully automate the dynamic workload rebalance and self-healing of a Kafka cluster. It provides great value to Kafka users by simplifying the operation of Kafka clusters.
https://github.com/linkedin/cruise-control/tags
BSD 2-Clause "Simplified" License
2.74k stars 585 forks source link

Suppress warning logs(Skip generating) in CC #2162

Closed lenin-joseph closed 3 months ago

lenin-joseph commented 3 months ago

Hi,

Whenever we restart the Kafka pods, we see the below warning in the CC logs. The warning message seems misleading that "metrics are missing". Can this be suppressed? Does cruise control have any heartbeat mechanism to check the health of the Kafka broker before trying to get the metrics from that broker?

{"version": "1.2.0", "timestamp": "2024-05-29T13:14:36.252+00:00", "severity": "warning", "service_id": "kafka-cluster", "message": "Skip generating metric sample for broker 0 because all broker metrics are missing.", "metadata": {"container_name": "cruise-control", "pod_name": "cluster-a-cruise-control-6979466c66-2ltnk"}, "extra_data": {"file": "SamplingUtils.java", "line": "295", "cluster_name": "cluster-a"}}

strimzi: 0.36.1 Kafka- 3.5.1 cruise-control-metrics-reporter-2.5.123.jar

Regards, Lenin

mhratson commented 3 months ago

Does cruise control have any heartbeat mechanism to check the health of the Kafka broker before trying to get the metrics from that broker?

CC doesn't "get" metrics from the broker. Metrics is produced by the cruise-control-metrics-reporter into __CruiseControlMetrics topic and later consumed by CC.

More here: WARN Skip generating metric sample for broker because all broker metrics are missing

mhratson commented 3 months ago

As for suppressing metrics you can either increase log level to skip WARNINGs or filter out those particular messages in your logging pipeline. There seem to be no way to suppress those warnings specifically at the moment.