search

linkedin / cruise-control

Cruise-control is the first of its kind to fully automate the dynamic workload rebalance and self-healing of a Kafka cluster. It provides great value to Kafka users by simplifying the operation of Kafka clusters.
https://github.com/linkedin/cruise-control/tags
BSD 2-Clause "Simplified" License
2.74k stars 587 forks source link

INFO Skipping proposal precomputing because load monitor does not have enough snapshots. (com.linkedin.kafka.cruisecontrol.analyzer.GoalOptimizer) #1602

Open amal786 opened 3 years ago

amal786 commented 3 years ago

i am unable to open a UI CC and none of the command is successful for me other that /state command

USER TASK ID CLIENT ADDRESS START TIME STATUS REQUEST URL
16b57bd6-1247-4ecf-a3da-a09061a27401 192.168.1.3 2021-06-30T14:01:23Z Completed GET /kafkacruisecontrol/kafka_cluster_state
ff4b9be8-23d2-4f59-adc4-011989827b81 192.168.1.3 2021-06-30T14:11:24Z CompletedWithError GET /kafkacruisecontrol/proposals?allow_capacity_estimation=False
26b20beb-5e50-4300-bd48-23094f662861 192.168.1.3 2021-06-30T14:15:01Z CompletedWithError GET /kafkacruisecontrol/proposals
e6a306f7-5ab0-43ec-b955-747f3427f63e 192.168.1.3 2021-06-30T14:19:12Z CompletedWithError GET /kafkacruisecontrol/load
a25abd53-99c4-47b1-a1f1-55bac1c91179 192.168.1.3 2021-06-30T14:07:46Z Completed GET /kafkacruisecontrol/state
32108ef4-d754-45e2-8d76-abca4f1aad05 192.168.1.3 2021-06-30T14:18:28Z Completed GET /kafkacruisecontrol/state

efeg commented 3 years ago

"... load monitor does not have enough snapshots" -> It is likely that your Cruise Control instance fails to collect metrics. There should be some relevant messages on (1) Cruise control logs and/or (2) Kafka logs (I assume you are using Cruise Control Metrics reporter) -- can you please share those?

amal786 commented 3 years ago

broker1.zip broker2.zip kafkacruisecontrol.log.2021-06-30-20.zip

amal786 commented 3 years ago

i am not able to gather any metrics from the brokers :

able to gather this

[root@node3 cruise-control]# cccli -a 192.168.1.8:9090 kafka_cluster_state Starting long-running poll of http://192.168.1.8:9090/kafkacruisecontrol/kafka_cluster_state? Summary: The cluster has 2 brokers, 145 replicas, 73 leaders, and 7 topics with avg replication factor: 1.99. [Leaders/Replicas per broker] avg: 36.50/72.50 max: 37/73 std: 0.50/0.50

Brokers: BROKER LEADER(S) REPLICAS OUT-OF-SYNC OFFLINE IS_CONTROLLER 1 37 73 0 0 true 2 36 72 0 0 false

LogDirs of brokers with replicas: BROKER ONLINE-LOGDIRS OFFLINE-LOGDIRS 1 [/data/kafka-logs] [] 2 [/home/data/kafka-logs] []

Under Replicated, Offline, and Under MinIsr Partitions: TOPIC PARTITION LEADER REPLICAS IN-SYNC OUT-OF-SYNC OFFLINE Offline Partitions: Partitions with Offline Replicas: Under Replicated Partitions: Under MinIsr Partitions:

efeg commented 3 years ago

Your broker logs show no indication of being configured properly to work with CruiseControlMetrics reporter. Please follow the steps documented in the Quick Start. In particular, please make sure that you follow the instructions that starts with This step is required if CruiseControlMetricsReporter is used for metrics collection...

amal786 commented 3 years ago

@efeg : is it necessary that the broker should be on the same java version as cruise control as the section of the quick start that says "Modify Kafka server configuration to set metric.reporters to com.linkedin.kafka.cruisecontrol.metricsreporter.CruiseControlMetricsReporter. For Apache Kafka, server properties are located at ./config/server.properties." gives exception start :[2021-07-01 21:33:12,936] ERROR [KafkaServer id=1] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) java.lang.UnsupportedClassVersionError: com/linkedin/kafka/cruisecontrol/metricsreporter/CruiseControlMetricsReporter has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0

amal786 commented 3 years ago

@efeg: any operation that i am trying to do except to get the state from cruise control fails whenever i try to do any operation on the cruise control i get the below error here i was trying to remove a broker using : curl -X POST -c /tmp/mycookie-jar.txt "http://192.168.1.5:9090/kafkacruisecontrol/remove_broker?brokerid=2&dryrun=true" Error processing POST request '/remove_broker' due to: 'com.linkedin.kafka.cruisecontrol.exception.KafkaCruiseControlException: com.linkedin.cruisecontrol.exception.NotEnoughValidWindowsException: There is no window available in range

efeg commented 3 years ago

@efeg : is it necessary that the broker should be on the same java version as cruise control as the section of the quick start that says "Modify Kafka server configuration to set metric.reporters to com.linkedin.kafka.cruisecontrol.metricsreporter.CruiseControlMetricsReporter. For Apache Kafka, server properties are located at ./config/server.properties." gives exception start :[2021-07-01 21:33:12,936] ERROR [KafkaServer id=1] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) java.lang.UnsupportedClassVersionError: com/linkedin/kafka/cruisecontrol/metricsreporter/CruiseControlMetricsReporter has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0

That is correct. To use the latest version of CruiseControlMetricsReporter, brokers should be on Java 11 or above. Alternatively, you can either:

  1. Retrieve and use the generated CruiseControlMetricsReporter jar from a CC version compiled with an earlier Java version (e.g. 2.5.20 or before, 2.4.29 or before, 2.0.136 or before) in broker, then run the latest version of CC as usual -- see https://github.com/linkedin/cruise-control/releases. Note that an old version of CruiseControlMetricsReporter would always be compatible with the latest version of CC -- i.e. you can retrieve CruiseControlMetricsReporter jar from an old version of CC, copy it to broker-side, and it will work fine with later versions of CC.
  2. Do not use CruiseControlMetricsReporter and instead use PrometheusMetricSampler with your own Prometheus server.

Error processing POST request '/remove_broker' due to: 'com.linkedin.kafka.cruisecontrol.exception.KafkaCruiseControlException: com.linkedin.cruisecontrol.exception.NotEnoughValidWindowsException: There is no window available in range

This is expected because in your current setup there is no metric collection, which CC can use to create windows.

Hope it helps!

TheIdhem commented 3 years ago

@efeg : is it necessary that the broker should be on the same java version as cruise control as the section of the quick start that says "Modify Kafka server configuration to set metric.reporters to com.linkedin.kafka.cruisecontrol.metricsreporter.CruiseControlMetricsReporter. For Apache Kafka, server properties are located at ./config/server.properties." gives exception start :[2021-07-01 21:33:12,936] ERROR [KafkaServer id=1] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) java.lang.UnsupportedClassVersionError: com/linkedin/kafka/cruisecontrol/metricsreporter/CruiseControlMetricsReporter has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0

i'm trying to solve this problem and in my case i update my java version to java 11 and it solved if you are using centos you can use this link: https://phoenixnap.com/kb/install-java-on-centos