search

linkedin / cruise-control

Cruise-control is the first of its kind to fully automate the dynamic workload rebalance and self-healing of a Kafka cluster. It provides great value to Kafka users by simplifying the operation of Kafka clusters.
https://github.com/linkedin/cruise-control/tags
BSD 2-Clause "Simplified" License
2.74k stars 587 forks source link

Authentication of CC on AMP(Amazon Managed Service for Prometheus) #1946

Open drozd0 opened 1 year ago

drozd0 commented 1 year ago

We want to integrate cruise control (CC) into our Kafka platform. We are using AWS to run our Kafka cluster and want to use AWS Prometheus for storing metrics.

Could you please give an example of how to integrate CC and AWS Prometheus, I didn't find any example. If I am using AWS Prometheus host:port settings without any credentials then I am getting: cruise-control | [2022-11-29 11:56:26,558] ERROR Error when attempting to query Prometheus metrics java.io.IOException: Received non-success response code on Prometheus API HTTP call, response code = 403, response body = {"message":"Missing Authentication Token"}

MSK documentation: https://docs.aws.amazon.com/msk/latest/developerguide/cruise-control.html They use Prometheus without any authentication in the example above: metric.sampler.class=com.linkedin.kafka.cruisecontrol.monitor.sampling.prometheus.PrometheusMetricSampler prometheus.server.endpoint=1.2.3.4:9090 # Replace with your Prometheus IP and port Could you please help?

Thank you in advance!

diljeetsingh-nr commented 1 year ago

This should help, https://docs.aws.amazon.com/msk/latest/developerguide/open-monitoring.html

drozd0 commented 1 year ago

@diljeetsingh-nr yeah, I saw it. We have an issue with the authentication of CC for AMP when CC tries to fetch metrics from Prometheus. Do you use CC with AWS Prometheus? I followed this instruction: https://docs.aws.amazon.com/msk/latest/developerguide/cruise-control.html but there is no configuration for authentication...