Monitoring Solution for cp-kafka and cp-zookeeper

confluentinc / cp-helm-charts

The Confluent Platform Helm charts enable you to deploy Confluent Platform services on Kubernetes for development, test, and proof of concept environments.

https://cnfl.io/getting-started-kafka-kubernetes

Apache License 2.0

788 stars 846 forks source link

Monitoring Solution for cp-kafka and cp-zookeeper #16

Closed qshao-pivotal closed 6 years ago

qshao-pivotal commented 6 years ago

Problem Statement

As an operator, after provisioning a kafka cluster, I need to know what is happening to Kafka(metrics) and also be able to view after fact(logs).

Potential Solution

Metrics
1. Prometheus JMX exporter + external Prometheus - all components exist, need a supported docker image for JMX exporter + definitions/transformation of all JMX metrics of interest.
2. Prometheus Kafka exporter + external Prometheus - OSS vs Confluent version of Kafka exporter?
3. Maybe Confluent Control Center?
Logs
1. Filebeat vs Logstash vs Fluentd, extract, transform and load to Elasticsearch
2. Confluent log extractor/transformer?

@mikkin

lokesh-shekar commented 6 years ago

Most of what we need to implement depends on what customers need :)

Prometheus JMX exporter seems like a good first step since all metrics are available and this is what we also use in Confluent Cloud currently.
Confluent Control Center is a great option for customers who need a UI. However Confluent Control Center has a dependency on Kafka.
Filebeat or Logstash or Fluentd, depends on customer. Confluent Cloud uses Filebeat currently.
We do not currently have log extractor/transformer. However, it is on the roadmap.

qshao-pivotal commented 6 years ago

Waiting for #33