omarsmak / kafka-consumer-lag-monitoring

Client tool that exports the consumer lag of Kafka consumer groups to Prometheus or your terminal
MIT License
50 stars 13 forks source link

Looking at building a Kubernetes operator for the monitoring #26

Open omarsmak opened 3 years ago

omarsmak commented 3 years ago

To simplify deploying this utility as Kubernetes operator could be good idea, for example a custom CRD that supports all the Kafka Properties without the user need to add any kafka properties files. Just define all these in the CRD and run the operator. Steps to implement this:

  1. Auto generate the CRD blueprint from the Kafka configurations (this is not really needed but for the schema validation). We can ignore this at first iteration.
  2. In the controller, parse the CRD properties and convert it to Map of properties.
  3. Run the monitoring client using the Map.

In order to ensure that works with Standalone mode and Docker mode. We will need to convert this project into multi module project that has one sub project for the client and other sub project for the K8s operator.