jakobhviid / Dashboard-Interface-Docker

MIT License
0 stars 0 forks source link

Kafka Monitoring #18

Open Omvk97 opened 4 years ago

Omvk97 commented 4 years ago

Kafka has JMX Metrics. Dashboard-Interface should have an interface which displays the most important information such as topics, who is writing to these topics.

With these metrics it could be nice if the interface could detect abnormal writes and alert the user

dstoft commented 3 years ago

After testing this with JConsole and Jolokia (HTTP version of JConsole), then I'm not sure that you can fetch information such as topics and who is writing to what topic with JMX Metrics. However, there are some other metrics that could be interesting. If we want the metrics to be available to C# developed system, then we probably have to start a Jolokia agent that listens to specific JVMs and makes the data available with a REST API. I have not found any other feasible solution yet, as JMX (and its underlying RMI technology) is clouded with a lot of old information on the internet.

I'm left wondering what value this would bring compared to the big implementation task it is. However, I do agree that information regarding what topics there are and who is writing what is important, but I'm not sure JMX is the way to go for this.

The following documentation shows what JMX metrics are available: https://docs.confluent.io/platform/current/kafka/monitoring.html https://kafka.apache.org/documentation/#monitoring