[Feature request] Collect kafka metrics using JMX

tchiotludo / akhq

Kafka GUI for Apache Kafka to manage topics, topics data, consumers group, schema registry, connect and more...

https://akhq.io/

Apache License 2.0

3.36k stars 652 forks source link

[Feature request] Collect kafka metrics using JMX #808

Open gergof opened 3 years ago

gergof commented 3 years ago

It would be a really great addition to be able to collect broker metrics using the JMX port of the broker. Per broker and per topic metrics like throughput would be really useful in certain scenarios.

tchiotludo commented 3 years ago

I've already think about this one, not really how to handle it the best way.

Most of saas people don't have JMX port available, but have some http endpoint with metrics.
having snapshot of metrics is cool, but people want history, so need to store the data. Is this the goal to akhq ? or maybe akhq can rely on third party (prometheus, ...) to just display data ?

Complex subject ;)

gergof commented 3 years ago

With SaaS you mostly have monitoring from you provider, but AWS for example lets you access the JMX port and in some other cases you can access the jmx port via Jolokia. But I agree, that it won't work for everybody.

As for the data retention, in my opinion it's ok to use a third party (like prometheus, influxDB or something like that), but it would also be great to have a short term history in-memory, so AKHQ could be used standalone (like netdata does it).

tchiotludo commented 3 years ago

I don't know that aws allow access to jmx. Maybe you describe what metrics you would like to be displayed and where ?

gergof commented 2 years ago

There's another tool called conduktor and I really like how the metrics are implemented there (no history only a snapshot for the current values). For every broker and topic there's a throughput (incoming and outgoing bytes/sec) metric. For me that would be sufficient. But there are a lot more listed here that could be useful. For example the consumer/producer metrics (bytes-consumed-rate, record-send-rate, record-size-avg, etc) look really useful as well.