Metrics visualization

[Haarolean]

Store and visualize metrics graps, for, like, consumer lag

Feedback from here:

Have a graph that show consumers lag over time like metricbeat/Kibana dashboard See a graph of the publish message count per topic over time (last hour, last day, last 10 day, etc).

UPD: 0) Research: determine set of metrics for visualisation 1) Consider an easy&fast solution: store some data as cache for consumer lag in ram and display graphs in UI 2) visual identification of lagging consumers? Red borders? 3) table inline small graphs

[Haarolean]

Also #206 #3047

[Mgrdich]

Results for the Library

Basically the decision was based

• On the features
• Documentation
• Bundle size
• Ease to use
• Performance
• Vanilla Javascript library that they were built on
• Customizability

We have two strong options.

• ReCharts Website , Github
• react-chartjs-2 Website , Github

1. ReCharts

Is a composable charting library built on React components.

Pros

• Built on top of SVG elements with a lightweight dependency on D3 submodules.
• Easy to use
• Very Very Good Documentation on their website
• StoryBook examples
• mostly written with Typescript , but has a @types definition.
• Easily customizable.

Cons

• a bit larger bundle size than react-chartjs-2 , but not with a huge margin

2. react-chartjs-2

Pros

• Built on top of Chart.js.
• Easy to use
• mostly written with Typescript , but has a @types definition.
• better bundle size than Recharts

Cons

• Not easily customizable.
• Not as much examples as Recharts.

Results

My Personal choice would be ReCharts, since the pros are more than the cons , and it has a very good examples.

Tips

Since Charts Components are heavy components , i would recommend to memoize those components with React.memo, and make them on their one , where they should change only when the props of them change not , when some node in their parent trigger a re-render by calling a setState.

[sherifkayad]

@Haarolean just for my understanding, what would be the backend of this feature? I mean would the Kafka UI then store such metrics by itself or generate them on the fly each time from Kafka? How about querying Prometheus for those and just visualize them like what's today possible in Grafana using the Kafka Exporter? I think querying prometheus would probably give some more accurate data and a good graph view .. what do you think?

[Haarolean]

@sherifkayad this is TBD yet. Probly exporting metrics into prometheus, might be a lil bit of storing short-term some metrics in memory.

[sherifkayad]

@Haarolean got ya. I just wanted to say if something like the Kafka Exporter (https://github.com/danielqsj/kafka_exporter) is already in place, then you wouldn't probably need to worry about lots of metrics e.g. lag metrics or throughput metrics. In that case, probably the Kafka UI can just read the metrics from Prometheus and that's it. Not sure if that makes sense to you or if there could be some metrics that the UI needs and those aren't available by the exporter.

[Haarolean]

@sherifkayad thanks for suggesting kafka-exporter. We might use that as a setup example for this feature.

[Haarolean]

This repo is not maintained (#4255) Please follow this issue in our fork for the updates: https://github.com/kafbat/kafka-ui/issues/233