Expose Neo4j Metrics into Kafka Topics

[conker84]

Just for discussion: add a feature that exposes neo4j metrics into kafka topics

[moxious]

(Jan 22 Discussion)

• Who/where is Kafka used for Monitoring? Use cases & customers
• Kafka can be used as ingress to ELK stack -- is it an equivalent 3rd option to Prometheus/grafana?
• Probably plugin only, maybe not connect agent because of polling which would be needed
• How to publish to kafka and specify which metrics go to which topics? Configuration parameters to describe how to map the metric language to kafka topics. E.g. 5 "metric families" with 60 metrics total, is this 5 topics, 60, or 1 with lots of different keys?
• General problem -- mapping hierarchy of metrics on to kafka topics -- put it all in one topic and let the user route it elsewhere as needed? Maybe as a first step.

[conker84]

Confluent offers a Prometheus Sink: https://docs.confluent.io/kafka-connect-prometheus-metrics/current/index.html

This means that we can export metrics in the format expected from the Sink and this gives the power to the user to easily have Neo4j metrics without any specific configuration, simply just leveraging the Kafka Connect architecture...

[moxious]

I see this could work, but it would require config, just not in Neo4j. So I could publish metrics (Neo4j) -> (Neo4j Streams) -> (Kafka) -> (Prometheus Sync)

But why would this be better than, or in what situation would this be preferable to (Neo4j) -> (Prometheus) -> (Thing that can sink prometheus metrics)

[ali-ince]

Closing as Neo4j already supports prometheus endpoint for metrics exposure.