search

LucaCanali / sparkMeasure

This is the development repository for sparkMeasure, a tool and library designed for efficient analysis and troubleshooting of Apache Spark jobs. It focuses on easing the collection and examination of Spark metrics, making it a practical choice for both developers and data engineers.
Apache License 2.0
690 stars 144 forks source link

Feature Request: Support Prometheus Gateway Sink in Flight Recorder Mode #54

Closed kiranatlu closed 6 months ago

kiranatlu commented 6 months ago

Implement a prometheus sink in in Flight Recorder Mode that formats metrics into a Prometheus-friendly format and pushes them to a configured Prometheus Pushgateway endpoint.

sparkmeasure currently offers Prometheus sink capabilities, facilitating the monitoring of Spark performance metrics within Prometheus ecosystems. Extending this functionality to include support for Flight Recorder Mode represents a significant enhancement. This feature would enable users to push metrics directly to a Prometheus Pushgateway endpoint.

LucaCanali commented 6 months ago

A first version of Prometheus Gateway Sink in Flight Recorder Mode at the commit: https://github.com/LucaCanali/sparkMeasure/commit/4a6f525f1dec8dcf5515b673a0ab440c4bd27cca

It should make it to the next sparkMeasure release (v0.24)