search

LucaCanali / sparkMeasure

This is the development repository for sparkMeasure, a tool and library designed for efficient analysis and troubleshooting of Apache Spark jobs. It focuses on easing the collection and examination of Spark metrics, making it a practical choice for both developers and data engineers.
Apache License 2.0
690 stars 144 forks source link

Add support for SparkMeasure Flight Recorder mode - HTTP Sink #53

Closed athityakumar closed 6 months ago

athityakumar commented 6 months ago

This would provide an easy way for SparkMeasure users to fan-off all the events to an HTTP API endpoint (OpenLineage is another spark listener that provides a similar HTTP sink support)

LucaCanali commented 6 months ago

A first version of Prometheus Gateway Sink in Flight Recorder Mode at the commit: https://github.com/LucaCanali/sparkMeasure/commit/4a6f525f1dec8dcf5515b673a0ab440c4bd27cca

It should make it to the next sparkMeasure release ( v0.24 )