search

LucaCanali / sparkMeasure

This is the development repository for sparkMeasure, a tool and library designed for efficient analysis and troubleshooting of Apache Spark jobs. It focuses on easing the collection and examination of Spark metrics, making it a practical choice for both developers and data engineers.
Apache License 2.0
690 stars 144 forks source link

Getting Spark Task related metrics using Prometheus Push Gateway #57

Closed krishhkk closed 2 months ago

krishhkk commented 3 months ago

Hi,

I am able to see the stage related metrics exposed to prometheus using push gateway sink. But, I am not seeing any Task related metrics.

  1. Is Task related metrics are supported by prometheus push gateway sink ?
  2. If supported, is there a separate configuration to expose Task metrics ?
LucaCanali commented 3 months ago

Hi @krishhkk I am glad to hear that the Prometheus Pushgateway sink works for you. Currently it collects Stage-level metrics only. I have just updated the documentation to make this explicit: Prometheus Pushgateway sink doc