search

LucaCanali / sparkMeasure

This is the development repository for sparkMeasure, a tool and library designed for efficient analysis and troubleshooting of Apache Spark jobs. It focuses on easing the collection and examination of Spark metrics, making it a practical choice for both developers and data engineers.
Apache License 2.0
705 stars 145 forks source link

Prometheus support #10

Closed alitet01 closed 6 years ago

alitet01 commented 6 years ago

Added export taskmetrics and stagemetrics to the Prometheus monitoring system. Some usefull info placed in additional prometheus.md document.

codecov-io commented 6 years ago

Codecov Report

Merging #10 into master will increase coverage by 8.65%. The diff coverage is 45.45%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #10      +/-   ##
==========================================
+ Coverage    8.27%   16.93%   +8.65%     
==========================================
  Files           4        5       +1     
  Lines         290      378      +88     
  Branches       34       44      +10     
==========================================
+ Hits           24       64      +40     
- Misses        266      314      +48
Impacted Files Coverage Δ
.../main/scala/ch/cern/sparkmeasure/taskmetrics.scala 0% <0%> (ø) :arrow_up:
...main/scala/ch/cern/sparkmeasure/stagemetrics.scala 0% <0%> (ø) :arrow_up:
.../main/scala/ch/cern/sparkmeasure/pushgateway.scala 85.1% <85.1%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update c2cf272...87f7acf. Read the comment docs.

LucaCanali commented 6 years ago

Thanks for the PR. OK for me. However, I'd rather change the name of the sendReport method to something like sendReportPrometheus, this is to "open the namespace" for additional implementations of metrics sinks (for example InfluxDB, Kafka)