search

banzaicloud / spark-metrics

Spark metrics related custom classes and sinks (e.g. Prometheus)
Apache License 2.0
176 stars 64 forks source link

Q: Expose AccumulatorSource #56

Open f1yegor opened 4 years ago

f1yegor commented 4 years ago

Question about exposing spark accumulators to Prometheus.

I think I read about it in the documentation, but couldn't find now. Could I expose value of spark accumulator? I assume this should work

*.source.jvm.class=org.apache.spark.metrics.source.JvmSource
*.source.jvm.class=org.apache.spark.metrics.source.LongAccumulatorSource

but I don't see spark accumulator value exposed.

sancyx commented 4 years ago

Hi @f1yegor, as I know org.apache.spark.metrics.source.LongAccumulatorSource is available only from Spark 3.0 which is still in preview and we don't yet support it with spark-metrics.

f1yegor commented 4 years ago

Ok, I will answer my question (I tested with Spark 3.0, but I believe it will work with 2.4+):

you need to register you accumulator like

val successCounter = spark.sparkContext.longAccumulator("records_success")

LongAccumulatorSource.register(
      spark.sparkContext,
      Map(
        successCounter.name.get -> successCounter)
    )

and include

*.source.jvm.class=org.apache.spark.metrics.source.LongAccumulatorSource

in $SPARK_HOME/metrics.properties metric will be reported with timeseries driver_accumulatorsource_records_success