[SUPPORT] Pushing hoodie metrics to prometheus having error

[data-storyteller]

Tips before filing an issue

• Have you gone through our FAQs?

• Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

• If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced

I am working on pushing HUDI metrics to Prometheus. The HUDI metrics are working good with GRAPHITE but not Prometheus.

HUDI Configs -

      --hoodie-conf   hoodie.metrics.on=true
      --hoodie-conf  hoodie.metrics.reporter.type=PROMETHEUS
      --hoodie-conf  hoodie.metrics.pushgateway.host=host
      --hoodie-conf hoodie.metrics.prometheus.port=9090

To Reproduce

Steps to reproduce the behavior:

1. Run Prometheus server
2. Run Spark job with above hudi config

Expected behavior

Spark Job should push the HUDI metrics to prometheus

Environment Description

• Hudi version : 0.9.0

• Spark version : 3.1.1

• Hive version : -

• Hadoop version : -

• Storage (HDFS/S3/GCS..) : -

• Running on Docker? (yes/no) : -

Additional context

Add any other context about the problem here.

Stacktrace

Exception in thread "main" java.lang.NoSuchMethodError: 'void io.prometheus.client.dropwizard.DropwizardExports.<init>(org.apache.hudi.com.codahale.metrics.MetricRegistry)'
    at org.apache.hudi.metrics.prometheus.PrometheusReporter.<init>(PrometheusReporter.java:49)
    at org.apache.hudi.metrics.MetricsReporterFactory.createReporter(MetricsReporterFactory.java:75)
    at org.apache.hudi.metrics.Metrics.<init>(Metrics.java:50)
    at org.apache.hudi.metrics.Metrics.init(Metrics.java:96)
    at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamerMetrics.<init>(HoodieDeltaStreamerMetrics.java:44)
    at org.apache.hudi.utilities.deltastreamer.DeltaSync.<init>(DeltaSync.java:224)
    at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.<init>(HoodieDeltaStreamer.java:606)
    at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.<init>(HoodieDeltaStreamer.java:143)
    at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.<init>(HoodieDeltaStreamer.java:107)
    at org.apache.hudi.integ.testsuite.HoodieDeltaStreamerWrapper.<init>(HoodieDeltaStreamerWrapper.java:39)
    at org.apache.hudi.integ.testsuite.HoodieTestSuiteWriter.<init>(HoodieTestSuiteWriter.java:88)

[nsivabalan]

  option("hoodie.metrics.on", "true").
  option("hoodie.metrics.reporter.type", "PROMETHEUS_PUSHGATEWAY").
  option("hoodie.metrics.pushgateway.host", "localhost").
  option("hoodie.metrics.pushgateway.port", "9091").
  option("hoodie.metrics.pushgateway.job.name", "spark_metrics_test").
  option("hoodie.metrics.pushgateway.random.job.name.suffix", "false").
  option("hoodie.metrics.pushgateway.delete.on.shutdown", "false").

this worked.

Before executing the spark write, I had started push gateway locally.

docker pull prom/pushgateway
docker run -d -p 9091:9091 prom/pushgateway

[rubenssoto]

@nsivabalan I follow your configs and didnt have any error but I cant found the metrics on my prometheus, how should I search? What is the prefix?

thank you

[rubenssoto]

I think spark never try to write to prometheus, even if I put a wrong address, no error.

[xushiyan]

@data-storyteller @rubenssoto can you check out this guide prepared by @nsivabalan (to be merged to website) and see the instructions help? https://github.com/apache/hudi/commit/959bd6eef8c90c11616840f975ef40a46222a913?short_path=aff66ea#diff-aff66ea1c34953a024c85c6e2fe86b8521b6cd3d623377a96d8d79c6caa8de13

@data-storyteller

Exception in thread "main" java.lang.NoSuchMethodError: 'void io.prometheus.client.dropwizard.DropwizardExports.<init>(org.apache.hudi.com.codahale.metrics.MetricRegistry)'

Looks like it's a jar issue. Are you using hudi bundle jar? can you print your classpath too?

[xushiyan]

I think spark never try to write to prometheus, even if I put a wrong address, no error.

@rubenssoto can you share your settings? @liujinhui1994 could you give any suggestions or hint to the prometheus problems above?

[nsivabalan]

@rubenssoto : Can you please try out the suggestions from raymond and let us know if it worked. we wish to document the right steps in our website. would appreciate if you can confirm if it works.

[nsivabalan]

CC @kywe665

[rubenssoto]

@nsivabalan sorry it was my fault, I was using prometheus not prometheus push gateway, I thought it was the same thing, but it is not.

[nsivabalan]

@rubenssoto : so, can I take it as it worked for you? (the proposed steps). once you ack, will close it out.