Filter duplicate metrics

banzaicloud / spark-metrics

Spark metrics related custom classes and sinks (e.g. Prometheus)

Apache License 2.0

175 stars 64 forks source link

Filter duplicate metrics #47

Closed stoader closed 4 years ago

stoader commented 4 years ago

Filter out metrics that come from different metric families and are duplicate by (name,type) prior posting to PushGateway

onzo-mateuszzakarczemny commented 4 years ago

@stoader I'm trying to rebase my branch and accommodate your changes. Could you explain me when its possible to have duplicates? I'm not sure how to write test case for that. When I'm trying to add duplicated metric I'm getting:

testDeduplication failed: java.lang.IllegalArgumentException: A metric named test-metric-sample1 already exists

stoader commented 4 years ago

Deploy spark master in standalone mode and enable the Sink for all metrics generated by Spark master.

See https://github.com/banzaicloud/spark-metrics/issues/43#issuecomment-554246909 for how to reproduce it.