prometheus / graphite_exporter

Server that accepts metrics via the Graphite protocol and exports them as Prometheus metrics
Apache License 2.0
354 stars 100 forks source link

How to find the metrics that are NOT matched by the rules #239

Open altmind opened 1 year ago

altmind commented 1 year ago

For graphite exporter mappings i see there's a special action: "drop" for a catch-all rule that can discard the incoming that dont match.

---

- match: "."
  match_type: regex
  action: drop
  name: "delete"

I wonder if there's a way to have a catch-all rule that saves the received metric to a file or to the stdout, so i can add track/add the unmatched metrics? I checked the statsd_exporter github and was unable to find all the possible actions there, maybe you know?

matthiasr commented 11 months ago

At the moment, there is no such feature. You can get some insight by enabling debug logging, which logs all incoming events. However, this can be very high volume.

I think this is a good idea, but we need to be careful about the log volume. We should probably only log each metric once. This leaves open what should happen with metrics that come with their own tagging; I guess since we only allow matching on metric names, we would still only need to log once per metric name without loss in visibility? What would the path forward be for someone who receives and drops very bad metrics with unbounded cardinality, i.e. a different name each time?

matthiasr commented 11 months ago

Implementation wise, since we re-use the statsd exporter mapping code here, the change would have to happen over there.