Make mapping of metric parts to Prometheus metrics/labels configurable

[juliusv]

Prometheus's true power comes from its multi-dimensional data model: metric names with arbitrary key/value dimensions. I wonder if it would be a good idea to allow some configurability of the PrometheusSink to allow such mappings. If I understand the go-metrics data model correctly, it's StatsD/Graphite-like, in the sense that a metric simply has a number of parts (separated by dots or whatever else in the end). In Prometheus's statsd-bridge, we already have a very simple mapping language for this type of componentized metric into more useful Prometheus metrics: https://github.com/prometheus/statsd_bridge#metric-mapping-and-configuration

However, it could / should arguably be even a bit more powerful, like allowing regex-based rewriting of metrics and labels.

/cc @stapelberg, who built the PrometheusSink and got bitten by getting the hostname in metric names via this package (via Hashicorp's Raft package using go-metrics).

[juliusv]

By the way, I'm mainly interested in opinions for now. Happy to implement something myself here at some point if there's an agreement (although it's not terribly urgent for myself - others might have a more pressing need to build it and that's cool too).

[armon]

I don't have strong opinions here, I just merged in the work that @stapelberg did as we don't use Prometheus ourselves.

[juliusv]

Yup, makes sense. Ok! I'll just let this sit here for now if that's ok, until either I or someone else finds the time to implement it.

[armon]

Sounds good

[stapelberg]

@juliusv According to https://github.com/prometheus/statsd_bridge/blame/master/mapper.go, you are (almost) the sole author of the mapper code in the statsd bridge. Would you be willing to relicense the code as MIT (which go-metrics uses)? Then we could just copy over the mapper (or move it into a shared repository, whichever way you prefer) into go-metrics and expose it in the interface.

[juliusv]

@stapelberg Hey, I'd be happy to relicense under any conditions necessary to be useful. Just wondering if we want to take exactly that code or wait for some more flexible regex-based code like in the JMX Exporter (https://github.com/prometheus/jmx_exporter) that enables more use cases. Or, since mappers could just be injected objects, we could add the current statsd-bridge-style mapper now and add another one later. Or we only do the more complete one and leave the simple one out. What do you think?

[juliusv]

Perhaps as a first step, we should add an interface for passing a mapper to a given PrometheusSink, and the interface methods that mapper has, and then anyone can do anything they want anyways :)

[stapelberg]

@juliusv I took a brief look at the JMX one, but it’s written in Java and I’m not very fluent in that. Can you outline the advantage the JMX mapper has?

Given that the interface seems to be getMapping(statsdMetric string) (labels prometheus.Labels, present bool) (can the JMX one satisfy that?), I think adding the interface and the simple mapper would be a good first step, and then we can just drop in a more complex mapper if need be.

[juliusv]

@stapelberg SGTM. Actually, besides the actual interface definition, there's work needed in the PrometheusSink itself to act on those mappings and dynamically generate Prometheus metric vectors (metrics with label dimensions). I don't have a lot of time to spare at the moment - would you be interested in working on that? Otherwise I could try to squeeze it into some free time slot at some point, but it could take a while.

[juliusv]

As for the mapper interface definition, perhaps it would make sense for the metric name to be treated specially (instead of being just part of the labels), because otherwise the mapper needs to use and depend on the internal metric name label name (__name__). So:

mapMetric(goMetric string) (name string, labels prometheus.Labels, present bool)

[stapelberg]

Just to be clear, I don’t currently have enough time to drive this. @juliusv, I’d be happy if you could take a stab at it at some point.

[jeinwag]

I did some work on this. Added a PrometheusMapper interface an implemented a basic RegexpMapper, similar to what's possible in jmx_exporter. Please comment.