search

Stackdriver / stackdriver-prometheus-sidecar

A sidecar for the Prometheus server that can send metrics to Stackdriver.
https://cloud.google.com/monitoring/kubernetes-engine/prometheus
Apache License 2.0
121 stars 43 forks source link

Filter prometheus metric labels #133

Closed memory closed 5 years ago

memory commented 5 years ago

Stackdriver has a hard quota of 10 labels per external metric, but prometheus metrics may have an arbitrary number of labels.

Add a --droplabel flag (repeatable) to assemble a list of prometheus labels to never forward to stackdriver.

memory commented 5 years ago

This may help with some of the issues noted in https://github.com/Stackdriver/stackdriver-prometheus-sidecar/issues/114

jkohen commented 5 years ago

Thanks for the contribution!

@StevenYCChou you've been thinking about this recently, can you take a look?

memory commented 5 years ago

n.b. if people like the general approach, I'll be happy to add unit tests and update the documentation.

memory commented 5 years ago

Ping? :)

StevenYCChou commented 5 years ago

Hi @memory,

Another alternative is to have configuration interface as existing <relabel_config> in Prometheus Configuration. I think it provides the same user experience Prometheus users already familiar with, and we may be able to utilize the library in Prometheus project as well.

I think this Prometheus <relabel_config> interface with relabel_action:label_drop will solve the issue you described. What do you think?

To make it further, are there other your usecases that <relabel_config> could not be covered?

memory commented 5 years ago

After some thought and some burn-in time on my end, I'm closing this PR as I've come to think that it's frankly kind of a bad idea: dropping labels off of metrics (unsurprisingly) results in what stackdriver sees as duplicate and out of order entries; the correct solution is to use a recording rule to aggregate across labels that you want to drop and ship the recorded metrics to stackdriver; for which see https://github.com/Stackdriver/stackdriver-prometheus-sidecar/issues/104 for further discussions.