Document real world examples of using relabel_config

[davidmankin]

WRT http://prometheus.io/docs/operating/configuration/#%3Crelabel_config%3E

The documentation for relabel_config is very clinical and hard to understand. (At least it is accurate, as far as I can tell.)

In both relabel_configs and metric_relabel_configs some real-world examples would go a long way. (In fact, relabel_configs doesn't even have a paragraph. metric_relabel_configs says "One use for this is to blacklist time series that are too expensive to ingest" but gives no hint at how to accomplish that.)

It would be great if the examples gave some typical uses for relabeling, and included examples of relabeling a kubernetes_sd_config or similar.

I found the relabelling much easier to understand after I found this blog post which contained a few sparse examples: https://prometheus.io/blog/2015/06/01/advanced-service-discovery/ but, being a blog post instead of docs, it's already out of date.

[fabxc]

I agree that more examples would are necessary and that blog posts come with the issue of going out of date. I'm strongly in favor for keeping the core docs clinical and precise though.

If I want to look something up and know what I'm looking for, this shouldn't be disturbed by all the examples. One doesn't want to scan through a lot of text that may or may not contain feature specification.

A guide section would probably be the way to go for having up-to-date examples with more detailed explanation.

As so often I see the issue that this (as the current documentation) will need versioning at some point. Any suggestions about decent software for that are welcome.

[davidmankin]

Further, it'd be great if there could be some suggestions or howto on developing, testing, and debugging relabel configs. At least point out that the applied relabels appear the /status page and the original input labels appear in a tooltip. (I have no idea how to go about testing/debugging metric relabelling. Is there a similar feature?)

[davidmankin]

The Kubernetes documentation is versioned with the release. In github they have a header that says "warning this is the HEAD documentation" and then they strip it out when snapshotting the docs for a release.

[davidmankin]

I don't mind clinical & precise reference docs as long as they include a reference to a guide section of the docs.

[marekbruchaty]

I'm finding it very difficult to understand the relabeling functionality from the official docs. Few straight forward examples would be extremely useful.

[grobie]

The lack of examples and readability of our docs is one of the most common things I hear from people at meetups.

Fortunately with interactive web pages it's not a black/white or concise=no-examples question. I don't see any reason we can't provide additional examples on the same page which only get shown upon click or something like that.

[brian-brazil]

The challenge with relabel examples is that they quickly get into configuration management philosophy, even when hidden they'd quickly become unwieldy as there's just so many possibilities.

I'm for a strong separation between reference docs and guides. I've seen where they've gotten mixed for the sake of getting people started, and it ended up making everyone unhappy.

being a blog post instead of docs, it's already out of date.

Our blog posts are meant to be up to date. Please file a bug for any inaccuracies you find.

[leonerd]

If it's any help I just stumbled upon this bug while struggling with a lack of docs. After much fiddling and flailing I've managed to make my use-case work.

My problem was I wanted to relabel bridge_JOBNAME_METRIC_NAME_HERE into bridge_METRIC_NAME_HERE{job=JOBNAME}. I've solved it by:

    metric_relabel_configs:
      # Extract the bridge name into the job label
      - source_labels: ["__name__"]
        regex: "bridge_([^_]+)_.*"
        target_label: "job"
        replacement: "$1"
      # Rewrite the label name to remove the bridge name out of it
      - source_labels: ["__name__"]
        regex: "bridge_[^_]+_(.*)"
        target_label: "__name__"
        replacement: "$1"

[brian-brazil]

You might want to throw that up on http://tips.robustperception.io/

[buchanae]

I'm still dumbfounded by using relabeling and service discovery. Normally I would just hit the +1 button, but this issue is nearly 2 years old.

[discordianfish]

Well, about time to look into this again :) I agree with @grobie, we can both have a concise reference docs and some more practical guide/tips/examples. I remember vividly when I introduced Prometheus at Docker, the biggest concern was lack of examples/snippets to get people started. From a UX perspective, I think some sidebar you can open additionally to the reference, which shows further tips/examples.

I think it would be neat to have a separate repo for this with lower review requirements, so changes to the example don't need to be vetted by maintainers. A wiki might be even better, but something that is mapped to the docs and viable side by side (so you see right away when things don't add up and examples need updating). Something that gets loaded via js, so doesn't make the reference docs slower/bigger?

[bekir-murat]

Afaik, there is no need for fancy UI stuff, even simple links to examples in a git repo would go a long way. We decided to use Prometheus for our monitoring infrastructure but we are questioning our choice now that we are digging deeper and using more advanced features. Our main gripes are with configuration and discovery.

[jamtur01]

I wrote the getting started as a guide - https://prometheus.io/docs/prometheus/latest/getting_started/ (https://github.com/prometheus/docs/pull/895) and added a guide section - it'd be very easy to add further guides: using exporters, relabelling, service discovery, and querying all spring to mind as ideas.

[bekir-murat]

Thank you for this, I used the getting started guide for our initial setup. Would it still be a getting started guide though, if these subjects were added ?

[jamtur01]

That'd be up to the Prometheus team. As there was only one guide at the time (Getting started) it got rolled into the existing docs and the Guide section not used.

[bojleros]

I'am looking for advice. Let's have a look at this data:

ifDescr{ifDescr="bridge",ifIndex="7"} 1 ifDescr{ifDescr="ether2-master",ifIndex="2"} 1 ifDescr{ifDescr="ether3",ifIndex="3"} 1 ifDescr{ifDescr="ether4",ifIndex="4"} 1 ifDescr{ifDescr="ether5",ifIndex="5"} 1 ifDescr{ifDescr="gateway",ifIndex="1"} 1 ifDescr{ifDescr="wlan1",ifIndex="6"} 1 ifHCOutOctets{ifIndex="1"} 1.046238444e+09 ifHCOutOctets{ifIndex="2"} 8.013592689e+09 ifHCOutOctets{ifIndex="3"} 2.897353597e+09 ifHCOutOctets{ifIndex="4"} 0 ifHCOutOctets{ifIndex="5"} 1.5678323668e+10 ifHCOutOctets{ifIndex="6"} 4.869636567e+09 ifHCOutOctets{ifIndex="7"} 1.4733548523e+10

Do you have any idea how can i include ifDescr label into ifHC.*Octets by matching ifIndex ? I do know that this question is not strictly related to this issue however i was unable to find docs/answers and it would make a nice example.

[sahilg1]

Can someone please tell me what am I doing wrong here? Given the address to be something like 10.45.22.234:2122, I want the instance label to hold 10.45.22.234 and the address to be 10.45.22.234:9274.

  - source_labels: ['job']
    regex: ^(prometheus-node-exporter).*
    action: keep
  - source_labels: [__address__]
    regex: ^(.*):(.*)$
    target_label: instance
    replacement: ${1}
  - source_labels: [__address__]
    regex: ^(.*):(.*)$
    target_label: __address__
    replacement: ${1}:9274

Right now all I am getting is instance=":9274" and address=http://:9274/metrics

In another case, I have config -

  - source_labels: ['job']
    regex: ^(blackbox-exporter).*
    action: keep
  - source_labels: [__address__]
    regex: ^(.*):(.*)$
    target_label: instance
    replacement: ${1}

Here it spits instance="10.45.22.234:2122" (It doesn't parse out the group reference and gives the whole address) Prometheus v2.2.1

@grobie @leonerd @brian-brazil @jamtur01 I have been stuck with this for a while, appreciate any help!

[pgporada]

~I'm having a hard time figuring out the same use case as @sahilg1 .~ As far as I can tell, your regex has to match the entire label you're attempting to manipulate.

Example: __address__="prod-loadbalancer-43:9100" or __address__="staging-database-02:9100"

  - job_name: example_job_name_whatever
    relabel_configs:
      - source_labels: ['__address__']
        regex: '(\w+)\-(\w+)\-(\d+):(\d+)'
        target_label: 'environment'
        replacement: '$1'
      - source_labels: ['__address__']
        regex: '(\w+)\-(\w+)\-(\d+):(\d+)'
        target_label: 'role'
        replacement: '$2'
      - source_labels: ['__address__']
        regex: '(\w+)\-(\w+)\-(\d+):(\d+)'
        target_label: 'instance_number'
        replacement: '$3'

I am then able to use the new labels like this, some_custom_stat{environment="prod"} or some_other_stat{environment="staging",role!~"database"}

If someone needs an RE2 regex tester, I used https://regex101.com/ and selected golang.

[brian-brazil]

It makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.

[ravishivt]

Here's a perspective from a new prometheus user on learning these concepts. I spent a a couple days hunting down information about service discovery and relabel configs to debug a particular issue. The official docs were helpful in some regards but very lacking in others. I was lost on the big picture concepts of even the purpose of relabel_configs and why you would want to do it. Only until I found videos like https://www.youtube.com/watch?v=b5-SvvZ7AwI and the associated blog posts on https://www.robustperception.io/blog did it start to make sense. Even then, I watched/read those with some suspicion on whether they are worth my time since they are 2 years old. Saying "blog posts are meant to be up to date" is great but from the reader's perspective, it's hard for the user to trust that as it's not a common practice with blog posts.

Having examples in the official docs would help accomplish two goals. First is obviously to provide specific examples of common patterns that the user can use as a reference to write or debug their own configs. The second goal, less obvious than the first, is for the reader to glean through examples on why/when you would even use these features. It helps paint the big picture for them.

Also, I wasn't aware of the Prometheus Service Discovery UI status page until many hours of struggling. Only until I stumbled on this issue did I see it referenced. This page is tremendously helpful to see the mapping between discovered and target labels but it's only helpful if people know it's a resource available to them. This page should be promoted more on the official docs.

[ravishivt]

Here's another good blog post for others struggling with these concepts. https://prometheus.io/blog/2015/06/01/advanced-service-discovery/

This is another example of a blog post that should be converted to the official docs. The post itself is three years old and it reads partially as a v0.14.0 release note instead of a guide. Even with those issues, the post contains some of the most helpful information regarding relabelling that I was able to find.

[RichiH]

@lucperkins Coming up with a few specific examples and collating all information would be worthwhile.

[retrogradeorbit]

One thing that confused me was the name 'relabel' is itself misleading. I was under the assumption that it was a tool for changing the labels, not for creating new labels with new metric data, and not for processing the metric values themselves that are under those labels into new labels. When the documentation is improved this kind of high level overview of what can be done, and why, should be up front.

[SuperQ]

I have suggested calling it target_relabel_configs to differentiate it from metric_relabel_configs.

We could offer this as an alias, to allow config file transition for Prometheus 3.x.

[carlbordum]

Is the Guides section of the docs intended for stuff like this?

https://prometheus.io/docs/guides/

[brian-brazil]

Yes, guides is where things like this would go.

[AndyFHAF]

good

[gillg]

Just a related idea (not trivial !)... If someone see how to create a JS webapp or a webassembly in go to test relabeling_rules it could be the graal 😅

I know lot of devs not confortable with regex, and lot of sites like https://www.debuggex.com/ help them a lot to create / test regex. I know also lot of Prometheus adopters (including me) not very comfortable with relabel configs ! It would be awsome to put a on it, a list a metrics (copied from /metrics endpoint) and see the result.

That was my 2cents

[saliceti]

@gillg this is very useful: https://relabeler.promlabs.com/