Docs: document distributor's metric_relabel_configs

[pracucci]

Mimir distributor supports per-tenant metric_relabel_configs, which allows you to manipulate received series' labels or drop received series matching specific labels. We should document this feature and provide some examples.

[Rustin170506]

@pracucci I found and read the codes and I would like to help solve this issue. Can you please give me some instructions? Because it is difficult for me to find a proper place to add it. Also, what descriptions would you like to have in the documentation? Because it has multiple small features.

[pracucci]

Currently, this configuration option is only mentioned at docs/sources/operators-guide/configure/reference-configuration-parameters/index.md:

# (experimental) List of metric relabel configurations. Note that in most
# situations, it is more effective to use metrics relabeling directly in the
# Prometheus server, e.g. remote_write.write_relabel_configs.
[metric_relabel_configs: <relabel_config...> | default = ]

I would suggest to create a new dedicated page to explain what it does and how to configure it. This new page could be like other "configure ..." pages at docs/sources/operators-guide/configure/ (for example, look at docs/sources/operators-guide/configure/configure-out-of-order-samples-ingestion.md).

What should we document:

• The use case: you need to relabel incoming metrics or drop some of them
• How to do it: prefer to do it at the sender side (Prometheus or Grafana Agent) but if, for any reason, you can't do it at the sender side, you can do it in the distributor setting metric_relabel_configs
• A couple of examples: how to drop series matching a given label matcher, how to change a series label matching a given label matcher

[putrasattvika]

Hi, I'm currently exploring this feature. Can you explain why "in most situations, it is more effective to use metrics relabeling directly in the Prometheus server" ?

Is it simply because by we're able to distribute the computational load by delegating relabeling to Prometheus instances (which are most likely located in separate VMs); or is there something that prevents Distributor from being as efficient as Prometheus on metrics relabeling?

[pracucci]

Can you explain why "in most situations, it is more effective to use metrics relabeling directly in the Prometheus server" ?

If you use relabelling for dropping series or drop some labels, then you have the network transfer if you drop them on the Prometheus server. If it's a pure relabelling, then there may be no significative impact on networking.

[putrasattvika]

If you use relabelling for dropping series or drop some labels, then you have the network transfer if you drop them on the Prometheus server. If it's a pure relabelling, then there may be no significative impact on networking.

Noted, thank you for the explanation. I think this would be informative if added into the feature documentation.

[pracucci]

I think this would be informative if added into the feature documentation.

We're looking for some help to write this doc. @putrasattvika do you wanna pick it up if @hi-rustin won't take it?

[Rustin170506]

I think this would be informative if added into the feature documentation.

We're looking for some help to write this doc. @putrasattvika do you wanna pick it up if @hi-rustin won't take it?

I'm OK with you picking it up.

[putrasattvika]

I'm still exploring Mimir at this point, so I can't promise you anything. But if it turns out that we're going to use Mimir in the future, I might revisit this issue if it hasn't been worked on yet.