Support additional relabelings in scrape classes

[SuperQ]

What is missing?

With the new scrape classes feature, it would be useful to allow additional relabelings that are added after the operator-managed relabelings, but before the relabelings supplied by the scrape target (ServiceMonitor, PodMonitor, etc)

This is an extension of the request in https://github.com/prometheus-operator/prometheus-operator/issues/3255.

Why do we need it?

This is needed in order to allow Prometheus admins to extend and enforce relabelings for downstream users of various scrape configs.

Examples:

Always propagate the deployment label:

kind: Prometheus
spec:
  scrapeClasses:
    - name: deployments
      default: true
      additionalRelabelings:
        - sourceLabels: [__meta_kubernetes_pod_label_deployment]
          targetLabel: deployment

Always propagate release information provided by CD tooling:

kind: Prometheus
spec:
  scrapeClasses:
    - name: versioning
      default: true
      additionalRelabelings:
        - sourceLabels: [__meta_kubernetes_pod_annotation_example_com_version]
          targetLabel: release_version

[rexagod]

Working on this.

[nicolastakashi]

@rexagod Tge ScrapeClasse design is not finished yet.

[rexagod]

Ah, I see. I guess I'll wait it out then, thanks for the heads up!

[sathieu]

I have a different use-case: We define a tenant as a list of namespaces (or namespace regexps). I want to create a tenant label for each metric.

I need something like:

kind: Prometheus
spec:
  scrapeClasses:
    - name: deployments
      default: true
      additionalMetricsRelabelings:
        - sourceLabels: [namespace]
          regex: tenant1-.*
          targetLabel: tenant
          replacement: tenant1
        - sourceLabels: [namespace]
          regex: foo|bar
          targetLabel: tenant
          replacement: foo-bar

This metrics relabeling should be enforced after scrape target metrics relabeling. Also if the scrape target references an unexisting scrapeClass, the default scrape class should be applied (for security reasons).

[simonpasquier]

@sathieu this is an interesting use case but I'm not sure that scrape classes (as they are currently envisioned #6113) are the right option. The design proposal calls out the following non-goal:

• Provide a way for Prometheus administrators to enforce/override settings on scrape configurations.

Even if the operator enforced the default scrape class when the scrape resource refers to a non-existing name, you'd have to repeat the same relabel configs for all scrape classes if you define several. I would rather see it as an extension of the existing .spec.enforcedNamespaceLabel mechanism.

cc @nicolastakashi @ArthurSens

[nicolastakashi]

Btw @simonpasquier on the design proposal we assume default scrape class will be assigned automatically to all monitor resources which didn't specify a scrape class.

One scrape class may be designated as the default class, in which case that class is applied to any scrape resource that doesn't specify a value for scrapeClassName.

I still think it's a valid use case have metricsRelabelings on the scrape class and inject this on every endpoint from the monitor object, although if endpoint from the monitor object also defines a metricsRelabelings the values will be merged.

[ArthurSens]

I still think it's a valid use case have metricsRelabelings on the scrape class and inject this on every endpoint from the monitor object, although if endpoint from the monitor object also defines a metricsRelabelings the values will be merged.

I'm aligned with this. We've received lots of requests for a way to apply relabel configs across all targets and I feel like ScrapeClasses are a good fit for the job

[sathieu]

According to current implementation:

https://github.com/prometheus-operator/prometheus-operator/blob/9715735f601172987eacb81e8d304abb5caaa71d/pkg/apis/monitoring/v1/prometheus_types.go#L1818-L1821

So my usecase https://github.com/prometheus-operator/prometheus-operator/issues/5947#issuecomment-1845611166 is still not covered. How to move forward?

[nicolastakashi]

@sathieu could you elaborate a bit more? I don't follow why the current metrics relabel config didn't work for your use case.

[simonpasquier]

IIUC @sathieu wants a metricRelabelings field in ScrapeClass, in addition to relabelings. I think that it makes sense to add this.

https://github.com/prometheus-operator/prometheus-operator/blob/9715735f601172987eacb81e8d304abb5caaa71d/pkg/apis/monitoring/v1/prometheus_types.go#L1816-L1826

[sathieu]

Yes. I have started a PR for this https://github.com/prometheus-operator/prometheus-operator/pull/6492. But I need to setup my go env to move forward.