Metric relabel ignored

[milosgajdos]

Component(s)

collector

Describe the issue you're reporting

We have a small EKS cluster that runs kube-state-metrics for reporting the metrics from it.

The metrics are scraped by adot collector which has the following scrape config (the rest of the config is omitted):

receivers:
  prometheus:
    config:
      global:
        scrape_interval: 1m
        scrape_timeout: 40s

      scrape_configs:
      - job_name: 'kube-state-metrics'
        kubernetes_sd_configs:
          - role: endpoints
        relabel_configs:
          - source_labels: [__meta_kubernetes_service_name]
            regex: kube-state-metrics
            action: keep
          - target_label: cluster
            replacement: our-cluster-name

This should add cluter="cluster-name" label to every metric scraped from kube-state-metrics job alas, it's completely ignored for some reason.

Other things we've tried:

          scrape_configs:
          - job_name: 'kube-state-metrics'
            kubernetes_sd_configs:
              - role: endpoints
            relabel_configs:
              - source_labels: [__meta_kubernetes_service_name]
                regex: kube-state-metrics
                action: keep
            metric_relabel_configs:
              - target_label: cluster
                replacement: our-cluster-name

I am not sure if this is a bug or if we are missing something here, but this seems like it should work as it works for other prometheus scrapers. Equally, if you check https://relabeler.promlabs.com/ this should add the cluster label in. If anyone has any ideas that'd be greatly appreciated!

[pavolloffay]

This might be a bug in the prometheus receiver which should be reported in https://github.com/open-telemetry/opentelemetry-collector-contrib/