[kube-prometheus-stack] Overriding external label don't work for label cluster

Mojito2110 commented 1 week ago

Describe the bug a clear and concise description of what the bug is.

Overriding external label don't work for label cluster. External label value replacement no longer works for cluster label with chart helm version 58.7.2 I deploy Prometheus with the chart helm kube-stack-prometheus by adding the Openshift cluster name in an external label with the key/value pair cluster=xxxx to identify the cluster in alertmanager and other external backends like Thanos.

The cluster label can be set in the values.yaml like this prometheus: prometheusSpec: externalLabels: cluster: "k501h"

or with the helm command like this --set prometheus.prometheusSpec.externalLabels.cluster="k501h"

What's your helm version?

version.BuildInfo{Version:"v3.9.0", GitCommit:"7ceeda6c585217a19a1131663d8cd1f7d641b2a7", GitTreeState:"clean", GoVersion:"go1.17.5"}

What's your kubectl version?

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"windows/amd64"} Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.11+749fe1d", GitCommit:"9654661a61cc44110a8a3a801a82482ab63d063d", GitTreeState:"clean", BuildDate:"2024-03-15T23:22:40Z", GoVersion:"go1.20.12 X:strictfipsruntime", Compiler:"gc", Platform:"linux/amd64"}

Which chart?

kube-prometheus-stack

What's the chart version?

58.7.2

What happened?

 external_labels:
    cluster: aa-application-monitoring/prometheus-apps-prometheus
    prometheus_replica: prometheus-prometheus-apps-prometheus-0

This section can be find in the Prometheus UI (config page) or with the command oc get secret/prometheus-prometheus-apps-prometheus -o jsonpath='{.data.prometheus.yaml.gz}' | base64 -d | gunzip -c

What you expected to happen?

I expect the outpout below

 external_labels:
    cluster: k501h
    prometheus_replica: prometheus-prometheus-apps-prometheus-0

How to reproduce it?

No response

Enter the changed values of values.yaml?

externalLabels: cluster: k501h

Enter the command that you execute and failing/misfunctioning.

helm upgrade --install --version 58.7.2 prometheus-apps prometheus/kube-prometheus-stack -f values.yaml --set prometheus.prometheusSpec.externalLabels.cluster=k501h

Anything else we need to know?

Override cluster label works with a older version 50.3.1 (app_version v0.67.1)

zeritti commented 1 week ago

Overriding external label don't work for label cluster.

An install with the default values setting prometheus.prometheusSpec.externalLabels.cluster does not confirm the described behaviour (checked with 65.2.0 and 58.7.2), i.e. label cluster gets set to the desired value.

helm install -n monitoring kps prometheus-community/kube-prometheus-stack \
  --set prometheus.prometheusSpec.externalLabels.cluster="mycluster"

gives in Prometheus UI

global:
  external_labels:
    cluster: mycluster
    prometheus: monitoring/kps-kube-prometheus-stack-prometheus
    prometheus_replica: prometheus-kps-kube-prometheus-stack-prometheus-0

There are a few fields in the prometheus CR, though, that are affecting global.external_labels. The corresponding chart's values are

# current defaults

prometheus:
  prometheusSpec:
    replicaExternalLabelName: ""
    replicaExternalLabelNameClear: false
    prometheusExternalLabelName: ""
    prometheusExternalLabelNameClear: false

It looks like prometheusExternalLabelName has been set to cluster in your installation (we see in your config that external label prometheus is not present and could have been replaced with cluster).

Mojito2110 commented 1 week ago

Close issue, resolve my problem, i must remove prometheusExternalLabelName: cluster in the values.yaml

prometheus-community / helm-charts