[kube-prometheus-stack] Grafana "Use an existing secret for the admin user" used to change password?

[dcfitzpatrick00]

Describe the bug a clear and concise description of what the bug is.

My values.yaml has the following to use an existingSecret for Grafana:

    grafana:
      admin:
        existingSecret: "kube-prometheus-stack-grafana"
        userKey: admin-user
        passwordKey: admin-password

I'm using SealedSecrets / AgroCD - To deploy an UPDATED SealedSecret file with new Grafana password. The deployment works but Grafana ignores the new password within the secret. Grafana is still accepting the original password from the previous version of the secret, from the initial install. Trying to determine if this is by design or bug.

What's your helm version?

$ helm version version.BuildInfo{Version:"v3.10.2", GitCommit:"50f003e5ee8704ec937a756c646870227d7c8b58", GitTreeState:"clean", GoVersion:"go1.18.8"}

What's your kubectl version?

$ kubectl version WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.3+k3s1", GitCommit:"f2585c1671b31b4b34bddbb3bf4e7d69662b0821", GitTreeState:"clean", BuildDate:"2022-10-25T19:59:38Z", GoVersion:"go1.19.2", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v4.5.7 Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.3+k3s1", GitCommit:"f2585c1671b31b4b34bddbb3bf4e7d69662b0821", GitTreeState:"clean", BuildDate:"2022-10-25T19:59:38Z", GoVersion:"go1.19.2", Compiler:"gc", Platform:"linux/amd64"}

Which chart?

kube-prometheus-stack

What's the chart version?

41.7.4

What happened?

Troubleshooting: This is the only secret I have with the word grafana in it:

$ k get secrets -A | grep grafana monitoring kube-prometheus-stack-grafana Opaque 3 2d1h The admin ID & password are defined within the secret:

$ k describe secrets -n monitoring kube-prometheus-stack-grafana
Name: kube-prometheus-stack-grafana Namespace: monitoring Labels: Annotations:

Type: Opaque

Data

admin-password: 17 bytes admin-user: 5 bytes ldap-toml: 0 bytes I can even inspect the secret and see SealedSecrets has applied the new password (I've changed it again since):

$ kubectl get secret --namespace monitoring kube-prometheus-stack-grafana -o jsonpath='{.data.admin-password}' | base64 -d ZvtZwNRGsFU5nsLOz I only have a single Grafana pod, I've restart it to force it to pick up the new password:

$ k get pods -A | grep grafana
monitoring grafana-5c84cc4c77-jtdqp 3/3 Running 0 51s When I describe the pod, it states the ENV variable will be set from the secret:

Environment: GF_SECURITY_ADMIN_USER: <set to the key 'admin-user' in secret 'kube-prometheus-stack-grafana'> Optional: false GF_SECURITY_ADMIN_PASSWORD: <set to the key 'admin-password' in secret 'kube-prometheus-stack-grafana'> Optional: false I can view the ENV variables passed inside the Pod showing the new password is being picked up:

$ k exec -n monitoring grafana-5c84cc4c77-jtdqp -c grafana -- env | grep ADMIN
GF_SECURITY_ADMIN_USER=admin GF_SECURITY_ADMIN_PASSWORD=ZvtZwNRGsFU5nsLOz However, the Grafana dashboard ignored new password; only the original password "prom-operator" is being accepted for account "admin".

Where is it getting this old password from? I assume it stored in the database and the secrets are initial install only and then no longer used? I was unable to find this information in the documentation.

What you expected to happen?

I expected the admin password to be updated based on the value stored within the secret.

How to reproduce it?

Change password stored within Grafana secret and restart the pod. The ENV variable GF_SECURITY_ADMIN_PASSWORD reflects the new password but is ignored by Grafana.

Enter the changed values of values.yaml?

No response

Enter the command that you execute and failing/misfunctioning.

Grafana Dashboard only accepts initial secret password value, not updated password.

Anything else we need to know?

I could understand if this is by design, but I can't find a reference saying that Kubernetes secret is used for Grafana initial install only and ignored afterwards.

[dcfitzpatrick00]

This is issue https://github.com/prometheus-community/helm-charts/issues/2710 created by @reefland which should not have been closed because it is still an issue.

[rouke-broersma]

Kube prom stack uses the upstream grafana chart. You should ask there: https://github.com/grafana/helm-charts/tree/main/charts/grafana

[Barsonax]

I just did it like so and it worked, it restarted itself and now accepts the new credentials:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: kube-prometheus-stack
spec:
  repo: https://prometheus-community.github.io/helm-charts
  chart: kube-prometheus-stack
  targetNamespace: monitoring
  version: 66.2.2
  valuesContent: |-
    grafana:
      admin:
        existingSecret: "grafana-credentials"
        userKey: username
        passwordKey: password

Maybe this is an older issue or you are doing something different?

[mkhodave]

I have tried same with similar configuration but in pulumi typescript code as following

      grafana: {
        enabled: true,
        admin: {
          existingSecret: 'grafana-secret',
          userKey: 'username',
          passwordKey: 'password',
        },

but still it refers default password even I can see ENV VARs GF_SECURITY_ADMIN_USER & GF_SECURITY_ADMIN_PASSWORD correctly showing my secret value. still new password is not working , any suggestions ?