prometheus-operator StatefulSet does not rollout all pods

[angelbarrera92]

Hi team!

We are facing something similar to #145

The reloader works fine for regular statefulsets but something is not working correctly with statefulsets managed by prometheus-operator.

Reloader detects the change correctly, but it just triggers the update of one out of N (3 in my case). I'm almost sure this is an incompatibility with Prometheus Operator, WDYT?

Did you face this before?

Thanks

[faizanahmad055]

I do not think this has anything to do with the reloader itself. Reloader just updates the statefulset so based on how the statefulset is configured, it will reload/recreate the pods.

[karl-johan-grahn]

@angelbarrera92 can you provide more detailed reproduction steps?

[cowjen01]

Hello @karl-johan-grahn!

We have a secret called prometheus-nodes-tls with two keys tls.key and tls.crt. We want to watch for changes of this secret (typically the certificate expire and it is automatically refreshed) and restart the prometheus/alertmanager services to load the secret.

Prometheus operator (version 40.x) values.yaml:

prometheus:
  annotations:
    "secret.reloader.stakater.com/reload": "prometheus-nodes-tls"
  serviceMonitor:
    scheme: https
    tlsConfig:
      insecureSkipVerify: true
  prometheusSpec:
    replicas: 2
    scrapeTimeout: 30s
    web:
      tlsConfig:
        clientAuthType: "NoClientCert"
        keySecret:
          name: "prometheus-nodes-tls"
          key: "tls.key"
        cert:
          secret:
            name: "prometheus-nodes-tls"
            key: "tls.crt"
    alertingEndpoints:
      - name: "prometheus-stack-alertmanager"
        namespace: "prometheus-stack"
        port: "http-web"
        scheme: "https"
        tlsConfig:
          caFile: "/mnt/secrets-store/trustedCert"
          serverName: "prometheus-stack-alertmanager"
        apiVersion: v2
        pathPrefix: "/"
    volumes:
      - name: prometheus-secrets-store
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: prometheus-vault
    volumeMounts:
      - name: prometheus-secrets-store
        mountPath: "/mnt/secrets-store"
        readOnly: true
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: openebs-hostpath
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 2Gi

Pod-reloader values.yaml:

reloader:
  ignoreSecrets: false
  ignoreConfigMaps: false
  reloadStrategy: "annotations"
  enableHA: true
  deployment:
    replicas: 3
  serviceMonitor:
    enabled: true
  podDisruptionBudget:
    enabled: true
    minAvailable: 1

The problem is that just one of the 2 replicas is reloaded.

[faizanahmad055]

@cowjen01 Can you please share reloader logs for the restart? Reloader itself doesn't control the reload strategy of the deployment/statefulset. It will just update either the annotation or environment-variable in deployment/statefulset. You can test this by editing the annotation or environment-variable in the deployment/statefulset and see how it restarts the pods.

[yyardeng]

hello, i am running into the same issue - reloader version 0.0.124, prometheus operator version 0.71.2. i have the following annotation on the prometheus statefulset (managed by the prometheus operator) to reload the prometheus pods when the secret containing the thanos object storage information is updated: secret.reloader.stakater.com/reload: "thanos-objstore-config"

but when the secret is updated, only one of the 2 statefulset pods gets reloaded. other statefulsets, not managed by prometheus operator, are reloaded as usual, so i am assuming the problem lies in some incompatibility between the two.

the update strategy defined for the prometheus statefulset is RollingUpdate.

reloader helm values include:

reloader: isArgoRollouts: false isOpenshift: false ignoreSecrets: false ignoreConfigMaps: false reloadOnCreate: false reloadStrategy: default ignoreNamespaces: "" logFormat: "" watchGlobally: true

reloader logs: time="2024-02-18T14:51:01Z" level=info msg="Changes detected in 'thanos-objstore-config' of type 'SECRET' in namespace 'monitoring', Updated 'prometheus-kube-prometheus-stack-prometheus' of type 'StatefulSet' in namespace 'monitoring'"