Statfulset is not restarted after secret changes

[sdwerwed]

metadata.annotations: reloader.stakater.com/auto: "true" is not working, no information why rollout restart not starting. I followed the README info how to set the annotation and looks like it is not working as described. I use the latest reloader helm chart v1.0.22 with default values https://artifacthub.io/packages/helm/stakater/reloader/1.0.22 Statefulset and reloader are in different namespaces

Logs of reloader-reloader:

time="2023-04-26T18:41:05Z" level=info msg="Environment: Kubernetes"
time="2023-04-26T18:41:05Z" level=info msg="Starting Reloader"
time="2023-04-26T18:41:05Z" level=warning msg="KUBERNETES_NAMESPACE is unset, will detect changes in all namespaces."
time="2023-04-26T18:41:05Z" level=info msg="created controller for: configMaps"
time="2023-04-26T18:41:05Z" level=info msg="Starting Controller to watch resource type: configMaps"
time="2023-04-26T18:41:05Z" level=info msg="created controller for: secrets"
time="2023-04-26T18:41:05Z" level=info msg="Starting Controller to watch resource type: secrets"

Statefulset manifest

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: fluentd
  annotations:
    reloader.stakater.com/auto: "true"
spec:
  serviceName: "fluentd-headless"
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: fluentd
      app.kubernetes.io/instance: fluentd
  template:
    metadata:
      labels:
        app: aggregator
        app.kubernetes.io/component: aggregator
        app.kubernetes.io/instance: fluentd
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: fluentd
    spec:
      containers:
      - name: fluentd
        image: <REDACTED_IMAGE>
        ports:
        - containerPort: 24224
          name: tcp
          protocol: TCP
        env:
        - name: FLUENTD_CONF
          value: fluentd.conf
        - name: FLUENTD_OPT
        - name: OPENSEARCH_FLUENTD_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: fluentd-opensearch
        - name: OPENSEARCH_URL
          valueFrom:
            secretKeyRef:
              key: url
              name: fluentd-opensearch
        - name: OPENSEARCH_USERNAME
          valueFrom:
            secretKeyRef:
              key: username
              name: fluentd-opensearch
      ...

[faizanahmad055]

Hi, I assume the secret you are referring to is this: fluentd-opensearch. In this case, you either have to run the reloader in the same namespace or run it in cluster scope by setting the watchGlobally flag to true.

[sdwerwed]

@faizanahmad055 It is globally already, reloader.watchGlobally: true in the default values.yaml

Looks like the issue is on the way the secret is changing, if I run kubectl edit secret fluentd-opensearch reloader works as expected. If the secret is getting deleted and gets applied with new values it is not detecting the changes.

[faizanahmad055]

In that case, you need to set the reloadOnCreate flag.

[sdwerwed]

In that case, you need to set the reloadOnCreate flag.

That works thnx! I think would be nice to set true as the default to avoid such confusion. Many sync controllers might delete and apply instead of edit in place.

[karl-johan-grahn]

@sdwerwed can this be closed?

[sdwerwed]

@sdwerwed can this be closed?

I would recommend to make the default reloadOnCreate flag true, to avoid such kind confusion unless there is a reason to keep it as it is.