search

stakater / Reloader

A Kubernetes controller to watch changes in ConfigMap and Secrets and do rolling upgrades on Pods with their associated Deployment, StatefulSet, DaemonSet and DeploymentConfig – [✩Star] if you're using it!
https://docs.stakater.com/reloader/
Apache License 2.0
7.11k stars 476 forks source link

[BUG] Not all Pods get restartet after Secret Change #701

Open teimyBr opened 1 week ago

teimyBr commented 1 week ago

Describe the bug We have like 20 Deployment in our cluster. all have on the Deployment the Annotations: reloader.stakater.com/auto: true

This all are AKHQ Deployment with Kafka Secrets.

Every 5 Days the Secrets get changed. (at the same time)

Sometimes: Only 17/18 get restartet by reloader.

level=info msg="Changes detected in 'root-ca-cert-truststore' of type 'SECRET' in namespace 'test1', Updated 'akhq' of type 'Deployment' in namespace 'test1'"

Log look like this without any error.

To Reproduce Who can i debug this deeper ?

Expected behavior All pods get restartet

Environment

teimyBr commented 1 week ago

More: He logs level=info msg="Changes detected in 'root-ca-cert-truststore' of type 'SECRET' in namespace 'test1', Updated 'akhq' of type 'Deployment' in namespace 'test1'"

But sometimes the pod doesnt restartet

so 17 or 18 have restartet and 1-2 NOT

But the Log said he restartet all 20.

There is no error log that something has failed.

theBNT commented 1 week ago

Is the restarting done in a fire-and-request approach and if the API server has issues, they are lost or is there some ACK/retry involved?

MuneebAijaz commented 1 week ago

The pods which are not restarted, are they the same ones everytime or random?

teimyBr commented 1 week ago

Very Random. We watch this over the last 3-4 weeks. sometime this is deployment 17 then next time deployment 3. Never ever the same.

Tried it also with latest version there is it still there

aswindevs commented 5 days ago

Facing the same issue. Sometimes not all deployments gets rolled out. We have 56 deployments in total.

MuneebAijaz commented 5 days ago

any more information about what values are being used to install Reloader? and are all deployments backed by any CD tool, if yes, is there a possibility of CD tool and Reloader clashing in updating Deployments?

shameemshah commented 5 days ago

+1 Facing the same issue.

teimyBr commented 4 days ago

We are using reloader helm chart.

Chart.yaml

apiVersion: v2
name: reloader
version: 0.0.0
dependencies:
- name: reloader
  version: 1.0.114
  repository: https://stakater.github.io/stakater-charts

values.yaml 

reloader:
  reloader:
    deployment:
      resources:
        limits:
          cpu: 500m
          memory: 256Mi
        requests:
          cpu: 10m
          memory: 128Mi