search

stakater / Reloader

A Kubernetes controller to watch changes in ConfigMap and Secrets and do rolling upgrades on Pods with their associated Deployment, StatefulSet, DaemonSet and DeploymentConfig – [✩Star] if you're using it!
https://docs.stakater.com/reloader/
Apache License 2.0
7.12k stars 476 forks source link

[BUG]- All Pods restart irrespective of liveness check status or rolling update strategy #664

Closed laerdal-azhar closed 1 month ago

laerdal-azhar commented 1 month ago

I have an app deployed which has both configmap update & some app changes in the same. Below rolling update strategy strategy: rollingUpdate: maxSurge: 1 maxUnavailable: 0 type: RollingUpdate

liveness probe values livenessProbe: httpGet: path: /health port: 3000 initialDelaySeconds: 10 periodSeconds: 5 timeoutSeconds: 5 failureThreshold: 3 successThreshold: 1

Now all pods restart simultaneously when this happens without respecting the max surge & max unavailable strategy, then eventualy when liveness probe fails all pods crashloop & go down causing the app to go down.

Is there any way to fix this?

SheryarButt commented 1 month ago

Hi @laerdal-azhar What reload strategy are you using?

Reloader actually doesn't restart or kill any pod manually, it updates the parent resource that end up updating the pods according to the deployment strategy.

laerdal-azhar commented 1 month ago

this is my rolling update strategy strategy: rollingUpdate: maxSurge: 1 maxUnavailable: 0 type: RollingUpdate

reload annotation is annotations: reloader.stakater.com/auto: "true"

laerdal-azhar commented 1 month ago

any update on this?

MuneebAijaz commented 1 month ago

@laerdal-azhar are you sure that all the pods restart at the same time? by what i know and ive tested, it spins up a 4th pod first, waits for it to be ready, then terminates one of the old pods. and continues this process till every pod is updated.

Regardless, Reloader doesnt affect update strategy of pods, it only edits an ENV in deployment, seeing that kubernetes trigger the update

laerdal-azhar commented 1 month ago

thanks for the update I will try to test again, you can close the issue I will get back if I still face the issue.