search

stakater / Reloader

A Kubernetes controller to watch changes in ConfigMap and Secrets and do rolling upgrades on Pods with their associated Deployment, StatefulSet, DaemonSet and DeploymentConfig – [✩Star] if you're using it!
https://docs.stakater.com/reloader/
Apache License 2.0
7.74k stars 514 forks source link

[ENHANCE] Add conflict retries to handle "the object has been modified; please apply your changes to the latest version and try again" #766

Open sthomson-wyn opened 1 month ago

sthomson-wyn commented 1 month ago

Is your feature request related to a problem? Please describe. Seemingly frequently, I run into "the object has been modified; please apply your changes to the latest version and try again". This is normal and expected in k8s where multiple controllers might make modifications. However, it looks like reloader just goes back to the start and retries everything again.

Describe the solution you'd like Use the build in k8s client retry utility

Describe alternatives you've considered I can't really find a way to stop reloader from failing upon running into this issue

sthomson-wyn commented 1 month ago

I'll make a PR for this

sthomson-wyn commented 1 month ago

I'll also note that I run into this often when doing a rolling upgrade on a large amount of deployments, reloader just doesn't get through them fast enough. Too much time passes between the initial retrieval of the deployments, and the attempted update

MuneebAijaz commented 1 month ago

preferred solution is to send patch requests instead of update requests. We wont need retry logic then, in my experience Ive never seen patch requests clash with latest CR on the cluster. But this retry-all logic might needs a look