search

stakater / Reloader

A Kubernetes controller to watch changes in ConfigMap and Secrets and do rolling upgrades on Pods with their associated Deployment, StatefulSet, DaemonSet and DeploymentConfig – [✩Star] if you're using it!
https://docs.stakater.com/reloader/
Apache License 2.0
7.45k stars 493 forks source link

Reloader frequent restarts due to failed liveness probes #252

Open messiahUA opened 3 years ago

messiahUA commented 3 years ago

I'm facing an issue with frequent restarts on v0.0.95.

Liveness endpoint (/metrics) response may sometimes be in the range of 1-5 seconds. Of course I can just increase the timeout (which is 1 second by default), but this only hides the problem. I believe there is some inefficiency in the code which affects even /metrics responses.

There are quite a lot of secrets and configmaps in the cluster, so it might put a strain, but there are no cpu and memory limits, so it should just take as much as needed and continue working. I think that /metrics should have its own thread or it would be even better to have a dedicated /readiness and /liveness endpoints which will really check and appropriately report the status of the service. Otherwise its unreliable to run in production especially considering there is no HA, so if pod is restarted I believe it will lose any info about the resources and may miss triggering reloads.

rasheedamir commented 3 years ago

There are quite a lot of secrets and configmaps in the cluster

How many exactly?

messiahUA commented 3 years ago

How many exactly?

979 (configmaps + secrets)

although want to note that only some of the workloads have reloader annotation with only one secret referenced each

rasheedamir commented 3 years ago

@faizanahmad055 can you take a look?

faizanahmad055 commented 3 years ago

@rasheedamir sure will take a look.