search

mittwald / kube-httpcache

Varnish Reverse Proxy on Kubernetes
MIT License
294 stars 71 forks source link

Upgraded to Kubernetes 1.28.5 and no endpoints were available for frontendWatch #230

Closed kmcrawford closed 4 months ago

kmcrawford commented 4 months ago

During our last kubernetes upgrade to 1.28.5 all pods were lost and we were unable to get Varnish to start.

The error in the logs were: 

error while establishing watch: Get "[https://10.2.0.1:443/api/v1/namespaces/varnish/endpoints?fieldSelector=metadata.name%3Dvarnish-kube-httpcache&watch=true](https://10.2.0.1/api/v1/namespaces/varnish/endpoints?fieldSelector=metadata.name%3Dvarnish-kube-httpcache&watch=true)": dial tcp 10.2.0.1:443: connect: connection refused
I0414 09:54:05.155610    1 endpoints_watch.go:31] retrying after 30s
W0414 09:54:35.212376    1 endpoints_watch.go:67] service 'varnish-kube-httpcache' has no endpoints

Using the new PDB from this PR will help us in the future with this: https://github.com/mittwald/kube-httpcache/pull/216

How ever to fix the issue I needed to turn off frontendWatch get the cluster to stand up then turn it back on.

helm upgrade  --reuse-values --namespace varnish varnish varnish-deployment-chart --set kube-httpcache.cache.frontendWatch=false

Wait for it to come up, then: 

helm rollback  --namespace varnish varnish

The bigger issue was when we first had the issue we did a helm uninstall and let our CI/CD reinstall it, but it had the same issue since no endpoints were available. Turning off frontendWatch letting it stand up, then turning frontendWatch back on fixed the issue.

kmcrawford commented 4 months ago

I have tried to recreate this in a non-production environment and am unable to reproduce