Some pods of `nginx-ingress` and `node-problem-detector` daemonsets are not up to date

kiwix / operations

Kiwix Kubernetes Cluster

http://charts.k8s.kiwix.org/

5 stars 0 forks source link

Some pods of `nginx-ingress` and `node-problem-detector` daemonsets are not up to date #119

Closed benoit74 closed 9 months ago

benoit74 commented 10 months ago

See the UP-TO-DATE columns below:

Code in https://gist.github.com/benoit74/68d3fdb2bdc77e641cdb218575b417e8 is able to print info to find them:

Daemonset kube-system/nginx-ingress:
- node scw-kiwix-prod-foreign-20abab10b97b430bbe7b571 (generation 1 instead of generation 2)
- node scw-kiwix-prod-foreign-8dd780078597459d8d396a8 (generation 1 instead of generation 2)

Daemonset kube-system/node-problem-detector:
- node scw-kiwix-prod-foreign-8dd780078597459d8d396a8 (generation 1 instead of generation 2)

Should we plan some downtime (especially for nginx ingress), or just do this asap?

benoit74 commented 10 months ago

Btw:

this has been raised by: https://kiwixorg.grafana.net/alerting/grafanacloud-kiwixorg-prom/KubeDaemonSetRolloutStuck/find
we move this check / script in the weekly routine?

rgaudin commented 10 months ago

Please use the node labels next time (storage and services in this case).

what's the impact of this?
When was this new generation created? I believe it was already the case in July when I looked at the bastion issue.
any idea of the reason?

FYI, nginx-ingress is the one thing we need to remove to update k8s and reinstall after (differently)

benoit74 commented 10 months ago

gist updated to also retrieve node label
for nginx, the change between both revisions is minimal ; there is only a new annotation kubectl.kubernetes.io/restartedAt: "2023-04-24T14:05:48Z" ; we can probably assume this has been done at that time ; impact is probably minimal even if I don't know why Scaleway would have like us to restart nginx
for node-problem-detector, the change consists in moving from image: k8s.gcr.io/node-problem-detector/node-problem-detector:v0.8.9 to image: registry.k8s.io/node-problem-detector/node-problem-detector:v0.8.9 ; I did not found when it occured in the metadata ; impact is null (next time the pod restart it will use the updated new image location
the reason is probably that Scaleway did updated the Daemonset definition, but since the update strategy is "OnDelete", the pods have not being updated (it may be obvious to you, I'm not sure this is what you are asking for)

rgaudin commented 10 months ago

OK, thank you

benoit74 commented 9 months ago

I restarted the node-problem-detector since it has no impact. I silenced the nginx alert in Grafana since it might have an impact, won't change anything to the running image and will be renewed sooner or later when upgrading the cluster in any case.