yuha0
commented
3 years ago Not sure if it was caused by the base image upgrade (alpine 3.13) but we have seen the same issue as well in our environment after upgrading to 0.44.0. It had happened for 4 times in two days before we downgraded to 0.43.0. We haven't seen a single instance after the downgrade.
All 4 incidents were started with nginx config reload
First we saw very high CPU loads on nodes hosting Nginx pods. Those nodes have 128 cores and the loads got up to 1k-2k:
According to Nginx logs, every one of those load spikes were followed by a config reload:
I0225 17:59:31.825591 6 controller.go:146] "Configuration changes detected, backend reload required"
I0225 17:59:32.429276 6 controller.go:163] "Backend successfully reloaded"
I0225 17:59:32.429503 6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-68974744d7-7v54w", UID:"668e9e5b-37e7-4493-ba37-51b10522c7b0", APIVersion:"v1", ResourceVersion:"1342977486", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
2021/02/25 17:59:36 [alert] 63#63: worker process 35497 exited on signal 11 (core dumped)
2021/02/25 17:59:36 [alert] 63#63: worker process 34639 exited on signal 11 (core dumped)
2021/02/25 17:59:36 [alert] 63#63: worker process 34938 exited on signal 11 (core dumped)
2021/02/25 17:59:36 [alert] 63#63: worker process 34249 exited on signal 11 (core dumped)
2021/02/25 17:59:36 [alert] 63#63: worker process 35369 exited on signal 11 (core dumped)
2021/02/25 17:59:36 [alert] 63#63: worker process 34430 exited on signal 11 (core dumped)
2021/02/25 17:59:36 [alert] 63#63: worker process 35181 exited on signal 11 (core dumped)
2021/02/25 17:59:36 [alert] 63#63: worker process 34138 exited on signal 11 (core dumped)At this point, readiness/liveness probes became a hit or miss, if it was lucky enough to hit a liveness failure threshold, kubelet will restart the pod and the system load would recover back to normal. Otherwise, it could affect other pods on the same node, or even knock down the whole node.
Stono
alfianabdi
ReillyTevera
ghouscht
ReillyBrogan
longwuyuan
LuckySB
bowei
ramijebara
merlindorin
shiraz-zscaler
bnu0
rikatz
jabdoa2
popawu
luryus
TwoStone
gxthrj
chris-ng-scmp
elghazal-a
laghoule
NGINX Ingress controller version: 0.44.0
Kubernetes version (use
kubectl version): 1.18.3 Environment:What happened: We encountered a major production outage a few days ago that was traced back to ingress-nginx. ingress-nginx pod logs were filled with messages like the following:
We discovered that the following message was being printed to the system log as well (timed with the worker exits):
I ultimately identified that whatever was occurring was linked to the version of ingress-nginx we were using and reverted production to 0.43.0 until we could identify the underlying issue.
We have a few other lower-load ingress-nginx deployments that have remained at 0.44.0 and have observed apparently random worker crashes however there are always enough running workers and these are infrequent enough that things seemingly remain stable.
I was able to get a worker coredump from one of those infrequent crashes and the backtrace is as follows:
One of the major differences between 0.43.0 and 0.44.0 is the update to Alpine 3.13, perhaps the version of musl in use is the issue and it would be appropriate to revert that change until Alpine has released a fixed version?
/kind bug