Open hassenius opened 1 year ago
@hassenius: This issue is currently awaiting triage.
If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted
label and provide further guidance.
The triage/accepted
label can be added by org members by writing /triage accepted
in a comment.
hi @hassenius ,
Thanks for reporting this. I think not being able to reproduce at will means you will have to gather info on a bunch of host related resources ranging from filehandles, inodes, i/o, hardware status with dmesg etc etc. And also post kubectl get events -A
taken at the time this occurs.
Another observation is that while several people run the service of the controller as type nodePort, the official kubernetes.io docs itself does not recommend nodePort for production use. At this stage there is no info from your cluster to base this statement on, but the relevance is host/other resources you could be exhausting (and maybe contributed by using nodePort)
Thanks for the attention @longwuyuan .
I think there's elements in this
1) What is the root cause of this state occurring? (Since we have not identified any good way of reproducing this at will, it's quite difficult to do the forensic work, and any suggestions on what to capture when the situation does arise is very welcome. Maybe I'm best opening a separate issue for this?)
2) Why does the liveness probe not pick up the problem, letting kubelet restart the pod (which does resolve the issue when it occurs) For the purpose of this we have managed to isolate two ingress pods in a cluster that is stuck in this state (after being identified, we removed the owner reference, and relabelled them so that traffic is not sent to the broken pods).
I think its not easy to get specific. But general direction is things like ;
Most significant factor here is nothing is known about your environment so can not generalize on anything. Everything from your hardware to your Virtualization to the linux kernel right down to any K8S or controller related config to the traffic can be at play. I already gave you one example. If you have a nodePort service for the controller, do you even know how many connections are related to that Port on the host TCP/IP stack etc etc
Thanks,
First to clarify the comment about the NodePort recommendation. Do you mean not to use hostport or not to use nodeport? We use nodeport, and I'm not sure if we have many good alternatives, as the connection between the external loadbalancer and the ingress is orchestrated by an external tool before the cluster is created. I thought this was a pretty common configuration as mentioned in the ingress doc
An Ingress does not expose arbitrary ports or protocols. Exposing services other than HTTP and HTTPS to the internet typically uses a service of type [Service.Type=NodePort](https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport) or [Service.Type=LoadBalancer](https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer).
For the troubleshooting and further details on what we do see in the troubled pods ingress pods:
netstat -n | grep CLOSE_WAIT | wc -l
11820
netstat -n | grep -v CLOSE_WAIT | wc -l 81
(This seems to be the biggest symptom. Why all these connections stuck in CLOSE_WAIT forever?)
2. nginx is not able to accept connections to port 80
bash-5.1$ curl -m 5 -vvv http://localhost
GET / HTTP/1.1 Host: localhost User-Agent: curl/7.79.1 Accept: /
Nothing is registered and logged when the pod is in the problem state
but health check seems ok (this is the same from within the pod, or from any other cluster node or pod)
bash-5.1$ curl -m 5 -vvv http://localhost:10254/healthz
* Trying 127.0.0.1:10254...
* Connected to localhost (127.0.0.1) port 10254 (#0)
> GET /healthz HTTP/1.1
> Host: localhost:10254
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Mon, 19 Sep 2022 18:21:06 GMT
< Content-Length: 2
<
* Connection #0 to host localhost left intact
*
It doesn't seem to be constrained in any particular way
sysctl -a | grep conntrack_max
net.netfilter.nf_conntrack_max = 3145728
cat /proc/sys/fs/file-max
59185830
ulimit -a
real-time non-blocking time (microseconds, -R) unlimited
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 2314720
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1048576
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 4194304
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
It seems to happen on different hosts, and the hosts it happens on does not have any kind of other problem pods that we can see, outside the nginx pods.
My suggestion is ;
/remove-kind bug /kind support
Observation based on your posts ;
Per @rikatz "https://github.com/kubernetes/ingress-nginx/blob/main/rootfs/etc/nginx/template/nginx.tmpl#L681 we need to check this lua code and this part of the code if it is locking connections"
Sorry to be slow circling back to this. The issue still periodically bites us, but really, this particular issue report is really more about the health check not picking up the accurate health of the pod as one would expect, rather than the root cause of why it happens.
Here is an example
bash-5.1$ curl http://localhost:10254/healthz
okbash-5.1$ curl http://localhost:10254/healthz?verbose
[+]ping ok
[+]nginx-ingress-controller ok
healthz check passed
bash-5.1$ curl -m 1 http://localhost/healthz
curl: (28) Operation timed out after 1000 milliseconds with 0 bytes received
What happened: We currently have intermittent issues where ingress controllers stop accepting new connections (on 80 and 443 ports), so the controllers are by definition not in a healthy state, but the healtz enpoint works just fine.
if I exec into a running nginx controller, you can see this very clearly
What you expected to happen: Health checks to fail when pod is not healthy
NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):
Kubernetes version (use
kubectl version
):Environment:
Cloud provider or hardware configuration: bare metal, on-prem
OS (e.g. from /etc/os-release): RHEL 8.6
Kernel (e.g.
uname -a
): 4.18.0-372.9.1.el8.x86_64Install tools: kubespray
Current state of ingress object, if applicable:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/ingress-nginx-controller NodePort 10.9.252.114 80:32080/TCP,443:32443/TCP 9d app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
service/ingress-nginx-controller-admission ClusterIP 10.9.244.170 443/TCP 9d app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
service/ingress-nginx-controller-metrics ClusterIP 10.9.253.239 10254/TCP 9d app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR deployment.apps/ingress-nginx-controller 3/3 3 3 9d controller docker-local./ingress-nginx/controller:v1.1.2@sha256:6a18680809f9bdf7bba4092cede2b5f3ee3566fb65c1bf4cfce328c5ee94bac7 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR replicaset.apps/ingress-nginx-controller-5c5cfcb869 3 3 3 9d controller docker-local./ingress-nginx/controller:v1.1.2@sha256:6a18680809f9bdf7bba4092cede2b5f3ee3566fb65c1bf4cfce328c5ee94bac7 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=5c5cfcb869
netstat -a -n | grep CLOSE_WAIT | wc -l 11735