Controller failed to manage multiple ingress changes in parallel

[davinkevin]

What happened:

We have approximately 20 instances of the same app in a cluster. We've deployed a new version of the app for all of them using gitops + fluxcd + helm. FYI, each app has 2 ingress definitions, one for http and another for grpc.

When all the system started to deploy new version of manifests, we had a lot of errors like this:

Helm upgrade failed: cannot patch "xyz-ingress" with kind Ingress:
Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post "https: //ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1/ingresses?timeout=10s": EOF && cannot patch "xyz-ingress-bazel" with kind Ingress: Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post "[https://ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1/ingresses?timeout=10s](https://ingress-nginx-controller-admission.ingress-nginx.svc/networking/v1/ingresses?timeout=10s)": dial tp 10.80.126.183:8443: connect: connection refused

What you expected to happen:

Of course, I would have expected to see no error during the reconciliation process. It's the second time this problem occurs in the cluster, always after an upgrade of an "important" number of instances.

NGINX Ingress controller version:

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.1.3
  Build:         9d3a285f19a704524439c75b947e2189406565ab
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.10

-------------------------------------------------------------------------------

Kubernetes version (use kubectl version): 1.22

Environment:

• Cloud provider or hardware configuration: EKS
• How was the ingress-nginx-controller installed: Using helm:

ingress-nginx           ingress-nginx                       17          2022-05-11 14:45:07.855810865 +0000 UTC deployed    ingress-nginx-4.0.19

values.yaml:

USER-SUPPLIED VALUES:
controller:
  autoscaling:
    enabled: true
    maxReplicas: 6
    minReplicas: 2
    targetCPUUtilizationPercentage: 60
    targetMemoryUtilizationPercentage: 90
  config:
    compute-full-forwarded-for: true
    use-forwarded-headers: true
  metrics:
    enabled: true
    serviceMonitor:
      additionalLabels:
        release: kube-prometheus-stack
      enabled: true
  resources:
    limits:
      cpu: 500m
      memory: 250Mi
    requests:
      cpu: 500m
      memory: 250Mi
  service:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: nlb
  topologySpreadConstraints:
  - labelSelector:
      matchLabels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
    maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
  updateStrategy:
    rollingUpdate:
      maxSurge: 50%
      maxUnavailable: 0
    type: RollingUpdate

• Current State of the controller:

ingressclasses

Name:         nginx
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.1.3
              helm.sh/chart=ingress-nginx-4.0.19
              helm.toolkit.fluxcd.io/name=ingress-nginx
              helm.toolkit.fluxcd.io/namespace=ingress-nginx
Annotations:  meta.helm.sh/release-name: ingress-nginx
              meta.helm.sh/release-namespace: ingress-nginx
Controller:   k8s.io/ingress-nginx
Events:       <none>

Describe pods:

Name:             ingress-nginx-controller-7d9b6dbdc7-k6q5q
Namespace:        ingress-nginx
Priority:         0
Service Account:  ingress-nginx
Node:             ip-10-80-127-245.ec2.internal/10.80.127.245
Start Time:       Tue, 13 Sep 2022 19:24:30 +0200
Labels:           app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=ingress-nginx
                  app.kubernetes.io/name=ingress-nginx
                  pod-template-hash=7d9b6dbdc7
Annotations:      kubernetes.io/psp: eks.privileged
Status:           Running
IP:               10.80.126.183
IPs:
  IP:           10.80.126.183
Controlled By:  ReplicaSet/ingress-nginx-controller-7d9b6dbdc7
Containers:
  controller:
    Container ID:  docker://9ef5452f5dfce1d9f049e2bd3114cd11ff48c67924722de7e139d56aee7391c4
    Image:         k8s.gcr.io/ingress-nginx/controller:v1.1.3@sha256:31f47c1e202b39fadecf822a9b76370bd4baed199a005b3e7d4d1455f4fd3fe2
    Image ID:      docker-pullable://k8s.gcr.io/ingress-nginx/controller@sha256:31f47c1e202b39fadecf822a9b76370bd4baed199a005b3e7d4d1455f4fd3fe2
    Ports:         80/TCP, 443/TCP, 10254/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
      --election-id=ingress-controller-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    State:          Running
      Started:      Wed, 21 Dec 2022 09:37:03 +0100
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Fri, 16 Dec 2022 14:17:45 +0100
      Finished:     Wed, 21 Dec 2022 09:37:01 +0100
    Ready:          True
    Restart Count:  7
    Limits:
      cpu:     500m
      memory:  250Mi
    Requests:
      cpu:      500m
      memory:   250Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-7d9b6dbdc7-k6q5q (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bf6fl (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  kube-api-access-bf6fl:
    Type:                     Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:   3607
    ConfigMapName:            kube-root-ca.crt
    ConfigMapOptional:        <nil>
    DownwardAPI:              true
QoS Class:                    Guaranteed
Node-Selectors:               kubernetes.io/os=linux
Tolerations:                  node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                              node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Topology Spread Constraints:  topology.kubernetes.io/zone:ScheduleAnyway when max skew 1 is exceeded for selector app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Events:                       <none>
---
Name:             ingress-nginx-controller-7d9b6dbdc7-z49df
Namespace:        ingress-nginx
Priority:         0
Service Account:  ingress-nginx
Node:             ip-10-80-92-145.ec2.internal/10.80.92.145
Start Time:       Tue, 13 Sep 2022 17:30:00 +0200
Labels:           app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=ingress-nginx
                  app.kubernetes.io/name=ingress-nginx
                  pod-template-hash=7d9b6dbdc7
Annotations:      kubernetes.io/psp: eks.privileged
Status:           Running
IP:               10.80.73.186
IPs:
  IP:           10.80.73.186
Controlled By:  ReplicaSet/ingress-nginx-controller-7d9b6dbdc7
Containers:
  controller:
    Container ID:  docker://21cf457b0fa9c41fe133f7604c1d7718f49f6bdd90e455a2da6952e799242929
    Image:         k8s.gcr.io/ingress-nginx/controller:v1.1.3@sha256:31f47c1e202b39fadecf822a9b76370bd4baed199a005b3e7d4d1455f4fd3fe2
    Image ID:      docker-pullable://k8s.gcr.io/ingress-nginx/controller@sha256:31f47c1e202b39fadecf822a9b76370bd4baed199a005b3e7d4d1455f4fd3fe2
    Ports:         80/TCP, 443/TCP, 10254/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
      --election-id=ingress-controller-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    State:          Running
      Started:      Wed, 21 Dec 2022 09:38:40 +0100
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Mon, 19 Dec 2022 11:35:15 +0100
      Finished:     Wed, 21 Dec 2022 09:38:39 +0100
    Ready:          True
    Restart Count:  6
    Limits:
      cpu:     500m
      memory:  250Mi
    Requests:
      cpu:      500m
      memory:   250Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-7d9b6dbdc7-z49df (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mrnvz (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  kube-api-access-mrnvz:
    Type:                     Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:   3607
    ConfigMapName:            kube-root-ca.crt
    ConfigMapOptional:        <nil>
    DownwardAPI:              true
QoS Class:                    Guaranteed
Node-Selectors:               kubernetes.io/os=linux
Tolerations:                  node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                              node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Topology Spread Constraints:  topology.kubernetes.io/zone:ScheduleAnyway when max skew 1 is exceeded for selector app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Events:                       <none>

• Others:

Pods consumed during the operation way more than the limit and got killed multiple time by the orchestrator.

[k8s-ci-robot]

@davinkevin: This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[longwuyuan]

/remove-kind bug

Hi, there is some customization visible like resources and topology, in addition to 2 min replicas for the controller. In the context of this, the actual events and reasons are not very explicit, based on just that connection error to port 8443. If you had enough resources available and replicas not being overtly loaded, the validation of the posted json, from the helm-chart, to the api-server should succeed, as you noted.

So more information is needed on multiple objects like the controller pod load, the network load, the kubectl get events -A, the kubectl descrbe of the ingress objects in question, etc etc

[davinkevin]

Some data at the moment I did the upgrade:

CPU Usage:

Memory usage:

Bandwitdh:

The usage seen here are just from the upgrade process, because users are not online when we do the upgrade.

I'm not able to extract events and ingress in question weren't updated because the upgrade process was blocked due to this error, so they stayed the same, without any events associated to them.

[longwuyuan]

Do you see exhausted resources. Not just cpu or memory but in the context of the limits you have set in kubernetes and the limits from OS like filehandles, inodes etc. That connection error is basically saying that a working socket at 8443 before the upgrade went belly up during the upgrade.

Also, it may not hurt to bump the ingress-nginx version. You are on a old one and there are several PRs merged (not meaning that one relates to your issue)

[davinkevin]

Because it's a managed system, we didn't define so much for OS or Kubernetes. But from the monitoring system, we don't have any alert related to file descriptor or inode exhaustion. Additionally, pods are running in specific nodes and has auto-scaling setup which is working well usually when we have a traffic spike.

Here, I think the admission controller just consume too much ram, it got killed by k8s (OOM) and this leads to the connection error… because nothing is able to answer at this specific moment.

Of course, upgrade the ingress is planned, but not before xmas 😇, and I need to test it before, especially to confirm the values.yaml compatibility.

[longwuyuan]

I am not sure if its a new behaviour after changes or not, but OOM killed would be related to 250Mi resource config. So with this data it looks lesss like a problem with the controller and more with your config. Have you tried removing the resource limits temporarily ?

[davinkevin]

It's unfortunate, because we would need to increase the memory available for every instances all the time for a surge happening only during upgrades?

Is there a way to get some kind of control-plane (like a lot of ingress controller), with a pod dedicated to this kind of action (only upgrading nginx rules), with more resources than others? It can prevents any downtime due to manifests publications…

Because in our case, due to all this OOM, we lost the whole traffic in the cluster during the upgrade period, which is really not safe in our case.

We will for sure upgrade the ingress-nginx, but we would really like to make our system more resilient to this kind of problem too, so if you have some advice, it's welcome! 😇

Thanks

[longwuyuan]

I think you have the option to base your decision on data. For example, you can remove the limits and let it run for a few cycles. Then review the usage of resources during normal and eventful use. Once you have that data, you will be in a better position to make informed decision on limits.

[davinkevin]

Of course, we did that for the current setup. We also have a defined HPA enabled to accept the extra load too.

I plan to upgrade the ingress controller this week and check if this fix the problem and evaluate alternative solution after this.

[smlx]

@davinkevin did you find a solution to this?

[davinkevin]

cc @deepy who is more on that subject than me atm.

[deepy]

Upgrading the controller might have helped, we no longer trigger this all the time in the small cluster and despite the big cluster having grown we're seeing less failures (but only once in a blue moon do we get 0 failures) We get a brief moment of throttling so there's still some tweaking to do, but haven't run out of memory in a while despite still using 250Mi as the limit

[longwuyuan]

This error

dial tp 10.80.126.183:8443: connect: connection refused

is a classic root-cause when exactly what happnened is exactly as shown. This is possible momentarily for any pod and not just the ingress-nginx controller pod. It is suspected that wither the load on the host or the load on the network or running out of inodes or similar such transitional states can cause failure to create a new connection to a pod temporarily. The mitigation is a play with Observability stack giving input on the multiple factors including inodes and accordingly adjusting resource availability.

No action item to track here so closing this issue.

/close

[k8s-ci-robot]

@longwuyuan: Closing this issue.

In response to [this](https://github.com/kubernetes/ingress-nginx/issues/9433#issuecomment-2337457332): >This error >``` >dial tp 10.80.126.183:8443: connect: connection refused >``` > >is a classic root-cause when exactly what happnened is exactly as shown. This is possible momentarily for any pod and not just the ingress-nginx controller pod. It is suspected that wither the load on the host or the load on the network or running out of inodes or similar such transitional states can cause failure to create a new connection to a pod temporarily. The mitigation is a play with Observability stack giving input on the multiple factors including inodes and accordingly adjusting resource availability. > >No action item to track here so closing this issue. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.