liveness probe fails and nginx-ingress-controller restarts on load with unavailable upstreams

[zbitmanis]

Is this a BUG REPORT or FEATURE REQUEST? (choose one):BUG REPORT

NGINX Ingress controller version: 0.16., 0.17.,0.18.*,0.19.0, 0.20.0, 0.21.0

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.7", GitCommit:"0c38c362511b20a098d7cd855f1314dad92c2780", GitTreeState:"clean", BuildDate:"2018-08-20T09:56:31Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.7", GitCommit:"0c38c362511b20a098d7cd855f1314dad92c2780", GitTreeState:"clean", BuildDate:"2018-08-20T09:56:31Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Environment:

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release): NAME="Ubuntu" VERSION="16.04.5 LTS (Xenial Xerus)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 16.04.5 LTS" VERSION_ID="16.04" HOME_URL="http://www.ubuntu.com/" SUPPORT_URL="http://help.ubuntu.com/" BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/" VERSION_CODENAME=xenial UBUNTU_CODENAME=xenial
• Kernel (e.g. uname -a): Linux k8s-master-35100059-0 4.15.0-1021-azure #21~16.04.1-Ubuntu SMP Fri Aug 10 12:36:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Install tools:
• Others:

What happened: On load on ingress which points to unavailable upstreams leads to regular livenes probe fail and regular nginx ingress pod restart initiated by kubelet

ingress controllers are located on dedicated nodes (taint + tolerations )

What you expected to happen: ingrress controler should detect that all upstreams are unresolvable

How to reproduce it (as minimally and precisely as possible):

• Deploy Pod with exposed port

  kind: Deployment 
  metadata:
  name: nginx-unavailable
  namespace: nginx-ingress 
  labels:
  app: nginx-load-generator
  spec:
  replicas: 2
  selector:
  matchLabels:
    run: nginx-unresolver
  template: 
    metadata:
      labels:
        run: nginx-unresolver
        app: nginx-load-generator
    spec:
      containers:
        - name: nginx-unavailable
          image: nginx
          ports: 
             - containerPort: 80
               protocol: TCP

• Deploy Service which which points one to exposed second unexposed port

  apiVersion: v1
  kind: Service 
  metadata:
  name: nginx-lg-svc
  namespace: nginx-ingress 
  labels:
  app: nginx-load-generator
  spec:
  ports:
  - name: http
  port: 8080
  protocol: TCP
  targetPort: 8080
  - name: working
  port: 8081
  protocol: TCP
  targetPort: 80
  selector:
  run: nginx-unresolver
  type: ClusterIP

• Deploy ingress

  apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
  name: nginx-lgunr-foo
  namespace: nginx-ingress 
  spec:
  rules:
  - host: nginx-unresolver-foo.example.org
    http:
      paths:
      - backend:
          serviceName: nginx-lg-svc
          servicePort: 8080

• Generate load from single host using tsenart/vegeta

• contents of bdload.template GET http://nginx-unresolver-foo.example.org

  • simulate load vegeta attack -insecure -rate=1000 -duration=600s -targets=bdload.templat

• controller deployment

  apiVersion: extensions/v1beta1
  kind: Deployment
  metadata:
  labels:
  k8s-app: nginx-ingress
  name: nginx-ingress-controller
  namespace: kube-system
  spec:
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 10
  selector:
  matchLabels:
    k8s-app: nginx-ingress
  strategy:
  rollingUpdate:
    maxSurge: 25%
    maxUnavailable: 25%
  type: RollingUpdate
  template:
  metadata:
    creationTimestamp: null
    labels:
      k8s-app: nginx-ingress
  spec:
    containers:
    - args:
      - /nginx-ingress-controller
      - --default-backend-service=$(POD_NAMESPACE)/nginx-default-http-backend
      - --configmap=$(POD_NAMESPACE)/nginx-configuration
      - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
      - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
      - --publish-service=$(POD_NAMESPACE)/nginx-ingress
      - --annotations-prefix=nginx.ingress.kubernetes.io
      - --sort-backends=true
      - --v=3
      env:
      - name: POD_NAME
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: metadata.name
      - name: POD_NAMESPACE
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: metadata.namespace
      image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.21.0
      imagePullPolicy: IfNotPresent
      livenessProbe:
        failureThreshold: 3
        httpGet:
          path: /healthz
          port: 10254
          scheme: HTTP
        initialDelaySeconds: 10
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 1
      name: nginx-ingress-controller
      ports:
       /..... skip ..../ 

  Anything else we need to know:

• enabled to experimental-allowed-unsafe-sysctls='net.ipv4.,net.core. - minimal mitigation the same result Added to nginx-controller

      security.alpha.kubernetes.io/unsafe-sysctls: net.core.somaxconn=1024,net.ipv4.tcp_max_syn_backlog=1024,net.ipv4.tcp_syn_retries=3,net.ipv4.tcp_synack_retries=2

[aledbf]

@zbitmanis please enable the proxy-next-upstream: http_502 http_503 http_504 option https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#proxy-next-upstream in the configuration configmap. By default there is no retries

[aledbf]

Closing. Please reopen if the suggested change does not work.