Sporadically 502 response

[amarkevich]

What happened:

Elastic Heartbeat within the cluster perform HEAD requests to https://app.corp.com/context/ every 15 seconds. Application runs in the same cluster. Once per 2-3 hrs there is 502 response.

What you expected to happen:

Succeeded response

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.): 1.9.4

Kubernetes version (use kubectl version): 1.26.6

Environment:

• Cloud provider or hardware configuration: AKS
• OS (e.g. from /etc/os-release): AKSCBLMariner
• Kernel (e.g. uname -a): V1-202309.06.0
• Install tools:
• Basic cluster related info:

Custom CoreDNS configuration:

corp.com:53 {
  errors
  cache 30
  forward . %ONPREM_DNS1% %ONPREM_DNS2%
}

app.corp.com resolved to Ingress NGINX Controller ExternalIP

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/session-cookie-name: "JSESSIONID"
spec:
  tls:
  - hosts:
    - app.corp.com
    secretName: server-tls
  rules:
    - host: app.corp.com
      http:
        paths:
          - backend:
              service:
                name: app
                port:
                  number: 443
            path: /
            pathType: Prefix

• How was the ingress-nginx-controller installed: Helm chart 4.8.3

• Current State of the controller:

• Current state of ingress object, if applicable:

• Others:

  [error] 30#30: *19211632 SSL_do_handshake() failed (SSL: error:0A00010B:SSL routines::wrong version number) while SSL handshaking to upstream, client: 127.0.0.1, server: app.corp.com, request: "HEAD /context/ HTTP/1.1", upstream: "https://0.0.0.1:80/context/", host: "app.corp.com"
  [warn] 30#30: *19211632 upstream server temporarily disabled while SSL handshaking to upstream, client: 127.0.0.1, server: app.corp.com, request: "HEAD /context/ HTTP/1.1", upstream: "https://0.0.0.1:80/context/", host: "app.corp.com"
  [warn] 30#30: *19211632 [lua] sticky.lua:193: balance(): failed to get new upstream using upstream nil while connecting to upstream, client: 127.0.0.1, server: app.corp.com, request: "HEAD /context/ HTTP/1.1", upstream: "https://0.0.0.1:80/context/", host: "app.corp.com"
  [warn] 30#30: *19211632 [lua] balancer.lua:335: balance(): no peer was returned, balancer: sticky_balanced while connecting to upstream, client: 127.0.0.1, server: app.corp.com, request: "HEAD /context/ HTTP/1.1", upstream: "https://0.0.0.1:80/context/", host: "app.corp.com"
  [error] 30#30: *19211632 no live upstreams while connecting to upstream, client: 127.0.0.1, server: app.corp.com, request: "HEAD /context/ HTTP/1.1", upstream: "https://upstream_balancer/context/", host: "app.corp.com"

How to reproduce this issue:

Anything else we need to know:

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[tao12345666333]

502 is a backend error, you should check your service

[amarkevich]

502 is a backend error, you should check your service

upstream: "https://0.0.0.1:80/context/" means due to hostname resolution issue request goes to 0.0.0.1 instread of service/pod IP

[longwuyuan]

so hostname resolution issues are better discussed in sig-networking channel of K8S slack maybe. I am curious about the problem in ingress-controller on this issue.

/remove-kind bug

[amarkevich]

@longwuyuan this is a part of NGINX configuration:

http {
    upstream upstream_balancer {
            server 0.0.0.1; # placeholder

same time another application works correctly

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app2
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-buffer-size: "128k"
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/session-cookie-name: "AUTH_SESSION_ID"
spec:
  tls:
  - hosts:
    - app2.corp.com
    secretName: server-tls
  rules:
  - host: app2.corp.com
    http:
      paths:
        - backend:
            service:
              name: app2
              port:
                number: 8080
          path: /
          pathType: Prefix

[github-actions[bot]]

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.