External requests hang when using externalTrafficPolicy: Cluster on 1.8.1 and K8 1.24

david-moxon commented 1 year ago

What happened: Fresh install of ingress-nginx 1.8.1 on a new AKS cluster running K8 1.24.10.

For some reason the ngress-nginx-controller service is set to externalTrafficPolicy: Local by default.

When changing this to Cluster, all external requests to the Load Balancer External IP hang and eventually time out.

What you expected to happen: Firstly, I expected the default externalTrafficPolicy to be set to Cluster.

Secondly, when setting to Cluster I would expect a 404 response from Nginx.

Note that when set to Local I do get a 404 response, as expected.

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.8.1
  Build:         dc88dce9ea5e700f3301d16f971fa17c6cfe757d
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6

-------------------------------------------------------------------------------

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-14T09:53:42Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.10", GitCommit:"5c3a0bd0f4d44c0f0200eef6eed403c48d99fcfa", GitTreeState:"clean", BuildDate:"2023-06-12T18:45:58Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration: Azure / AKS
OS (e.g. from /etc/os-release): linux
Kernel (e.g. uname -a): Linux cc-27812255-68d4df6448-ms8h5 5.4.0-1109-azure #115~18.04.1-Ubuntu SMP Mon May 22 20:06:37 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Install tools:
- Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc. New Kubernetes service cluster created using Azure UI

Basic cluster related info:

kubectl version v1.24.10

kubectl get nodes -o wide

NAME                                STATUS   ROLES   AGE   VERSION    INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
aks-agentpool-21586046-vmss000000   Ready    agent   21h   v1.24.10   10.224.0.4    <none>        Ubuntu 18.04.6 LTS   5.4.0-1111-azure   containerd://1.7.1+azure-1

How was the ingress-nginx-controller installed: Helm

If helm was used then please show output of helm ls -A | grep -i ingress

ingress-nginx                   ingress-nginx   1               2023-07-31 12:09:30.107701909 +0000 UTC deployed        ingress-nginx-4.7.1                                                             1.8.1

If helm was used then please show output of helm -n <ingresscontrollernamepspace> get values <helmreleasename>


helm list -n ingress-nginx
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
ingress-nginx   ingress-nginx   1               2023-07-31 12:09:30.107701909 +0000 UTC deployed        ingress-nginx-4.7.1     1.8.1

helm -n ingress-nginx get values ingress-nginx USER-SUPPLIED VALUES: null

  - If helm was not used, then copy/paste the complete precise command used to install the controller, along with the flags and options used
  - if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances
Only one instance

- **Current State of the controller**:
  - `kubectl describe ingressclasses`

Name: nginx Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.8.1 helm.sh/chart=ingress-nginx-4.7.1 Annotations: meta.helm.sh/release-name: ingress-nginx meta.helm.sh/release-namespace: ingress-nginx Controller: k8s.io/ingress-nginx Events:

  - `kubectl -n <ingresscontrollernamespace> get all -A -o wide`

NAMESPACE NAME default pod/demo-6486d57d96-8sp2r default pod/test-service-6b94b8695f-rkn92 ingress-nginx pod/ingress-nginx-admission-create-kdrh8 ingress-nginx pod/ingress-nginx-admission-patch-s24tx ingress-nginx pod/ingress-nginx-controller-78d54fbd-pjqwb kube-system pod/ama-logs-rs-7c49c45d8f-qpv9f kube-system pod/ama-logs-stfnt kube-system pod/azure-ip-masq-agent-nz4w6 kube-system pod/cloud-node-manager-pdh4d kube-system pod/coredns-785fcf7bdd-8h52x kube-system pod/coredns-785fcf7bdd-gx4cr kube-system pod/coredns-autoscaler-65bb858f95-5k9ff kube-system pod/csi-azuredisk-node-2kkxr kube-system pod/csi-azurefile-node-crmhc kube-system pod/konnectivity-agent-59699fd4df-2v6n9 kube-system pod/konnectivity-agent-59699fd4df-4psjn kube-system pod/kube-proxy-7tvzh kube-system pod/metrics-server-6f8bd495cc-48mbq kube-system pod/metrics-server-6f8bd495cc-zvzj6 READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES 1/1 Running 0 20h 10.244.0.25 aks-agentpool-21586046-vmss000000 1/1 Running 0 20h 10.244.0.18 aks-agentpool-21586046-vmss000000 0/1 Completed 0 20h 10.244.0.26 aks-agentpool-21586046-vmss000000 0/1 Completed 0 20h 10.244.0.27 aks-agentpool-21586046-vmss000000 1/1 Running 0 20h 10.244.0.23 aks-agentpool-21586046-vmss000000 1/1 Running 0 21h 10.244.0.8 aks-agentpool-21586046-vmss000000 2/2 Running 0 21h 10.244.0.2 aks-agentpool-21586046-vmss000000 1/1 Running 0 21h 10.224.0.4 aks-agentpool-21586046-vmss000000 1/1 Running 0 21h 10.224.0.4 aks-agentpool-21586046-vmss000000 1/1 Running 0 21h 10.244.0.10 aks-agentpool-21586046-vmss000000 1/1 Running 0 21h 10.244.0.6 aks-agentpool-21586046-vmss000000 1/1 Running 0 21h 10.244.0.7 aks-agentpool-21586046-vmss000000 3/3 Running 0 21h 10.224.0.4 aks-agentpool-21586046-vmss000000 3/3 Running 0 21h 10.224.0.4 aks-agentpool-21586046-vmss000000 1/1 Running 0 20h 10.244.0.20 aks-agentpool-21586046-vmss000000 1/1 Running 0 20h 10.244.0.21 aks-agentpool-21586046-vmss000000 1/1 Running 0 21h 10.224.0.4 aks-agentpool-21586046-vmss000000 2/2 Running 0 21h 10.244.0.11 aks-agentpool-21586046-vmss000000 2/2 Running 0 21h 10.244.0.12 aks-agentpool-21586046-vmss000000

NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR default service/demo ClusterIP 10.0.41.176 80/TCP 20h app=demo default service/kubernetes ClusterIP 10.0.0.1 443/TCP 21h default service/test-service ClusterIP 10.0.93.32 80/TCP 20h app=test-service ingress-nginx service/ingress-nginx-controller LoadBalancer 10.0.109.108 20.49.158.189 80:30154/TCP,443:31296/TCP 20h app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx ingress-nginx service/ingress-nginx-controller-admission ClusterIP 10.0.188.184 443/TCP 20h app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx kube-system service/kube-dns ClusterIP 10.0.0.10 53/UDP,53/TCP 21h k8s-app=kube-dns kube-system service/metrics-server ClusterIP 10.0.82.122 443/TCP 21h k8s-app=metrics-server

NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE CONTAINERS IMAGES SELECTOR kube-system daemonset.apps/ama-logs 1 1 1 1 1 21h ama-logs,ama-logs-prometheus mcr.microsoft.com/azuremonitor/containerinsights/ciprod:3.1.10,mcr.microsoft.com/azuremonitor/containerinsights/ciprod:3.1.10 component=ama-logs-agent,tier=node kube-system daemonset.apps/ama-logs-windows 0 0 0 0 0 21h ama-logs-windows mcr.microsoft.com/azuremonitor/containerinsights/ciprod:win-3.1.10 component=ama-logs-agent-windows,tier=node-win kube-system daemonset.apps/azure-ip-masq-agent 1 1 1 1 1 21h azure-ip-masq-agent mcr.microsoft.com/aks/ip-masq-agent-v2:v0.1.7 k8s-app=azure-ip-masq-agent,tier=node kube-system daemonset.apps/cloud-node-manager 1 1 1 1 1 21h cloud-node-manager mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.24.21 k8s-app=cloud-node-manager kube-system daemonset.apps/cloud-node-manager-windows 0 0 0 0 0 21h cloud-node-manager mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.24.21 k8s-app=cloud-node-manager-windows kube-system daemonset.apps/csi-azuredisk-node 1 1 1 1 1 21h liveness-probe,node-driver-registrar,azuredisk mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.10.0,mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.8.0,mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.26.5 app=csi-azuredisk-node kube-system daemonset.apps/csi-azuredisk-node-win 0 0 0 0 0 21h liveness-probe,node-driver-registrar,azuredisk mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.10.0,mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.8.0,mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.26.5 app=csi-azuredisk-node-win kube-system daemonset.apps/csi-azurefile-node 1 1 1 1 1 21h liveness-probe,node-driver-registrar,azurefile mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.10.0,mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.8.0,mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.24.3 app=csi-azurefile-node kube-system daemonset.apps/csi-azurefile-node-win 0 0 0 0 0 21h liveness-probe,node-driver-registrar,azurefile mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.10.0,mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.8.0,mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.24.3 app=csi-azurefile-node-win kube-system daemonset.apps/kube-proxy 1 1 1 1 1 21h kube-proxy mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.24.10-hotfix.20230612 component=kube-proxy,tier=node

NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR default deployment.apps/demo 1/1 1 1 20h httpd httpd app=demo default deployment.apps/test-service 1/1 1 1 20h nginx nginx app=test-service ingress-nginx deployment.apps/ingress-nginx-controller 1/1 1 1 20h controller registry.k8s.io/ingress-nginx/controller:v1.8.1@sha256:e5c4824e7375fcf2a393e1c03c293b69759af37a9ca6abdb91b13d78a93da8bd app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx kube-system deployment.apps/ama-logs-rs 1/1 1 1 21h ama-logs mcr.microsoft.com/azuremonitor/containerinsights/ciprod:3.1.10 rsName=ama-logs-rs kube-system deployment.apps/coredns 2/2 2 2 21h coredns mcr.microsoft.com/oss/kubernetes/coredns:v1.9.4 k8s-app=kube-dns,version=v20 kube-system deployment.apps/coredns-autoscaler 1/1 1 1 21h autoscaler mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:v1.8.8 k8s-app=coredns-autoscaler kube-system deployment.apps/konnectivity-agent 2/2 2 2 21h konnectivity-agent mcr.microsoft.com/oss/kubernetes/apiserver-network-proxy/agent:v0.0.33-hotfix.20221110 app=konnectivity-agent kube-system deployment.apps/metrics-server 2/2 2 2 21h metrics-server-vpa,metrics-server mcr.microsoft.com/oss/kubernetes/autoscaler/addon-resizer:1.8.19,mcr.microsoft.com/oss/kubernetes/metrics-server:v0.6.3 k8s-app=metrics-server

NAMESPACE NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR default replicaset.apps/demo-6486d57d96 1 1 1 20h httpd httpd app=demo,pod-template-hash=6486d57d96 default replicaset.apps/test-service-6b94b8695f 1 1 1 20h nginx nginx app=test-service,pod-template-hash=6b94b8695f ingress-nginx replicaset.apps/ingress-nginx-controller-78d54fbd 1 1 1 20h controller registry.k8s.io/ingress-nginx/controller:v1.8.1@sha256:e5c4824e7375fcf2a393e1c03c293b69759af37a9ca6abdb91b13d78a93da8bd app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=78d54fbd kube-system replicaset.apps/ama-logs-rs-7c49c45d8f 1 1 1 21h ama-logs mcr.microsoft.com/azuremonitor/containerinsights/ciprod:3.1.10 pod-template-hash=7c49c45d8f,rsName=ama-logs-rs kube-system replicaset.apps/coredns-785fcf7bdd 2 2 2 21h coredns mcr.microsoft.com/oss/kubernetes/coredns:v1.9.4 k8s-app=kube-dns,pod-template-hash=785fcf7bdd,version=v20 kube-system replicaset.apps/coredns-autoscaler-65bb858f95 1 1 1 21h autoscaler mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:v1.8.8 k8s-app=coredns-autoscaler,pod-template-hash=65bb858f95 kube-system replicaset.apps/konnectivity-agent-58866bc44d 0 0 0 21h konnectivity-agent mcr.microsoft.com/oss/kubernetes/apiserver-network-proxy/agent:v0.0.33-hotfix.20221110 app=konnectivity-agent,pod-template-hash=58866bc44d kube-system replicaset.apps/konnectivity-agent-59699fd4df 2 2 2 20h konnectivity-agent mcr.microsoft.com/oss/kubernetes/apiserver-network-proxy/agent:v0.0.33-hotfix.20221110 app=konnectivity-agent,pod-template-hash=59699fd4df kube-system replicaset.apps/metrics-server-54c58754d5 0 0 0 21h metrics-server-vpa,metrics-server mcr.microsoft.com/oss/kubernetes/autoscaler/addon-resizer:1.8.19,mcr.microsoft.com/oss/kubernetes/metrics-server:v0.6.3 k8s-app=metrics-server,pod-template-hash=54c58754d5 kube-system replicaset.apps/metrics-server-6f8bd495cc 2 2 2 21h metrics-server-vpa,metrics-server mcr.microsoft.com/oss/kubernetes/autoscaler/addon-resizer:1.8.19,mcr.microsoft.com/oss/kubernetes/metrics-server:v0.6.3 k8s-app=metrics-server,pod-template-hash=6f8bd495cc

NAMESPACE NAME COMPLETIONS DURATION AGE CONTAINERS IMAGES SELECTOR ingress-nginx job.batch/ingress-nginx-admission-create 1/1 4s 20h create registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230407@sha256:543c40fd093964bc9ab509d3e791f9989963021f1e9e4c9c7b6700b02bfb227b controller-uid=68c82812-4edb-4155-90b2-0a273536d6dc ingress-nginx job.batch/ingress-nginx-admission-patch 1/1 4s 20h patch registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230407@sha256:543c40fd093964bc9ab509d3e791f9989963021f1e9e4c9c7b6700b02bfb227b controller-uid=0bad471d-d991-47ff-8eb2-95cd92814f76

  - `kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>`

Name: ingress-nginx-controller-78d54fbd-pjqwb Namespace: ingress-nginx Priority: 0 Service Account: ingress-nginx Node: aks-agentpool-21586046-vmss000000/10.224.0.4 Start Time: Mon, 31 Jul 2023 12:09:35 +0000 Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.8.1 helm.sh/chart=ingress-nginx-4.7.1 pod-template-hash=78d54fbd Annotations: Status: Running IP: 10.244.0.23 IPs: IP: 10.244.0.23 Controlled By: ReplicaSet/ingress-nginx-controller-78d54fbd Containers: controller: Container ID: containerd://58f33c602d187768c7a2b1a3ed485a7fd8e107efeee04c922b497d845884c512 Image: registry.k8s.io/ingress-nginx/controller:v1.8.1@sha256:e5c4824e7375fcf2a393e1c03c293b69759af37a9ca6abdb91b13d78a93da8bd Image ID: registry.k8s.io/ingress-nginx/controller@sha256:e5c4824e7375fcf2a393e1c03c293b69759af37a9ca6abdb91b13d78a93da8bd Ports: 80/TCP, 443/TCP, 8443/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP Args: /nginx-ingress-controller --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller --election-id=ingress-nginx-leader --controller-class=k8s.io/ingress-nginx --ingress-class=nginx --configmap=$(POD_NAMESPACE)/ingress-nginx-controller --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key State: Running Started: Mon, 31 Jul 2023 12:09:36 +0000 Ready: True Restart Count: 0 Requests: cpu: 100m memory: 90Mi Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5 Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3 Environment: POD_NAME: ingress-nginx-controller-78d54fbd-pjqwb (v1:metadata.name) POD_NAMESPACE: ingress-nginx (v1:metadata.namespace) LD_PRELOAD: /usr/local/lib/libmimalloc.so Mounts: /usr/local/certificates/ from webhook-cert (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-49fkk (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: webhook-cert: Type: Secret (a volume populated by a Secret) SecretName: ingress-nginx-admission Optional: false kube-api-access-49fkk: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: kubernetes.io/os=linux Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events:

  - `kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>`

Name: ingress-nginx-controller Namespace: ingress-nginx Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.8.1 helm.sh/chart=ingress-nginx-4.7.1 Annotations: meta.helm.sh/release-name: ingress-nginx meta.helm.sh/release-namespace: ingress-nginx Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx Type: LoadBalancer IP Family Policy: SingleStack IP Families: IPv4 IP: 10.0.109.108 IPs: 10.0.109.108 LoadBalancer Ingress: 20.49.158.189 Port: http 80/TCP TargetPort: http/TCP NodePort: http 30154/TCP Endpoints: 10.244.0.23:80 Port: https 443/TCP TargetPort: https/TCP NodePort: https 31296/TCP Endpoints: 10.244.0.23:443 Session Affinity: None External Traffic Policy: Local HealthCheck NodePort: 31019 Events: Type Reason Age From Message

Normal ExternalTrafficPolicy 46m service-controller Local -> Cluster Normal EnsuringLoadBalancer 45m (x3 over 17h) service-controller Ensuring load balancer Normal ExternalTrafficPolicy 45m service-controller Cluster -> Local Normal EnsuredLoadBalancer 45m (x3 over 17h) service-controller Ensured load balancer


- **Current state of ingress object, if applicable**:
  - `kubectl -n <appnnamespace> get all,ing -o wide`

NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES default pod/demo-6486d57d96-8sp2r 1/1 Running 0 20h 10.244.0.25 aks-agentpool-21586046-vmss000000 default pod/test-service-6b94b8695f-rkn92 1/1 Running 0 21h 10.244.0.18 aks-agentpool-21586046-vmss000000 ingress-nginx pod/ingress-nginx-admission-create-kdrh8 0/1 Completed 0 20h 10.244.0.26 aks-agentpool-21586046-vmss000000 ingress-nginx pod/ingress-nginx-admission-patch-s24tx 0/1 Completed 0 20h 10.244.0.27 aks-agentpool-21586046-vmss000000 ingress-nginx pod/ingress-nginx-controller-78d54fbd-pjqwb 1/1 Running 0 20h 10.244.0.23 aks-agentpool-21586046-vmss000000 kube-system pod/ama-logs-rs-7c49c45d8f-qpv9f 1/1 Running 0 22h 10.244.0.8 aks-agentpool-21586046-vmss000000 kube-system pod/ama-logs-stfnt 2/2 Running 0 22h 10.244.0.2 aks-agentpool-21586046-vmss000000 kube-system pod/azure-ip-masq-agent-nz4w6 1/1 Running 0 22h 10.224.0.4 aks-agentpool-21586046-vmss000000 kube-system pod/cloud-node-manager-pdh4d 1/1 Running 0 22h 10.224.0.4 aks-agentpool-21586046-vmss000000 kube-system pod/coredns-785fcf7bdd-8h52x 1/1 Running 0 21h 10.244.0.10 aks-agentpool-21586046-vmss000000 kube-system pod/coredns-785fcf7bdd-gx4cr 1/1 Running 0 22h 10.244.0.6 aks-agentpool-21586046-vmss000000 kube-system pod/coredns-autoscaler-65bb858f95-5k9ff 1/1 Running 0 22h 10.244.0.7 aks-agentpool-21586046-vmss000000 kube-system pod/csi-azuredisk-node-2kkxr 3/3 Running 0 22h 10.224.0.4 aks-agentpool-21586046-vmss000000 kube-system pod/csi-azurefile-node-crmhc 3/3 Running 0 22h 10.224.0.4 aks-agentpool-21586046-vmss000000 kube-system pod/konnectivity-agent-59699fd4df-2v6n9 1/1 Running 0 21h 10.244.0.20 aks-agentpool-21586046-vmss000000 kube-system pod/konnectivity-agent-59699fd4df-4psjn 1/1 Running 0 21h 10.244.0.21 aks-agentpool-21586046-vmss000000 kube-system pod/kube-proxy-7tvzh 1/1 Running 0 22h 10.224.0.4 aks-agentpool-21586046-vmss000000 kube-system pod/metrics-server-6f8bd495cc-48mbq 2/2 Running 0 21h 10.244.0.11 aks-agentpool-21586046-vmss000000 kube-system pod/metrics-server-6f8bd495cc-zvzj6 2/2 Running 0 21h 10.244.0.12 aks-agentpool-21586046-vmss000000

NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR default service/demo ClusterIP 10.0.41.176 80/TCP 20h app=demo default service/kubernetes ClusterIP 10.0.0.1 443/TCP 22h default service/test-service ClusterIP 10.0.93.32 80/TCP 21h app=test-service ingress-nginx service/ingress-nginx-controller LoadBalancer 10.0.109.108 20.49.158.189 80:30154/TCP,443:31296/TCP 20h app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx ingress-nginx service/ingress-nginx-controller-admission ClusterIP 10.0.188.184 443/TCP 20h app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx kube-system service/kube-dns ClusterIP 10.0.0.10 53/UDP,53/TCP 22h k8s-app=kube-dns kube-system service/metrics-server ClusterIP 10.0.82.122 443/TCP 22h k8s-app=metrics-server

NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE CONTAINERS IMAGES SELECTOR kube-system daemonset.apps/ama-logs 1 1 1 1 1 22h ama-logs,ama-logs-prometheus mcr.microsoft.com/azuremonitor/containerinsights/ciprod:3.1.10,mcr.microsoft.com/azuremonitor/containerinsights/ciprod:3.1.10 component=ama-logs-agent,tier=node kube-system daemonset.apps/ama-logs-windows 0 0 0 0 0 22h ama-logs-windows mcr.microsoft.com/azuremonitor/containerinsights/ciprod:win-3.1.10 component=ama-logs-agent-windows,tier=node-win kube-system daemonset.apps/azure-ip-masq-agent 1 1 1 1 1 22h azure-ip-masq-agent mcr.microsoft.com/aks/ip-masq-agent-v2:v0.1.7 k8s-app=azure-ip-masq-agent,tier=node kube-system daemonset.apps/cloud-node-manager 1 1 1 1 1 22h cloud-node-manager mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.24.21 k8s-app=cloud-node-manager kube-system daemonset.apps/cloud-node-manager-windows 0 0 0 0 0 22h cloud-node-manager mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.24.21 k8s-app=cloud-node-manager-windows kube-system daemonset.apps/csi-azuredisk-node 1 1 1 1 1 22h liveness-probe,node-driver-registrar,azuredisk mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.10.0,mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.8.0,mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.26.5 app=csi-azuredisk-node kube-system daemonset.apps/csi-azuredisk-node-win 0 0 0 0 0 22h liveness-probe,node-driver-registrar,azuredisk mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.10.0,mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.8.0,mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.26.5 app=csi-azuredisk-node-win kube-system daemonset.apps/csi-azurefile-node 1 1 1 1 1 22h liveness-probe,node-driver-registrar,azurefile mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.10.0,mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.8.0,mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.24.3 app=csi-azurefile-node kube-system daemonset.apps/csi-azurefile-node-win 0 0 0 0 0 22h liveness-probe,node-driver-registrar,azurefile mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.10.0,mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.8.0,mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.24.3 app=csi-azurefile-node-win kube-system daemonset.apps/kube-proxy 1 1 1 1 1 22h kube-proxy mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.24.10-hotfix.20230612 component=kube-proxy,tier=node

NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR default deployment.apps/demo 1/1 1 1 20h httpd httpd app=demo default deployment.apps/test-service 1/1 1 1 21h nginx nginx app=test-service ingress-nginx deployment.apps/ingress-nginx-controller 1/1 1 1 20h controller registry.k8s.io/ingress-nginx/controller:v1.8.1@sha256:e5c4824e7375fcf2a393e1c03c293b69759af37a9ca6abdb91b13d78a93da8bd app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx kube-system deployment.apps/ama-logs-rs 1/1 1 1 22h ama-logs mcr.microsoft.com/azuremonitor/containerinsights/ciprod:3.1.10 rsName=ama-logs-rs kube-system deployment.apps/coredns 2/2 2 2 22h coredns mcr.microsoft.com/oss/kubernetes/coredns:v1.9.4 k8s-app=kube-dns,version=v20 kube-system deployment.apps/coredns-autoscaler 1/1 1 1 22h autoscaler mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:v1.8.8 k8s-app=coredns-autoscaler kube-system deployment.apps/konnectivity-agent 2/2 2 2 22h konnectivity-agent mcr.microsoft.com/oss/kubernetes/apiserver-network-proxy/agent:v0.0.33-hotfix.20221110 app=konnectivity-agent kube-system deployment.apps/metrics-server 2/2 2 2 22h metrics-server-vpa,metrics-server mcr.microsoft.com/oss/kubernetes/autoscaler/addon-resizer:1.8.19,mcr.microsoft.com/oss/kubernetes/metrics-server:v0.6.3 k8s-app=metrics-server

NAMESPACE NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR default replicaset.apps/demo-6486d57d96 1 1 1 20h httpd httpd app=demo,pod-template-hash=6486d57d96 default replicaset.apps/test-service-6b94b8695f 1 1 1 21h nginx nginx app=test-service,pod-template-hash=6b94b8695f ingress-nginx replicaset.apps/ingress-nginx-controller-78d54fbd 1 1 1 20h controller registry.k8s.io/ingress-nginx/controller:v1.8.1@sha256:e5c4824e7375fcf2a393e1c03c293b69759af37a9ca6abdb91b13d78a93da8bd app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=78d54fbd kube-system replicaset.apps/ama-logs-rs-7c49c45d8f 1 1 1 22h ama-logs mcr.microsoft.com/azuremonitor/containerinsights/ciprod:3.1.10 pod-template-hash=7c49c45d8f,rsName=ama-logs-rs kube-system replicaset.apps/coredns-785fcf7bdd 2 2 2 22h coredns mcr.microsoft.com/oss/kubernetes/coredns:v1.9.4 k8s-app=kube-dns,pod-template-hash=785fcf7bdd,version=v20 kube-system replicaset.apps/coredns-autoscaler-65bb858f95 1 1 1 22h autoscaler mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:v1.8.8 k8s-app=coredns-autoscaler,pod-template-hash=65bb858f95 kube-system replicaset.apps/konnectivity-agent-58866bc44d 0 0 0 22h konnectivity-agent mcr.microsoft.com/oss/kubernetes/apiserver-network-proxy/agent:v0.0.33-hotfix.20221110 app=konnectivity-agent,pod-template-hash=58866bc44d kube-system replicaset.apps/konnectivity-agent-59699fd4df 2 2 2 21h konnectivity-agent mcr.microsoft.com/oss/kubernetes/apiserver-network-proxy/agent:v0.0.33-hotfix.20221110 app=konnectivity-agent,pod-template-hash=59699fd4df kube-system replicaset.apps/metrics-server-54c58754d5 0 0 0 22h metrics-server-vpa,metrics-server mcr.microsoft.com/oss/kubernetes/autoscaler/addon-resizer:1.8.19,mcr.microsoft.com/oss/kubernetes/metrics-server:v0.6.3 k8s-app=metrics-server,pod-template-hash=54c58754d5 kube-system replicaset.apps/metrics-server-6f8bd495cc 2 2 2 21h metrics-server-vpa,metrics-server mcr.microsoft.com/oss/kubernetes/autoscaler/addon-resizer:1.8.19,mcr.microsoft.com/oss/kubernetes/metrics-server:v0.6.3 k8s-app=metrics-server,pod-template-hash=6f8bd495cc

NAMESPACE NAME COMPLETIONS DURATION AGE CONTAINERS IMAGES SELECTOR ingress-nginx job.batch/ingress-nginx-admission-create 1/1 4s 20h create registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230407@sha256:543c40fd093964bc9ab509d3e791f9989963021f1e9e4c9c7b6700b02bfb227b controller-uid=68c82812-4edb-4155-90b2-0a273536d6dc ingress-nginx job.batch/ingress-nginx-admission-patch 1/1 4s 20h patch registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230407@sha256:543c40fd093964bc9ab509d3e791f9989963021f1e9e4c9c7b6700b02bfb227b controller-uid=0bad471d-d991-47ff-8eb2-95cd92814f76

  - `kubectl -n <appnamespace> describe ing <ingressname>`

Name: demo Labels: Namespace: default Address: 20.49.158.189 Ingress Class: nginx Default backend: Rules: Host Path Backends

test.dev.claritygo.com
/ demo:80 (10.244.0.25:80) Annotations: Events: Type Reason Age From Message

Normal Sync 9m47s (x2 over 10m) nginx-ingress-controller Scheduled for sync

  - If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag

curl -Iv 20.49.158.189:80

Trying 20.49.158.189:80...
Connected to 20.49.158.189 (20.49.158.189) port 80 (#0)

HEAD / HTTP/1.1 Host: 20.49.158.189 User-Agent: curl/7.86.0 Accept: /
Recv failure: Connection reset by peer
Closing connection 0 curl: (56) Recv failure: Connection reset by peer
Others:
- Any other related information like ;
- copy/paste of the snippet (if applicable)
- kubectl describe ... of any custom configmap(s) created and in use
- Any other related information that may help

How to reproduce this issue: Note I have not had time to reproduce this on Minikube or similar.

These steps are taken from the Quick Start guide here

Install the ingress controller

Install ingress-nginx using Helm

helm upgrade --install ingress-nginx ingress-nginx \
  --repo https://kubernetes.github.io/ingress-nginx \
  --namespace ingress-nginx --create-namespace

Wait for pods to be ready.

kubectl get pods --namespace=ingress-nginx

Change the externalTrafficPolicy to Cluster

Change the ingress-nginx-controller to use externalTrafficPolicy: Cluster

kubectl -n ingress-nginx patch service ingress-nginx-controller -p '{"spec":{"externalTrafficPolicy":"Cluster"}}'

At this stage you can run a curl on the external IP of the ingress-nginx-controller service.

curl -Iv <external_ip>:80
*   Trying 20.49.158.189:80...
* Connected to 20.49.158.189 (20.49.158.189) port 80 (#0)
> HEAD / HTTP/1.1
> Host: 20.49.158.189
> User-Agent: curl/7.86.0
> Accept: */*
> 
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

Create ingress

Create a demo app

kubectl create deployment demo --image=httpd --port=80
kubectl expose deployment demo

Create the ingress (replacing www.demo.io with your DNS)

kubectl create ingress demo --class=nginx \
  --rule="www.demo.io/*=demo:80"

Add a DNS entry for the above address pointing to the external IP of the ingress-nginx-controller service.

Try accessing this URL in a browser. The request hangs and eventually times out.

Change the externalTrafficPolicy to Local

Change the ingress-nginx-controller to use externalTrafficPolicy: Local

kubectl -n ingress-nginx patch service ingress-nginx-controller -p '{"spec":{"externalTrafficPolicy":"Local"}}'

At this stage you can run a curl on the external IP of the ingress-nginx-controller service.

curl -Iv <external_ip>:80
*   Trying 20.49.158.189:80...
* Connected to 20.49.158.189 (20.49.158.189) port 80 (#0)
> HEAD / HTTP/1.1
> Host: 20.49.158.189
> User-Agent: curl/7.86.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
HTTP/1.1 404 Not Found
< Date: Tue, 01 Aug 2023 09:01:57 GMT
Date: Tue, 01 Aug 2023 09:01:57 GMT
< Content-Type: text/html
Content-Type: text/html
< Content-Length: 146
Content-Length: 146
< Connection: keep-alive
Connection: keep-alive

< 
* Connection #0 to host 20.49.158.189 left intact

k8s-ci-robot commented 1 year ago

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

strongjz commented 1 year ago

/assign @rikatz

github-actions[bot] commented 1 year ago

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.

longwuyuan commented 1 month ago

As there is no kubectl logs for the controller pod, the suspect here is required ports being closed between nodes or some other packet filter.

No action item here for the project since its been a year of idle on this issue. Hence can not track any action item by keeping it open. So closing the issue.

/close

k8s-ci-robot commented 1 month ago

@longwuyuan: Closing this issue.

In response to [this](https://github.com/kubernetes/ingress-nginx/issues/10262#issuecomment-2349779640): >As there is no `kubectl logs` for the controller pod, the suspect here is required ports being closed between nodes or some other packet filter. > >No action item here for the project since its been a year of idle on this issue. Hence can not track any action item by keeping it open. So closing the issue. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

kubernetes / ingress-nginx