port 8441 which is enabled nginx ingress deployment and service is not accessible when we hit from upstream

[arunlakshmananl]

What happened:

I enabled the tcp port 8440 in nginx ingress controller deployment and service and try to consume from upstream system using hostname which mapped into ingress load balancer ip. but i am getting connection timed error. Request is not even coming to ingress pods since in ingress pod logs its not showing up

What you expected to happen:

connection should happen fine and request form upstream has to process without any issues

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.): 1.3.0

Kubernetes version (use kubectl version): 1.24.9

Environment: Prod

• Cloud provider or hardware configuration: AZURE

• OS (e.g. from /etc/os-release): Ubuntu

• Kernel (e.g. uname -a):

• Install tools:

  • Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc. cluster is created in Azure using terraform

• Basic cluster related info:

  • kubectl version $ kubectl version WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.1", GitCommit:"8f94681cd294aa8cfd3407b8191f6c70214973a4", GitTreeState:"clean", BuildDate:"2023-01-18T15:58:16Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"windows/amd64"} Kustomize Version: v4.5.7 Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.9", GitCommit:"57fbbcc2804848b95cad5519f5ec9d6355430db9", GitTreeState:"clean", BuildDate:"2023-02-08T17:22:38Z", GoVersion:"go1.18.9", Compiler:"gc", Platform:"linux/amd64"} WARNING: version difference between client (1.26) and server (1.24) exceeds the supported minor version skew of +/-1

  • kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME aks-eupgonline-22130134-vmss00000b Ready agent 15h v1.24.9 10.0.64.168 Ubuntu 18.04.6 LTS 5.4.0-1106-azure containerd://1.6.18+azure-1 aks-eupgonline-22130134-vmss00000c Ready agent 15h v1.24.9 10.0.65.86 Ubuntu 18.04.6 LTS 5.4.0-1106-azure containerd://1.6.18+azure-1 aks-mpronline-56970595-vmss000000 Ready agent 25d v1.24.9 10.0.64.5 Ubuntu 18.04.6 LTS 5.4.0-1106-azure containerd://1.6.18+azure-1 aks-mpronline-56970595-vmss00000f Ready agent 25d v1.24.9 10.0.64.20 Ubuntu 18.04.6 LTS 5.4.0-1106-azure containerd://1.6.18+azure-1 aks-msvonline-26340797-vmss000004 Ready agent 25d v1.24.9 10.0.64.84 Ubuntu 18.04.6 LTS 5.4.0-1106-azure containerd://1.6.18+azure-1 aks-msvonline-26340797-vmss00000k Ready agent 9d v1.24.9 10.0.64.187 Ubuntu 18.04.6 LTS 5.4.0-1106-azure containerd://1.6.18+azure-1 aks-spponline-14737782-vmss000000 Ready agent 25d v1.24.9 10.0.64.43 Ubuntu 18.04.6 LTS 5.4.0-1106-azure containerd://1.6.18+azure-1 aks-spponline-14737782-vmss000002 Ready agent 25d v1.24.9 10.0.64.199 Ubuntu 18.04.6 LTS 5.4.0-1106-azure containerd://1.6.18+azure-1 aks-ssoonline-19992502-vmss000001 Ready agent 25d v1.24.9 10.0.64.157 Ubuntu 18.04.6 LTS 5.4.0-1106-azure containerd://1.6.18+azure-1 aks-ssoonline-19992502-vmss000009 Ready agent 25d v1.24.9 10.0.64.35 Ubuntu 18.04.6 LTS 5.4.0-1106-azure containerd://1.6.18+azure-1 aks-system-13807298-vmss000000 Ready agent 25d v1.24.9 10.0.64.53 Ubuntu 18.04.6 LTS 5.4.0-1106-azure containerd://1.6.18+azure-1 aks-system-13807298-vmss000001 Ready agent 25d v1.24.9 10.0.64.109 Ubuntu 18.04.6 LTS 5.4.0-1106-azure containerd://1.6.18+azure-1 aks-tideoffline-34673408-vmss000004 Ready agent 25d v1.24.9 10.0.64.213 Ubuntu 18.04.6 LTS 5.4.0-1106-azure containerd://1.6.18+azure-1

• How was the ingress-nginx-controller installed:

  • If helm was used then please show output of helm ls -A | grep -i ingress

  • $ helm ls -A | grep -i ingress nginx-ingress-controller default 1 2022-02-01 17:40:16.026532 +0530 +0530 deployed ingress-nginx-4.0.13 1.1.0

  • If helm was used then please show output of helm -n <ingresscontrollernamepspace> get values <helmreleasename> $ helm -n default get values nginx-ingress-controller USER-SUPPLIED VALUES: controller: admissionWebhooks: patch: nodeSelector: beta.kubernetes.io/os: linux autoscaling: enabled: true nodeSelector: beta.kubernetes.io/os: linux podAnnotations: config.linkerd.io/proxy-cpu-request: 100m config.linkerd.io/proxy-memory-request: 125Mi linkerd.io/inject: enabled replicaCount: 2 resources: requests: memory: 300Mi service: annotations: service.beta.kubernetes.io/azure-load-balancer-internal: "true" loadBalancerIP: 10.0.64.140 defaultBackend: nodeSelector: beta.kubernetes.io/os: linux

  • If helm was not used, then copy/paste the complete precise command used to install the controller, along with the flags and options used

  • if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances

• Current State of the controller:

  • kubectl describe ingressclasses

  • kubectl -n <ingresscontrollernamespace> get all -A -o wide - result is huge

  • kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname> $ kubectl describe pods nginx-ingress-controller-ingress-nginx-controller-b96bc95ddkcdm Name: nginx-ingress-controller-ingress-nginx-controller-b96bc95ddkcdm Namespace: default Priority: 0 Service Account: nginx-ingress-controller-ingress-nginx Node: aks-system-13807298-vmss000000/10.0.64.53 Start Time: Fri, 26 May 2023 14:32:39 -0400 Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=nginx-ingress-controller app.kubernetes.io/name=ingress-nginx pod-template-hash=b96bc95d7 Annotations: config.linkerd.io/proxy-cpu-request: 100m config.linkerd.io/proxy-memory-request: 125Mi linkerd.io/inject: enabled Status: Running IP: 10.0.64.65 IPs: IP: 10.0.64.65 Controlled By: ReplicaSet/nginx-ingress-controller-ingress-nginx-controller-b96bc95d7 Containers: controller: Container ID: containerd://25479303dbea50bb56a34cfe2edb31cff8e4a57d660d9750c10a935eb5b72f5f Image: registry.k8s.io/ingress-nginx/controller:v1.3.0@sha256:d1707ca76d3b044ab8a28277a2466a02100ee9f58a86af1535a3edf9323ea1b5 Image ID: registry.k8s.io/ingress-nginx/controller@sha256:d1707ca76d3b044ab8a28277a2466a02100ee9f58a86af1535a3edf9323ea1b5 Ports: 80/TCP, 443/TCP, 8443/TCP, 8441/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP Args: /nginx-ingress-controller --publish-service=$(POD_NAMESPACE)/nginx-ingress-controller-ingress-nginx-controller --election-id=ingress-controller-leader --controller-class=k8s.io/ingress-nginx --ingress-class=nginx --configmap=$(POD_NAMESPACE)/nginx-ingress-controller-ingress-nginx-controller --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services --udp-services-configmap=$(POD_NAMESPACE)/udp-services --v=5 State: Running Started: Fri, 26 May 2023 14:32:40 -0400 Ready: True Restart Count: 0 Requests: cpu: 100m memory: 300Mi Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5 Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3 Environment: POD_NAME: nginx-ingress-controller-ingress-nginx-controller-b96bc95ddkcdm (v1:metadata.name) POD_NAMESPACE: default (v1:metadata.namespace) LD_PRELOAD: /usr/local/lib/libmimalloc.so Mounts: /usr/local/certificates/ from webhook-cert (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wwczx (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: webhook-cert: Type: Secret (a volume populated by a Secret) SecretName: nginx-ingress-controller-ingress-nginx-admission Optional: false kube-api-access-wwczx: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: beta.kubernetes.io/os=linux kubernetes.io/os=linux Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

  • kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename> $ kubectl describe svc nginx-ingress-controller-ingress-nginx-controller -n default E0526 18:02:57.269402 7264 memcache.go:255] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1 E0526 18:02:57.273778 7264 memcache.go:255] couldn't get resource list for tap.linkerd.io/v1alpha1: the server is currently unable to handle the request E0526 18:02:57.341457 7264 memcache.go:106] couldn't get resource list for tap.linkerd.io/v1alpha1: the server is currently unable to handle the request E0526 18:02:57.410910 7264 memcache.go:106] couldn't get resource list for tap.linkerd.io/v1alpha1: the server is currently unable to handle the request E0526 18:02:57.482313 7264 memcache.go:106] couldn't get resource list for tap.linkerd.io/v1alpha1: the server is currently unable to handle the request Name: nginx-ingress-controller-ingress-nginx-controller Namespace: default Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=nginx-ingress-controller app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/version=1.1.0 helm.sh/chart=ingress-nginx-4.0.13 Annotations: meta.helm.sh/release-name: nginx-ingress-controller meta.helm.sh/release-namespace: default service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz service.beta.kubernetes.io/azure-load-balancer-internal: true Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx-ingress-controller,app.kubernetes.io/name=ingress-nginx Type: LoadBalancer IP Family Policy: SingleStack IP Families: IPv4 IP: 192.168.46.169 IPs: 192.168.46.169 IP: 10.0.64.140 LoadBalancer Ingress: 10.0.64.140 Port: http 80/TCP TargetPort: http/TCP NodePort: http 32137/TCP Endpoints: 10.0.64.143:80,10.0.64.65:80 Port: https 443/TCP TargetPort: https/TCP NodePort: https 30837/TCP Endpoints: 10.0.64.143:443,10.0.64.65:443 Port: tlgeupg 8441/TCP TargetPort: 8441/TCP NodePort: tlgeupg 31024/TCP Endpoints: 10.0.64.143:8441,10.0.64.65:8441 Session Affinity: None External Traffic Policy: Cluster Events:

• Current state of ingress object, if applicable:

  • kubectl -n <appnnamespace> get all,ing -o wide
  • kubectl -n <appnamespace> describe ing <ingressname>
  • If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag

• Others:

  • Any other related information like ;
  • copy/paste of the snippet (if applicable)
  • kubectl describe ... of any custom configmap(s) created and in use
  • Any other related information that may help

How to reproduce this issue: try to add one extra port in ingress controller deployment and svc and try to access pods vai ingress controller using newly added port

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[longwuyuan]

/remove-kind bug

• Please edit your issue description and format it for markdown. It makes it eaasier
• Helm values does not show the tcp port key:values so how was the tcp port exposure configured
• Its not clear if the LB has the tcp port in Azure firewall
• Its not clear how you tried to access the port. Copy paste the command and output like curl -v and its response
• The version of controller in use seems too old. You might want to the update to the latest supported foe your K8S version and update

[longwuyuan]

/kind support

[arunlakshmananl]

• nginx-ingress-controller

Please edit your issue description and format it for markdown. It makes it eaasier --> teied but after saving its still the same Helm values does not show the tcp port key:values so how was the tcp port exposure configured --> TCP port configured in nginx ingress deployment and service which is running in default namespace Its not clear if the LB has the tcp port in Azure firewall - yes its configrued Its not clear how you tried to access the port. Copy paste the command and output like curl -v and its response
root@flink-eupgonline-secure-64497f847-gb6gm:/# curl -k http://10.0.128.120:8441 curl: (28) Failed to connect to 10.0.128.120 port 8441: Connection timed out root@flink-eupgonline-secure-64497f847-gb6gm:/#

The version of controller in use seems too old. You might want to the update to the latest supported foe your K8S version and update - earlier i was using 1.0.0 and now i upgraded nginx ingress to v1.3.0 since i am using AKS - 1.24.9 which is supported as well

[longwuyuan]

• "Connection timed out" means a problem in the network and not in the controller
• Helm values does not show the key:value for tcp port so no idea what to comment because without key:value for tcp port in helm values, only other way is is the --set flag and that is also not seen

I think its better if yo ucome discuss this on kubernetes.slack.com because there are less resources for support issues here and there are lots of users and engineers in slack. You can register at slack.k8s.io if required.

/kind support

[github-actions[bot]]

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.

[longwuyuan]

TCP/UDP forwarding is being deprecated so there is no action item for the project on this issue.

/close

[k8s-ci-robot]

@longwuyuan: Closing this issue.

In response to [this](https://github.com/kubernetes/ingress-nginx/issues/9998#issuecomment-2345846275): >TCP/UDP forwarding is being deprecated so there is no action item for the project on this issue. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.