search

kubernetes / ingress-nginx

Ingress NGINX Controller for Kubernetes
https://kubernetes.github.io/ingress-nginx/
Apache License 2.0
17.6k stars 8.27k forks source link

Unable to expose TCP Services #10307

Closed kirilovvv closed 9 months ago

kirilovvv commented 1 year ago

What happened:

Install Ingress Nginx controller via Helm Chart with exposed tcp services. After installation I was not able to consume the defined services: 

telnet <LoadBalancerIP> 25
Trying ***.***.***.***...
^C⏎

In the meanwhile connections to port 80 and 443 are working fine:

telnet <LoadBalancerIP> 80
Trying ***.***.***.***...
Connected to <LoadBalancerIP>.
Escape character is '^]'.

The LoadBalancer Service was created successfully (see below service description). Also the TCP Service Configmap was created successfully and passed as Args to the Deployment. In the logs of the controller there was error with fetching the Configmap:

W0812 17:34:25.510433       7 controller.go:446] Error getting ConfigMap "ingress-nginx-mail/ingress-nginx-mail-tcp": no object matching key "ingress-nginx-mail/ingress-nginx-mail-tcp" in local store

Also I have validated the the Service Account used by the controller has permissions to get the ngress-nginx-mail/ingress-nginx-mail-tcp Config Map.

What you expected to happen:

TCP Services to be loaded and be able to consume them.

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.): 1.8.1

Kubernetes version (use kubectl version): 1.25.11

Environment: EKS

Data

465:

mailserver/mail-frontend:465 993:

mailserver/mail-frontend:993 995:

mailserver/mail-frontend:995 25:

mailserver/mail-frontend:25

BinaryData

Events: 


    - Any other related information that may help

**How to reproduce this issue**:
Install ingress controller with TCP Services for port 25, 465, ... . Try accessing those services
`telnet <ingress-loadBalancer> 25`
`telnet <ingress-loadBalancer> 465`
k8s-ci-robot commented 1 year ago

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
longwuyuan commented 1 year ago

the port is configured. so try the telnet command from within the pod or from a adjacent pod in the namespace like kubectl -n ingress-nginx-mail create deploy test0 --image nginx:alpine and then exec into it and install telnet or netcat/nc etc.

But on a different note, no sane provider and specially AWS will let SMTP port 25 , without at least a warning as to how bad that is as a security problem.

/remove-kind bug

kirilovvv commented 1 year ago

The port is not configured:

root@busybox:/# telnet ingress-nginx-mail-controller 80
Trying 10.100.55.171...
Connected to ingress-nginx-mail-controller.ingress-nginx-mail.svc.cluster.local.
Escape character is '^]'.
root@busybox:/# telnet ingress-nginx-mail-controller 25
Trying 10.100.55.171...
telnet: Unable to connect to remote host: Connection refused

I believe here the error is thrown. If I'm right the ports are not configured at all.

Note: the mail server I am testing against is still a POC, I am aware with the security issues related to using non encrypted connections.

longwuyuan commented 1 year ago

Port configuration is visible in the svc definition already. Socket not responding is a different matter from port config. Information you have posted about port 25 not connecting is not pointing to a problem on the controller just yet. it requires you to show the pod to which the controller has configured TCP port 25.

I am testing if it works on kind cluster or not.

longwuyuan commented 1 year ago

As you can see in screenshot below, the controller is not having a bug to block port 25 image

kirilovvv commented 1 year ago

Connectivity to the pod is working fine. I have started a busy box container next to the ingress nginx controller:

root@busybox:/# telnet mail-frontend.mailserver.svc.cluster.local 25
Trying 10.100.123.100...
Connected to mail-frontend.mailserver.svc.cluster.local.
Escape character is '^]'.
220 *** ESMTP ready
^]
root@busybox:/# telnet mail-frontend.mailserver.svc.cluster.local 465
Trying 10.100.123.100...
Connected to mail-frontend.mailserver.svc.cluster.local.
Escape character is '^]'.
^]

So connectivity to the mail pod is ok.

Again the error log from the controller points me to a direction that the controller cannot get the tcp configmap.

Error log:

W0812 17:34:25.510433       7 controller.go:446] Error getting ConfigMap "ingress-nginx-mail/ingress-nginx-mail-tcp": no object matching key "ingress-nginx-mail/ingress-nginx-mail-tcp" in local store

I have also checked that the Service Account created has the required permissions to get that ConfigMap. Based on the log data I believe that this operation fails to execute properly and there stops the processing tcp configurations.

longwuyuan commented 1 year ago

@kirilovvv the test I showed that I could reach port 25 of a pod from outside the cluster, over the ingress. And besides that we have not touched that part of the code recently for a new bug to be likely.

You are also right that the error is related to controller.go Line 449. But that is a misleading information.

Here is the only data that is provided by you, that is related to your problem

root@busybox:/# telnet ingress-nginx-mail-controller 25
Trying 10.100.55.171...
telnet: Unable to connect to remote host: Connection refused

So if you can not get a socket there, the next data you need to consider is the ipaddress/hostname of the AWS LoadBalancer allowing connections on port 25. Maybe securityGroups of AWS. The controller does not control all that and the controller does not block port25, if configured.

As per that error related to the cinfigMap itself, you have shown the cm but not the service it sends traffic to. Is there a service in namespace of the mailserver, with the name "mail-frontend", with a port 25.

Beyond tht, did you change the default serviceAccount as well ?

kirilovvv commented 1 year ago

Well if I read the code - after that error the function that configures the streaming services exits from the function so the configuration is never completed..

Also as seen in the telnet command, I am trying to connect to the nginx service using the internal Service domain - so Security Groups, Load Balancers, etc AWS resources are not used - only k8s internal resources.

Here is the mail-frontend service:

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2023-04-05T12:14:12Z"
  labels:
    app: mail
  name: mail-frontend
  namespace: mailserver
  resourceVersion: "2955422"
  uid: 06186601-4867-4370-bef6-c1cf496a710f
spec:
  clusterIP: 10.100.123.100
  clusterIPs:
  - 10.100.123.100
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: smtp
    nodePort: 31583
    port: 25
    protocol: TCP
    targetPort: smtp
  - name: smtps
    nodePort: 32149
    port: 465
    protocol: TCP
    targetPort: smtps
  - name: submission
    nodePort: 31957
    port: 587
    protocol: TCP
    targetPort: submission
  - name: imap
    nodePort: 31846
    port: 143
    protocol: TCP
    targetPort: imap
- name: imap
    nodePort: 31846
    port: 143
    protocol: TCP
    targetPort: imap
  - name: imaps
    nodePort: 31850
    port: 993
    protocol: TCP
    targetPort: imaps
  - name: pop3s
    nodePort: 32103
    port: 995
    protocol: TCP
    targetPort: pop3s
  - name: pop3
    nodePort: 30962
    port: 110
    protocol: TCP
    targetPort: pop3
  selector:
    app: mail
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

I have tested the connectivity to that Service from a test pod, deployed in the nginx ingress namespace:

root@busybox:/# telnet mail-frontend.mailserver.svc.cluster.local 25
Trying 10.100.123.100...
Connected to mail-frontend.mailserver.svc.cluster.local.
Escape character is '^]'.
220 *** ESMTP ready
^]
root@busybox:/# telnet mail-frontend.mailserver.svc.cluster.local 465
Trying 10.100.123.100...
Connected to mail-frontend.mailserver.svc.cluster.local.
Escape character is '^]'.
^]

The controller uses the default service account (Service Account: ingress-nginx-mail, as provided in the the pod description above). I have tested that this Service account has permissions to Get/List ConfigMaps. Here are the provided permissions:

image
longwuyuan commented 1 year ago

@kirilovvv I can only follow the data trail and not opinions.

image

W0812 17:34:25.510433 7 controller.go:446] Error getting ConfigMap "ingress-nginx-mail/ingress-nginx-mail-tcp": no object matching key "ingress-nginx-mail/ingress-nginx-mail-tcp" in local store

So the issue of unreadable configMap is not reproducible.

And if a issue is not reproducible, then it becomes a blocker for a developer to fix root cause.

The template of a new issue asks many questions for this reason. It helps to get a better understanding of the issue reported. I don't see all the answers to questions asked in a new issue template.

So at least I can see your opinions. But to work collaboratively, the first requirement is un-biased data being presented, that relates to the complete live state of K8S objects related to the issue and the info on the infra where the issue occured.

If you do desire some meaningful commenting, you need to consider that the IP-Address of the pod as seen in latest telnet command is from the subnet 10.100.x.x/x and the it is expected to be a ipaddress from the CNI subnets. But a reader here has to to scratch his head to understand how come the external-IP you showed telnet failure for, is also from the subnet 10.100.x.x/x . So that needs clarification if your CNI subnet and the LB subnet are overlapping. Why did you NOT send telnet to this LB 

LoadBalancer Ingress:   ***.elb.eu-central-1.amazonaws.com

Why are you sending your telnet to an ipaddress from the subnet 10.10.x.x./x, instead of that LB ?

If possible, please post this data in one single post ;

kirilovvv commented 1 year ago

In the description/previous comments I have shown both telnet for the LB and using the internal service and they both fail. Also the telnets from the busybox are using the Service as mail-frontend Service is of type Cluster. Moreover I am not using the LB as its AWS Target Groups are unhealthy (for all ports except 80 and 443) and therefore won't accept traffic.

Here is the data that you requested:

NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR calico-apiserver service/calico-api ClusterIP 10.100.180.88 443/TCP 142d apiserver=true calico-system service/calico-kube-controllers-metrics ClusterIP 10.100.170.89 9094/TCP 142d k8s-app=calico-kube-controllers calico-system service/calico-typha ClusterIP 10.100.45.120 5473/TCP 143d k8s-app=calico-typha cert-manager service/cert-manager ClusterIP 10.100.63.179 9402/TCP 143d app.kubernetes.io/component=controller,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cert-manager cert-manager service/cert-manager-webhook ClusterIP 10.100.235.185 443/TCP 143d app.kubernetes.io/component=webhook,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=webhook default service/kubernetes ClusterIP 10.100.0.1 443/TCP 143d ingress-nginx-mail service/ingress-nginx-mail-controller LoadBalancer 10.100.55.171 k8s-ingressn-ingressn-4905a63ad9-519447fdc8742ee8.elb.eu-central-1.amazonaws.com 80:30655/TCP,443:31426/TCP,25:31927/TCP,465:31082/TCP,993:31157/TCP,995:32058/TCP 13d app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx-mail,app.kubernetes.io/name=ingress-nginx ingress-nginx-mail service/ingress-nginx-mail-controller-admission ClusterIP 10.100.157.181 443/TCP 13d app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx-mail,app.kubernetes.io/name=ingress-nginx ingress-nginx-mail service/ingress-nginx-mail-controller-metrics ClusterIP 10.100.39.192 10254/TCP 13d app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx-mail,app.kubernetes.io/name=ingress-nginx kube-system service/aws-load-balancer-webhook-service ClusterIP 10.100.102.179 443/TCP 142d app.kubernetes.io/instance=aws-load-balancer-controller,app.kubernetes.io/name=aws-load-balancer-controller kube-system service/kube-dns ClusterIP 10.100.0.10 53/UDP,53/TCP 143d k8s-app=kube-dns mailserver service/mail-frontend LoadBalancer 10.100.123.100 ***.elb.eu-central-1.amazonaws.com 25:31583/TCP,465:32149/TCP,587:31957/TCP,143:31846/TCP,993:31850/TCP,995:32103/TCP,110:30962/TCP,443:31340/TCP,80:31768/TCP 140d app=mail mailserver service/mailcluster-svc ClusterIP 10.100.9.252 7000/TCP,9000/TCP 12d app=mailserver monitoring service/grafana ClusterIP 10.100.188.70 80/TCP 141d app.kubernetes.io/instance=grafana,app.kubernetes.io/name=grafana monitoring service/prometheus-alertmanager ClusterIP 10.100.168.17 9093/TCP 141d app.kubernetes.io/instance=prometheus,app.kubernetes.io/name=alertmanager monitoring service/prometheus-alertmanager-headless ClusterIP None 9093/TCP 141d app.kubernetes.io/instance=prometheus,app.kubernetes.io/name=alertmanager monitoring service/prometheus-kube-state-metrics ClusterIP 10.100.5.11 8080/TCP 141d app.kubernetes.io/instance=prometheus,app.kubernetes.io/name=kube-state-metrics monitoring service/prometheus-prometheus-node-exporter ClusterIP 10.100.106.149 9100/TCP 141d app.kubernetes.io/instance=prometheus,app.kubernetes.io/name=prometheus-node-exporter monitoring service/prometheus-prometheus-pushgateway ClusterIP 10.100.149.214 9091/TCP 141d app.kubernetes.io/instance=prometheus,app.kubernetes.io/name=prometheus-pushgateway monitoring service/prometheus-server ClusterIP 10.100.104.14 80/TCP 141d app=prometheus,component=server,release=prometheus velero service/velero ClusterIP 10.100.157.226 8085/TCP 118d app.kubernetes.io/instance=velero,app.kubernetes.io/name=velero,name=velero

NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE CONTAINERS IMAGES SELECTOR calico-system daemonset.apps/calico-node 3 3 3 3 3 kubernetes.io/os=linux 143d calico-node docker.io/calico/node:v3.24.1 k8s-app=calico-node calico-system daemonset.apps/csi-node-driver 3 3 3 3 3 kubernetes.io/os=linux 142d calico-csi,csi-node-driver-registrar docker.io/calico/csi:v3.24.1,docker.io/calico/node-driver-registrar:v3.24.1 k8s-app=csi-node-driver kube-system daemonset.apps/aws-node 3 3 3 3 3 143d aws-node 602401143452.dkr.ecr.eu-central-1.amazonaws.com/amazon-k8s-cni:v1.12.6-eksbuild.1 k8s-app=aws-node kube-system daemonset.apps/ebs-csi-node 3 3 3 3 3 kubernetes.io/os=linux 143d ebs-plugin,node-driver-registrar,liveness-probe 602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/aws-ebs-csi-driver:v1.18.0,602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/csi-node-driver-registrar:v2.7.0-eks-1-26-7,602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/livenessprobe:v2.9.0-eks-1-26-7 app=ebs-csi-node,app.kubernetes.io/name=aws-ebs-csi-driver kube-system daemonset.apps/ebs-csi-node-windows 0 0 0 0 0 kubernetes.io/os=windows 143d ebs-plugin,node-driver-registrar,liveness-probe 602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/aws-ebs-csi-driver:v1.18.0,602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/csi-node-driver-registrar:v2.7.0-eks-1-26-7,602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/livenessprobe:v2.9.0-eks-1-26-7 app=ebs-csi-node,app.kubernetes.io/name=aws-ebs-csi-driver kube-system daemonset.apps/efs-csi-node 3 3 3 3 3 kubernetes.io/os=linux 106d efs-plugin,csi-driver-registrar,liveness-probe amazon/aws-efs-csi-driver:v1.5.5,public.ecr.aws/eks-distro/kubernetes-csi/node-driver-registrar:v2.7.0-eks-1-27-latest,public.ecr.aws/eks-distro/kubernetes-csi/livenessprobe:v2.9.0-eks-1-27-latest app=efs-csi-node,app.kubernetes.io/instance=aws-efs-csi-driver,app.kubernetes.io/name=aws-efs-csi-driver kube-system daemonset.apps/kube-proxy 3 3 3 3 3 143d kube-proxy 602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/kube-proxy:v1.25.9-minimal-eksbuild.1 k8s-app=kube-proxy monitoring daemonset.apps/prometheus-prometheus-node-exporter 3 3 3 3 3 141d node-exporter quay.io/prometheus/node-exporter:v1.5.0 app.kubernetes.io/instance=prometheus,app.kubernetes.io/name=prometheus-node-exporter

NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR calico-apiserver deployment.apps/calico-apiserver 2/2 2 2 142d calico-apiserver docker.io/calico/apiserver:v3.24.1 apiserver=true calico-system deployment.apps/calico-kube-controllers 1/1 1 1 142d calico-kube-controllers docker.io/calico/kube-controllers:v3.24.1 k8s-app=calico-kube-controllers calico-system deployment.apps/calico-typha 2/2 2 2 143d calico-typha docker.io/calico/typha:v3.24.1 k8s-app=calico-typha cert-manager deployment.apps/cert-manager 1/1 1 1 143d cert-manager-controller quay.io/jetstack/cert-manager-controller:v1.11.0 app.kubernetes.io/component=controller,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cert-manager cert-manager deployment.apps/cert-manager-cainjector 1/1 1 1 143d cert-manager-cainjector quay.io/jetstack/cert-manager-cainjector:v1.11.0 app.kubernetes.io/component=cainjector,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cainjector cert-manager deployment.apps/cert-manager-webhook 1/1 1 1 143d cert-manager-webhook quay.io/jetstack/cert-manager-webhook:v1.11.0 app.kubernetes.io/component=webhook,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=webhook ingress-nginx-mail deployment.apps/ingress-nginx-mail-controller 1/1 1 1 13d controller registry.k8s.io/ingress-nginx/controller:v1.5.1@sha256:4ba73c697770664c1e00e9f968de14e08f606ff961c76e5d7033a4a9c593c629 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx-mail,app.kubernetes.io/name=ingress-nginx kube-system deployment.apps/aws-load-balancer-controller 2/2 2 2 142d aws-load-balancer-controller public.ecr.aws/eks/aws-load-balancer-controller:v2.4.7 app.kubernetes.io/instance=aws-load-balancer-controller,app.kubernetes.io/name=aws-load-balancer-controller kube-system deployment.apps/coredns 2/2 2 2 143d coredns 602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/coredns:v1.9.3-eksbuild.3 eks.amazonaws.com/component=coredns,k8s-app=kube-dns kube-system deployment.apps/ebs-csi-controller 2/2 2 2 143d ebs-plugin,csi-provisioner,csi-attacher,csi-snapshotter,csi-resizer,liveness-probe 602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/aws-ebs-csi-driver:v1.18.0,602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/csi-provisioner:v3.4.1-eks-1-26-7,602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/csi-attacher:v4.2.0-eks-1-26-7,602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/csi-snapshotter:v6.2.1-eks-1-26-7,602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/csi-resizer:v1.7.0-eks-1-26-7,602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/livenessprobe:v2.9.0-eks-1-26-7 app=ebs-csi-controller,app.kubernetes.io/name=aws-ebs-csi-driver kube-system deployment.apps/efs-csi-controller 2/2 2 2 106d efs-plugin,csi-provisioner,liveness-probe amazon/aws-efs-csi-driver:v1.5.5,public.ecr.aws/eks-distro/kubernetes-csi/external-provisioner:v3.4.0-eks-1-27-latest,public.ecr.aws/eks-distro/kubernetes-csi/livenessprobe:v2.9.0-eks-1-27-latest app=efs-csi-controller,app.kubernetes.io/instance=aws-efs-csi-driver,app.kubernetes.io/name=aws-efs-csi-driver kube-system deployment.apps/external-dns 1/1 1 1 142d external-dns k8s.gcr.io/external-dns/external-dns:v0.10.2 app=external-dns kube-system deployment.apps/snapshot-controller 1/1 1 1 143d snapshot-controller registry.k8s.io/sig-storage/snapshot-controller:v6.2.1 app.kubernetes.io/instance=snapshot-controller,app.kubernetes.io/name=snapshot-controller kube-system deployment.apps/tigera-operator 1/1 1 1 143d tigera-operator quay.io/tigera/operator:v1.28.1 name=tigera-operator mailserver deployment.apps/mail 1/1 1 1 136d axigen axigen/axigen:10.5.1 app=mail mailserver deployment.apps/mailserver 1/1 1 1 12d axigen axigen/axigen:10.5.4 app=mailserver monitoring deployment.apps/grafana 1/1 1 1 141d grafana grafana/grafana:9.3.1 app.kubernetes.io/instance=grafana,app.kubernetes.io/name=grafana monitoring deployment.apps/prometheus-kube-state-metrics 1/1 1 1 141d kube-state-metrics registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.7.0 app.kubernetes.io/instance=prometheus,app.kubernetes.io/name=kube-state-metrics monitoring deployment.apps/prometheus-prometheus-pushgateway 1/1 1 1 141d pushgateway prom/pushgateway:v1.5.1 app.kubernetes.io/instance=prometheus,app.kubernetes.io/name=prometheus-pushgateway monitoring deployment.apps/prometheus-server 1/1 1 1 141d prometheus-server-configmap-reload,prometheus-server jimmidyson/configmap-reload:v0.8.0,quay.io/prometheus/prometheus:v2.41.0 app=prometheus,component=server,release=prometheus velero deployment.apps/velero 1/1 1 1 118d velero velero/velero:v1.11.0 app.kubernetes.io/instance=velero,app.kubernetes.io/name=velero

NAMESPACE NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR calico-apiserver replicaset.apps/calico-apiserver-58f444b5c 2 2 2 142d calico-apiserver docker.io/calico/apiserver:v3.24.1 apiserver=true,pod-template-hash=58f444b5c calico-system replicaset.apps/calico-kube-controllers-6b96b8cc86 1 1 1 142d calico-kube-controllers docker.io/calico/kube-controllers:v3.24.1 k8s-app=calico-kube-controllers,pod-template-hash=6b96b8cc86 calico-system replicaset.apps/calico-typha-6cdb87c98 0 0 0 143d calico-typha docker.io/calico/typha:v3.25.1 k8s-app=calico-typha,pod-template-hash=6cdb87c98 calico-system replicaset.apps/calico-typha-85898bf5d9 2 2 2 142d calico-typha docker.io/calico/typha:v3.24.1 k8s-app=calico-typha,pod-template-hash=85898bf5d9 cert-manager replicaset.apps/cert-manager-5667f46fc9 1 1 1 143d cert-manager-controller quay.io/jetstack/cert-manager-controller:v1.11.0 app.kubernetes.io/component=controller,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cert-manager,pod-template-hash=5667f46fc9 cert-manager replicaset.apps/cert-manager-59bf757d77 0 0 0 143d cert-manager-controller quay.io/jetstack/cert-manager-controller:v1.11.0 app.kubernetes.io/component=controller,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cert-manager,pod-template-hash=59bf757d77 cert-manager replicaset.apps/cert-manager-cainjector-547c9b8f95 1 1 1 143d cert-manager-cainjector quay.io/jetstack/cert-manager-cainjector:v1.11.0 app.kubernetes.io/component=cainjector,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cainjector,pod-template-hash=547c9b8f95 cert-manager replicaset.apps/cert-manager-webhook-6787f645b9 1 1 1 143d cert-manager-webhook quay.io/jetstack/cert-manager-webhook:v1.11.0 app.kubernetes.io/component=webhook,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=webhook,pod-template-hash=6787f645b9 ingress-nginx-mail replicaset.apps/ingress-nginx-mail-controller-5d54bccb87 0 0 0 13d controller registry.k8s.io/ingress-nginx/controller:v1.8.1@sha256:e5c4824e7375fcf2a393e1c03c293b69759af37a9ca6abdb91b13d78a93da8bd app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx-mail,app.kubernetes.io/name=ingress-nginx,pod-template-hash=5d54bccb87 ingress-nginx-mail replicaset.apps/ingress-nginx-mail-controller-64d595fb55 0 0 0 13d controller registry.k8s.io/ingress-nginx/controller:v1.8.1@sha256:e5c4824e7375fcf2a393e1c03c293b69759af37a9ca6abdb91b13d78a93da8bd app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx-mail,app.kubernetes.io/name=ingress-nginx,pod-template-hash=64d595fb55 ingress-nginx-mail replicaset.apps/ingress-nginx-mail-controller-6bbf5958ff 0 0 0 11d controller registry.k8s.io/ingress-nginx/controller:v1.8.1@sha256:e5c4824e7375fcf2a393e1c03c293b69759af37a9ca6abdb91b13d78a93da8bd app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx-mail,app.kubernetes.io/name=ingress-nginx,pod-template-hash=6bbf5958ff ingress-nginx-mail replicaset.apps/ingress-nginx-mail-controller-74d667cbdc 0 0 0 11d controller registry.k8s.io/ingress-nginx/controller:v1.8.1@sha256:e5c4824e7375fcf2a393e1c03c293b69759af37a9ca6abdb91b13d78a93da8bd app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx-mail,app.kubernetes.io/name=ingress-nginx,pod-template-hash=74d667cbdc ingress-nginx-mail replicaset.apps/ingress-nginx-mail-controller-78cc8799dc 1 1 1 10d controller registry.k8s.io/ingress-nginx/controller:v1.5.1@sha256:4ba73c697770664c1e00e9f968de14e08f606ff961c76e5d7033a4a9c593c629 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx-mail,app.kubernetes.io/name=ingress-nginx,pod-template-hash=78cc8799dc kube-system replicaset.apps/aws-load-balancer-controller-786f47b99f 2 2 2 142d aws-load-balancer-controller public.ecr.aws/eks/aws-load-balancer-controller:v2.4.7 app.kubernetes.io/instance=aws-load-balancer-controller,app.kubernetes.io/name=aws-load-balancer-controller,pod-template-hash=786f47b99f kube-system replicaset.apps/coredns-5c4469f767 2 2 2 109d coredns 602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/coredns:v1.9.3-eksbuild.3 eks.amazonaws.com/component=coredns,k8s-app=kube-dns,pod-template-hash=5c4469f767 kube-system replicaset.apps/coredns-cbbbbb9cb 0 0 0 143d coredns 602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/coredns:v1.9.3-eksbuild.2 eks.amazonaws.com/component=coredns,k8s-app=kube-dns,pod-template-hash=cbbbbb9cb kube-system replicaset.apps/ebs-csi-controller-587f5bd575 2 2 2 109d ebs-plugin,csi-provisioner,csi-attacher,csi-snapshotter,csi-resizer,liveness-probe 602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/aws-ebs-csi-driver:v1.18.0,602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/csi-provisioner:v3.4.1-eks-1-26-7,602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/csi-attacher:v4.2.0-eks-1-26-7,602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/csi-snapshotter:v6.2.1-eks-1-26-7,602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/csi-resizer:v1.7.0-eks-1-26-7,602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/livenessprobe:v2.9.0-eks-1-26-7 app=ebs-csi-controller,app.kubernetes.io/name=aws-ebs-csi-driver,pod-template-hash=587f5bd575 kube-system replicaset.apps/ebs-csi-controller-59b7dfccbf 0 0 0 143d ebs-plugin,csi-provisioner,csi-attacher,csi-snapshotter,csi-resizer,liveness-probe 602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/aws-ebs-csi-driver:v1.17.0,602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/csi-provisioner:v3.4.0-eks-1-26-latest,602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/csi-attacher:v4.2.0-eks-1-26-latest,602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/csi-snapshotter:v6.2.1-eks-1-26-latest,602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/csi-resizer:v1.7.0-eks-1-26-latest,602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/livenessprobe:v2.9.0-eks-1-26-latest app=ebs-csi-controller,app.kubernetes.io/name=aws-ebs-csi-driver,pod-template-hash=59b7dfccbf kube-system replicaset.apps/efs-csi-controller-f5f978597 2 2 2 106d efs-plugin,csi-provisioner,liveness-probe amazon/aws-efs-csi-driver:v1.5.5,public.ecr.aws/eks-distro/kubernetes-csi/external-provisioner:v3.4.0-eks-1-27-latest,public.ecr.aws/eks-distro/kubernetes-csi/livenessprobe:v2.9.0-eks-1-27-latest app=efs-csi-controller,app.kubernetes.io/instance=aws-efs-csi-driver,app.kubernetes.io/name=aws-efs-csi-driver,pod-template-hash=f5f978597 kube-system replicaset.apps/external-dns-b4d6f58b 1 1 1 142d external-dns k8s.gcr.io/external-dns/external-dns:v0.10.2 app=external-dns,pod-template-hash=b4d6f58b kube-system replicaset.apps/snapshot-controller-8696cb4477 1 1 1 143d snapshot-controller registry.k8s.io/sig-storage/snapshot-controller:v6.2.1 app.kubernetes.io/instance=snapshot-controller,app.kubernetes.io/name=snapshot-controller,pod-template-hash=8696cb4477 kube-system replicaset.apps/tigera-operator-6675dc47f4 1 1 1 142d tigera-operator quay.io/tigera/operator:v1.28.1 name=tigera-operator,pod-template-hash=6675dc47f4 kube-system replicaset.apps/tigera-operator-6d85b97b7d 0 0 0 143d tigera-operator quay.io/tigera/operator:v1.29.3 name=tigera-operator,pod-template-hash=6d85b97b7d mailserver replicaset.apps/mail-574d9d4767 0 0 0 108d axigen axigen/axigen:10.4.8 app=mail,pod-template-hash=574d9d4767 mailserver replicaset.apps/mail-579f547f57 0 0 0 136d axigen axigen/axigen:10.4.8 app=mail,pod-template-hash=579f547f57 mailserver replicaset.apps/mail-66cf898cb8 0 0 0 108d axigen axigen/axigen:10.4.10 app=mail,pod-template-hash=66cf898cb8 mailserver replicaset.apps/mail-7c768cc875 1 1 1 44d axigen axigen/axigen:10.5.1 app=mail,pod-template-hash=7c768cc875 mailserver replicaset.apps/mailserver-7bb857989 1 1 1 12d axigen axigen/axigen:10.5.4 app=mailserver,pod-template-hash=7bb857989 monitoring replicaset.apps/grafana-76b5b6b4c6 0 0 0 141d grafana grafana/grafana:9.3.1 app.kubernetes.io/instance=grafana,app.kubernetes.io/name=grafana,pod-template-hash=76b5b6b4c6 monitoring replicaset.apps/grafana-dfbf75d74 1 1 1 140d grafana grafana/grafana:9.3.1 app.kubernetes.io/instance=grafana,app.kubernetes.io/name=grafana,pod-template-hash=dfbf75d74 monitoring replicaset.apps/prometheus-kube-state-metrics-5fdc4f6d6c 1 1 1 141d kube-state-metrics registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.7.0 app.kubernetes.io/instance=prometheus,app.kubernetes.io/name=kube-state-metrics,pod-template-hash=5fdc4f6d6c monitoring replicaset.apps/prometheus-prometheus-pushgateway-59d858554f 1 1 1 141d pushgateway prom/pushgateway:v1.5.1 app.kubernetes.io/instance=prometheus,app.kubernetes.io/name=prometheus-pushgateway,pod-template-hash=59d858554f monitoring replicaset.apps/prometheus-server-6758784968 0 0 0 141d prometheus-server-configmap-reload,prometheus-server jimmidyson/configmap-reload:v0.8.0,quay.io/prometheus/prometheus:v2.41.0 app=prometheus,component=server,pod-template-hash=6758784968,release=prometheus monitoring replicaset.apps/prometheus-server-7f64f585f 1 1 1 137d prometheus-server-configmap-reload,prometheus-server jimmidyson/configmap-reload:v0.8.0,quay.io/prometheus/prometheus:v2.41.0 app=prometheus,component=server,pod-template-hash=7f64f585f,release=prometheus velero replicaset.apps/velero-55f6d8dbb5 1 1 1 118d velero velero/velero:v1.11.0 app.kubernetes.io/instance=velero,app.kubernetes.io/name=velero,pod-template-hash=55f6d8dbb5 velero replicaset.apps/velero-66fbf5d59b 0 0 0 118d velero velero/velero:v1.11.0 app.kubernetes.io/instance=velero,app.kubernetes.io/name=velero,pod-template-hash=66fbf5d59b

NAMESPACE NAME READY AGE CONTAINERS IMAGES monitoring statefulset.apps/prometheus-alertmanager 1/1 141d alertmanager quay.io/prometheus/alertmanager:v0.25.0

* kubectl -n ingress-nginx-mail describe cm,svc

Name: ingress-nginx-mail-controller Namespace: ingress-nginx-mail Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx-mail app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.5.1 helm.sh/chart=ingress-nginx-4.4.2 Annotations: meta.helm.sh/release-name: ingress-nginx-mail meta.helm.sh/release-namespace: ingress-nginx-mail

Data

server-snippet:

real_ip_header CF-Connecting-IP;

use-forwarded-headers:

true use-proxy-protocol:

true allow-snippet-annotations:

true

BinaryData

Events:

Name: ingress-nginx-mail-tcp Namespace: ingress-nginx-mail Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx-mail app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.5.1 helm.sh/chart=ingress-nginx-4.4.2 Annotations: meta.helm.sh/release-name: ingress-nginx-mail meta.helm.sh/release-namespace: ingress-nginx-mail

Data

993:

mailserver/mail-frontend:993 995:

mailserver/mail-frontend:995 25:

mailserver/mail-frontend:25 465:

mailserver/mail-frontend:465

BinaryData

Events:

Name: kube-root-ca.crt Namespace: ingress-nginx-mail Labels: Annotations: kubernetes.io/description: Contains a CA bundle that can be used to verify the kube-apiserver when using internal endpoints such as the internal service IP or kubern...

Data

ca.crt:

-----BEGIN CERTIFICATE-----

-----END CERTIFICATE----- BinaryData ==== Events: Name: ingress-nginx-mail-controller Namespace: ingress-nginx-mail Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx-mail app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.5.1 helm.sh/chart=ingress-nginx-4.4.2 Annotations: meta.helm.sh/release-name: ingress-nginx-mail meta.helm.sh/release-namespace: ingress-nginx-mail service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: * service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing service.beta.kubernetes.io/aws-load-balancer-type: external Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx-mail,app.kubernetes.io/name=ingress-nginx Type: LoadBalancer IP Family Policy: SingleStack IP Families: IPv4 IP: 10.100.55.171 IPs: 10.100.55.171 LoadBalancer Ingress: k8s-ingressn-ingressn-4905a63ad9-519447fdc8742ee8.elb.eu-central-1.amazonaws.com Port: http 80/TCP TargetPort: http/TCP NodePort: http 30655/TCP Endpoints: 172.26.4.200:80 Port: https 443/TCP TargetPort: https/TCP NodePort: https 31426/TCP Endpoints: 172.26.4.200:443 Port: 25-tcp 25/TCP TargetPort: 25-tcp/TCP NodePort: 25-tcp 31927/TCP Endpoints: 172.26.4.200:25 Port: 465-tcp 465/TCP TargetPort: 465-tcp/TCP NodePort: 465-tcp 31082/TCP Endpoints: 172.26.4.200:465 Port: 993-tcp 993/TCP TargetPort: 993-tcp/TCP NodePort: 993-tcp 31157/TCP Endpoints: 172.26.4.200:993 Port: 995-tcp 995/TCP TargetPort: 995-tcp/TCP NodePort: 995-tcp 32058/TCP Endpoints: 172.26.4.200:995 Session Affinity: None External Traffic Policy: Cluster Events: Name: ingress-nginx-mail-controller-admission Namespace: ingress-nginx-mail Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx-mail app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.5.1 helm.sh/chart=ingress-nginx-4.4.2 Annotations: meta.helm.sh/release-name: ingress-nginx-mail meta.helm.sh/release-namespace: ingress-nginx-mail Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx-mail,app.kubernetes.io/name=ingress-nginx Type: ClusterIP IP Family Policy: SingleStack IP Families: IPv4 IP: 10.100.157.181 IPs: 10.100.157.181 Port: https-webhook 443/TCP TargetPort: webhook/TCP Endpoints: 172.26.4.200:8443 Session Affinity: None Events: Name: ingress-nginx-mail-controller-metrics Namespace: ingress-nginx-mail Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx-mail app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.5.1 helm.sh/chart=ingress-nginx-4.4.2 Annotations: meta.helm.sh/release-name: ingress-nginx-mail meta.helm.sh/release-namespace: ingress-nginx-mail Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx-mail,app.kubernetes.io/name=ingress-nginx Type: ClusterIP IP Family Policy: SingleStack IP Families: IPv4 IP: 10.100.39.192 IPs: 10.100.39.192 Port: metrics 10254/TCP TargetPort: metrics/TCP Endpoints: 172.26.4.200:10254 Session Affinity: None Events: ``` * kubectl -n mailserver describe svc ``` Name: mail-frontend Namespace: mailserver Labels: app=mail Annotations: service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing service.beta.kubernetes.io/aws-load-balancer-type: external Selector: app=mail Type: LoadBalancer IP Family Policy: SingleStack IP Families: IPv4 IP: 10.100.123.100 IPs: 10.100.123.100 LoadBalancer Ingress: ***.elb.eu-central-1.amazonaws.com Port: transfer 25/TCP TargetPort: transfer/TCP NodePort: transfer 31583/TCP Endpoints: 172.26.7.63:25 Port: smtps 465/TCP TargetPort: smtps/TCP NodePort: smtps 32149/TCP Endpoints: 172.26.7.63:465 Port: submission 587/TCP TargetPort: submission/TCP NodePort: submission 31957/TCP Endpoints: 172.26.7.63:587 Port: imap 143/TCP TargetPort: imap/TCP NodePort: imap 31846/TCP Endpoints: 172.26.7.63:143 Port: imaps 993/TCP TargetPort: imaps/TCP NodePort: imaps 31850/TCP Endpoints: 172.26.7.63:993 Port: pop3s 995/TCP TargetPort: pop3s/TCP NodePort: pop3s 32103/TCP Endpoints: 172.26.7.63:995 Port: pop3 110/TCP TargetPort: pop3/TCP NodePort: pop3 30962/TCP Endpoints: 172.26.7.63:110 Port: https 443/TCP TargetPort: https/TCP NodePort: https 31340/TCP Endpoints: 172.26.7.63:443 Port: http 80/TCP TargetPort: http/TCP NodePort: http 31768/TCP Endpoints: 172.26.7.63:80 Session Affinity: None External Traffic Policy: Cluster Events: Name: mailcluster-svc Namespace: mailserver Labels: app=mailserver Annotations: Selector: app=mailserver Type: ClusterIP IP Family Policy: SingleStack IP Families: IPv4 IP: 10.100.9.252 IPs: 10.100.9.252 Port: cli 7000/TCP TargetPort: cli/TCP Endpoints: 172.26.18.80:7000 Port: web 9000/TCP TargetPort: web/TCP Endpoints: 172.26.18.80:9000 Session Affinity: None Events: ```
longwuyuan commented 1 year ago

Can you try tthis ;

kirilovvv commented 1 year ago

I created new service mail-frontend-test:

Name:              mail-frontend-test
Namespace:         mailserver
Annotations:       <none>
Selector:          app=mail
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.100.169.222
IPs:               10.100.169.222
Port:              smtp  25/TCP
TargetPort:        25/TCP
Endpoints:         172.26.7.63:25
Port:              imap  993/TCP
TargetPort:        993/TCP
Endpoints:         172.26.7.63:993
Session Affinity:  None
Events:            <none>

Tested it and it works:

root@busybox:/# telnet mail-frontend-test.mailserver 25
Trying 10.100.169.222...
Connected to mail-frontend-test.mailserver.svc.cluster.local.
Escape character is '^]'.
220 *** ESMTP ready
^]
telnet> Connection closed.
root@busybox:/# telnet mail-frontend-test.mailserver 993
Trying 10.100.169.222...
Connected to mail-frontend-test.mailserver.svc.cluster.local.
Escape character is '^]'.
^]
telnet> Connection closed.
root@busybox:/#

I also reconfigured the tcp part in my values.yaml:

tcp: 
  25: "mailserver/mail-frontend-test:25"
  465: "mailserver/mail-frontend-test:465"
  993: "mailserver/mail-frontend-test:993"
  995: "mailserver/mail-frontend-test:995"

After upgrade the same log persist:

W0827 13:39:14.610841       7 controller.go:369] Error getting ConfigMap "ingress-nginx-mail/ingress-nginx-mail-tcp": no object matching key "ingress-nginx-mail/ingress-nginx-mail-tcp" in local store
W0827 13:45:04.323017       7 controller.go:369] Error getting ConfigMap "ingress-nginx-mail/ingress-nginx-mail-tcp": no object matching key "ingress-nginx-mail/ingress-nginx-mail-tcp" in local store
W0827 13:45:09.302332       7 controller.go:369] Error getting ConfigMap "ingress-nginx-mail/ingress-nginx-mail-tcp": no object matching key "ingress-nginx-mail/ingress-nginx-mail-tcp" in local store
W0827 13:45:20.178936       7 controller.go:369] Error getting ConfigMap "ingress-nginx-mail/ingress-nginx-mail-tcp": no object matching key "ingress-nginx-mail/ingress-nginx-mail-tcp" in local store
W0827 13:45:35.001914       7 controller.go:369] Error getting ConfigMap "ingress-nginx-mail/ingress-nginx-mail-tcp": no object matching key "ingress-nginx-mail/ingress-nginx-mail-tcp" in local store

Telnet from outside the cluster:

~/git ❯❯❯ telnet k8s-ingressn-ingressn-4905a63ad9-519447fdc8742ee8.elb.eu-central-1.amazonaws.com 25                                                                                                       ✘ 130
Trying 18.184.230.162...
^C⏎                                                                                                                                                                                                             ~/git ❯❯❯ telnet k8s-ingressn-ingressn-4905a63ad9-519447fdc8742ee8.elb.eu-central-1.amazonaws.com 993                                                                                                      ✘ 130
Trying 18.184.230.162...
^C⏎                                                                                                                                                                                                             ~/git ❯❯❯ telnet k8s-ingressn-ingressn-4905a63ad9-519447fdc8742ee8.elb.eu-central-1.amazonaws.com 80                                                                                                       ✘ 130
Trying 18.184.230.162...
Connected to k8s-ingressn-ingressn-4905a63ad9-519447fdc8742ee8.elb.eu-central-1.amazonaws.com.
Escape character is '^]'.
^]
telnet> Connection closed.
~/git ❯❯❯ telnet k8s-ingressn-ingressn-4905a63ad9-519447fdc8742ee8.elb.eu-central-1.amazonaws.com 443
Trying 52.28.51.1...
Connected to k8s-ingressn-ingressn-4905a63ad9-519447fdc8742ee8.elb.eu-central-1.amazonaws.com.
Escape character is '^]'.
^]
telnet> ^C⏎
longwuyuan commented 1 year ago

I had commented on the state that was visible in your earlier post. So your latest update is 100% useless to me as the state has changed and my comments are invalid.

But I have tested this and there is no bug in the controller.

So the problem is now limited to your use case and it looks like you need personalized support.

There are not many resources here on github to track individual support issues. Please wait and see if other people comment and help solve your problem. Or you can come and discuss this at kubernetes.slack.com in the ingress-nginx-users

/kind support

github-actions[bot] commented 1 year ago

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.

rikatz commented 1 year ago

I'm not sure if helm is the right approach here, can you test following https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/?

rikatz commented 9 months ago

/close

No further answer, closing it.

k8s-ci-robot commented 9 months ago

@rikatz: Closing this issue.

In response to [this](https://github.com/kubernetes/ingress-nginx/issues/10307#issuecomment-1969921021): >/close > >No further answer, closing it. Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.