Ingress Time outs after 2 minutes even though time out is set to 3600s

kherath17 commented 12 months ago

What happened: I have setup an ingress service to route requests to Selenium Grid on K8, even though the requests are reached to grid que, it timeouts after 2 minutes.

Returned below response and execution stopped <head><title>504 Gateway Time-out</title></head> <body> <center>504 Gateway Time-out</center> </body> </html>

What you expected to happen: Ingress should honor the timeout values we have configured through below annotations : nginx.ingress.kubernetes.io/proxy-connect-timeout: "3600" nginx.ingress.kubernetes.io/proxy-read-timeout: "3600" nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.): v1.7.0

Kubernetes version (use kubectl version): v1.28.1

Environment:

Cloud provider or hardware configuration: AWS
OS (e.g. from /etc/os-release): linux
Kernel (e.g. uname -a):
Install tools:
- Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc.
Basic cluster related info:
- kubectl version
- kubectl get nodes -o wide
How was the ingress-nginx-controller installed:
- If helm was used then please show output of helm ls -A | grep -i ingress ->nginx-ingress-release default 1 2023-04-22 17:30:21.803047 +0530 +0530 deployed ingress-nginx-4.4.2 1.5.1
- If helm was used then please show output of helm -n <ingresscontrollernamepspace> get values <helmreleasename>
- If helm was not used, then copy/paste the complete precise command used to install the controller, along with the flags and options used
- if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances
Current State of the controller:
- kubectl describe ingressclasses Name: nginx Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=nginx-ingress-release app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.5.1 helm.sh/chart=ingress-nginx-4.4.2 Annotations: ingressclass.kubernetes.io/is-default-class: true meta.helm.sh/release-name: nginx-ingress-release meta.helm.sh/release-namespace: default Controller: k8s.io/ingress-nginx Events: none
- kubectl -n <ingresscontrollernamespace> get all -A -o wide
- kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>
- kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>
Current state of ingress object, if applicable:
- kubectl -n <appnnamespace> get all,ing -o wide
- kubectl -n <appnamespace> describe ing <ingressname>
- If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag
Others:
- Any other related information like ;
- copy/paste of the snippet (if applicable)
- kubectl describe ... of any custom configmap(s) created and in use
- Any other related information that may help

How to reproduce this issue: Setup Selenium Grid on K8 including below ingress and when requests are routed to Selenium Grid it should honor the timeouts mentioned

Sample Ingress File

apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: qlabv2-ingress annotations: `kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/rewrite-target: /$2 nginx.ingress.kubernetes.io/use-regex: "true" nginx.ingress.kubernetes.io/proxy-connect-timeout: "3600" nginx.ingress.kubernetes.io/proxy-read-timeout: "3600" nginx.ingress.kubernetes.io/proxy-send-timeout: "3600" spec: rules:

http: paths:
path: /sandbox_qlabv2(/|$)(.) pathType: Prefix backend: service: name: selenium-hub-service port: number: 80
path: /(/?)(session/./se/vnc) pathType: Prefix backend: service: name: selenium-hub-service port: number: 80

Anything else we need to know:

k8s-ci-robot commented 12 months ago

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

strongjz commented 12 months ago

Which load balancer did you deploy in EKS? Do you have endpoints for the service?

strongjz commented 12 months ago

/assign @strongjz

kherath17 commented 11 months ago

@strongjz apologies for the delayed response

Which load balancer did you deploy in EKS? - Nginx Load Balancer Do you have endpoints for the service? - We have shared end points with context paths

kubernetes / ingress-nginx

Ingress Time outs after 2 minutes even though time out is set to 3600s #10735

Sample Ingress File