search

Kong / kubernetes-ingress-controller

:gorilla: Kong for Kubernetes: The official Ingress Controller for Kubernetes.
https://docs.konghq.com/kubernetes-ingress-controller/
Apache License 2.0
2.2k stars 590 forks source link

Kong Ingress returns 500 error after 5 min #6300

Open javiramos1 opened 2 months ago

javiramos1 commented 2 months ago

Is there an existing issue for this?

Current Behavior

When we send a request we get a 500 HTTP error after exactly 5 minutes that is not coming from the app. Nor the service. It is coming from the Ingress

Expected Behavior

Respect the timeout annotations and do not timeout after 5 minutes.

Steps To Reproduce

We are using Kong in Kubernetes:

app.kubernetes.io/version: '3.2'
helm.sh/chart: kong-2.22.0

We are using AWS EKS and we have setup kong with service.beta.kubernetes.io/aws-load-balancer-type: nlb

We have an application with an Ingress and a Service.

The service has the timeout annotations:

konghq.com/read-timeout: '915000'
    konghq.com/send-timeout: '915000'

When we send a request we get a 500 HTTP error after exactly 5 minutes that is not coming from the app. I tried to port-forward to the service and it works fine. It only happens when using the ingress.

If I set the timeout annotations to 3 minutes then I get 504 timeout error after 3 minutes as expected. If I set it to longer than 5 minutes, then I get a 500 error.

I tried to disable all plugins but it did not help. We have no idea why this is happening. Any help would be greatly appreciated since we are really stuck, thank you!


### Kong Ingress Controller version

```shell
3.2

kong-2.22.0

Kubernetes version

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.14", GitCommit:"3321ffc07d2f046afdf613796f9032f4460de093", GitTreeState:"clean", BuildDate:"2022-11-09T13:40:19Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"29+", GitVersion:"v1.29.4-eks-036c24b", GitCommit:"9c0e57823b31865d0ee095997d9e7e721ffdc77f", GitTreeState:"clean", BuildDate:"2024-04-30T23:53:58Z", GoVersion:"go1.21.9", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.23) and server (1.29) exceeds the supported minor version skew of +/-1

Anything else?

No response

randmonkey commented 2 months ago

@javiramos1 Please confirm the version of your KIC and Kong gateway separately. The version 3.2 provided seems to be the version of Kong gateway. You can use the following command to get the image used for Ingress controller and Kong gateway

kubectl get pods -n <ns> -l app.kubernetes.io/name=kong -o jsonpath={.spec.containers[*].image}

Also, konghq.com/send-timeout is not a supported annotation and Kong service does not have a send_timeout attribute. You can set konghq.com/connect-timeout or konghq.com/write-timeout: https://docs.konghq.com/kubernetes-ingress-controller/latest/reference/annotations/#konghqcomconnect-timeout. However, if your KIC version is below 2.8, they are not supported.

javiramos1 commented 2 months ago

Hi, thanks for the reply. I will update the annotations and remove send-timeout.

We are using kong/kubernetes-ingress-controller:2.9 thanks!

randmonkey commented 1 month ago

@javiramos1 Can you please provide the further information for us to triage?