Having more than 100 `httproutes` may causes excessive resource conflicts

[gjkim42]

Is there an existing issue for this?

• [X] I have searched the existing issues

Current Behavior

When I create more than 100 httproutes, the kong ingress controller causes excessive resource conflicts on kube-apiserver side, as it cannot correctly determine the attachedRoutes of a Gateway resource but the value keeps fluctuating.

Expected Behavior

When I create more than 101 httproutes, the resource usage of kube-apiserver does not increase to an insane level, and attachedRoutes is correctly set.

Steps To Reproduce

1. Create 20 invalid httproutes and 100 valid httproutes.

$ cat test.sh
#!/usr/bin/env bash

function create_invalid_httproute {            
  local num="${1}"
  cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:    
  name: test-invalid-httproute-${num}
spec:            
  parentRefs:         
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: invalid-test-gateway
    namespace: test-gateway
  rules:           
  - backendRefs: 
    - group: "" 
      kind: Service
      name: non-existing
      port: 80
      weight: 1         
    matches:                              
    - path:
        type: PathPrefix
        value: /test-invalid-httproute-${num}/test
EOF                 
} 

function create_httproute {
  local num="${1}"
  cat <<EOF | kubectl create -f -
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: test-httproute-${num}
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: test-gateway
    namespace: test-gateway
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: test-service
      port: 80
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /test-httproute-${num}/test
EOF
}

for i in {1..20}; do
  create_invalid_httproute ${i}
done
for i in {1..100}; do
  create_httproute ${i}
done

$ ./test.sh

2. Make sure that kube-apiserver's (code_resource:apiserver_request_total:rate5m{verb="write", resource="gateways"}) metric is excessively increased.

3. Make sure that the issue is addressed after removing those httproutes.

Kong Ingress Controller version

v2.10.3

Kubernetes version

$ kubectl version
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"c285e781331a3785a7f436042c65c5641ce8a9e9", GitTreeState:"clean", BuildDate:"2022-03-16T15:52:18Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"linux/amd64"}

Anything else?

No response

[gjkim42]

https://github.com/Kong/kubernetes-ingress-controller/blob/7e567cc4192532561b630d58f6500009e067ba88/internal/controllers/gateway/gateway_utils.go#L700-L710

found that the issue is related to the fact that the mgrc.List returns httpRouteList without Continue token even if there are more than defaultEndpointSliceListPagingLimit(100) httpRoutes .

[gjkim42]

mgrc.List always returns exactly 100 items which are a subset of the total resources. It ends up generating different attachedRoutes values, as each given subset changes every time.

[gjkim42]

I believe that we MUST not use continue token in a cached client. We can simply remove those paginations.

[czeslavo]

@gjkim42 Thank you for reporting and handling the fix yourself. I believe you can apply for a contributor T-shirt having a bug fixed.