search

Kong / kubernetes-ingress-controller

:gorilla: Kong for Kubernetes: The official Ingress Controller for Kubernetes.
https://docs.konghq.com/kubernetes-ingress-controller/
Apache License 2.0
2.22k stars 592 forks source link

issue tracking in performance test #5568

Open tao12345666333 opened 9 months ago

tao12345666333 commented 9 months ago

Is there an existing issue for this?

Problem Statement

During the large-scale performance testing, I have discovered some issues. This issue is used for recording purposes.

I used 100,000 sets of resources for testing, each set containing one Ingress, one consumer, one plugin, and one secret. So in fact, besides the basic environment, I have created an additional 400,000 resources.

Proposed Solution

No response

Additional information

No response

Acceptance Criteria

No response

tao12345666333 commented 9 months ago

too large body

KIC got the 413 response from Gateway.

2024/02/06 07:46:00 [error] 1286#0: *641 client intended to send too large body: 19907571 bytes, client: 10.244.2.2, server: kong_admin, request: "POST /config?check_hash=1&flatten_errors=1 HTTP/2.0", host: "10.244.1.10:8444"
10.244.2.2 - - [06/Feb/2024:07:46:00 +0000] "POST /config?check_hash=1&flatten_errors=1 HTTP/2.0" 413 148 "-" "Go-http-client/2.0"
10.244.2.2 - - [06/Feb/2024:07:46:04 +0000] "GET /status HTTP/2.0" 200 1183 "-" "Go-http-client/2.0"
10.244.2.2 - - [06/Feb/2024:07:46:07 +0000] "GET /status HTTP/2.0" 200 1183 "-" "Go-http-client/2.0"
2024/02/06 07:46:17 [error] 1286#0: *641 client intended to send too large body: 19907571 bytes, client: 10.244.2.2, server: kong_admin, request: "POST /config?check_hash=1&flatten_errors=1 HTTP/2.0", host: "10.244.1.10:8444"
10.244.2.2 - - [06/Feb/2024:07:46:17 +0000] "POST /config?check_hash=1&flatten_errors=1 HTTP/2.0" 413 148 "-" "Go-http-client/2.0"
10.244.2.2 - - [06/Feb/2024:07:46:17 +0000] "GET /status HTTP/2.0" 200 1183 "-" "Go-http-client/2.0"
10.244.2.2 - - [06/Feb/2024:07:46:21 +0000] "GET /status HTTP/2.0" 200 1183 "-" "Go-http-client/2.0"
10.244.2.2 - - [06/Feb/2024:07:46:27 +0000] "GET /status HTTP/2.0" 200 1183 "-" "Go-http-client/2.0"
2024/02/06 07:46:34 [error] 1286#0: *641 client intended to send too large body: 19907571 bytes, client: 10.244.2.2, server: kong_admin, request: "POST /config?check_hash=1&flatten_errors=1 HTTP/2.0", host: "10.244.1.10:8444"
tao12345666333 commented 9 months ago

Admin API request failed.

2024-02-06T06:53:07Z    error   admission-server    Failed to fetch consumer from kong  {"error": "making HTTP request: Get \"https://10.244.1.5:8444/consumers/consumer-key-auth-name-9480\": dial tcp 10.244.1.5:8444: connect: connection refused"}
2024-02-06T06:53:07Z    error   admission-server    Failed to run validation    {"error": "making HTTP request: Get \"https://10.244.1.5:8444/consumers/consumer-key-auth-name-9480\": dial tcp 10.244.1.5:8444: connect: connection refused"}
2024-02-06T06:53:07Z    error   admission-server    Failed to run validation    {"error": "making HTTP request: Post \"https://10.244.1.5:8444/schemas/plugins/validate\": dial tcp 10.244.1.5:8444: connect: connection refused"}
2024-02-06T06:53:07Z    error   admission-server    Failed to fetch consumer from kong  {"error": "making HTTP request: Get \"https://10.244.1.5:8444/consumers/consumer-key-auth-name-9481\": dial tcp 10.244.1.5:8444: connect: connection refused"}
2024-02-06T06:53:07Z    error   admission-server    Failed to run validation    {"error": "making HTTP request: Get \"https://10.244.1.5:8444/consumers/consumer-key-auth-name-9481\": dial tcp 10.244.1.5:8444: connect: connection refused"}
2024-02-06T06:53:07Z    error   admission-server    Failed to run validation    {"error": "making HTTP request: Post \"https://10.244.1.5:8444/schemas/plugins/validate\": dial tcp 10.244.1.5:8444: connect: connection refused"}
2024-02-06T06:53:07Z    error   admission-server    Failed to fetch consumer from kong  {"error": "making HTTP request: Get \"https://10.244.1.5:8444/consumers/consumer-key-auth-name-9482\": dial tcp 10.244.1.5:8444: connect: connection refused"}
2024-02-06T06:53:07Z    error   admission-server    Failed to run validation    {"error": "making HTTP request: Get \"https://10.244.1.5:8444/consumers/consumer-key-auth-name-9482\": dial tcp 10.244.1.5:8444: connect: connection refused"}
2024-02-06T06:53:08Z    error   admission-server    Failed to run validation    {"error": "making HTTP request: Post \"https://10.244.1.5:8444/schemas/plugins/validate\": dial tcp 10.244.1.5:8444: connect: connection refused"}
2024-02-06T06:53:08Z    error   admission-server    Failed to fetch consumer from kong  {"error": "making HTTP request: Get \"https://10.244.1.5:8444/consumers/consumer-key-auth-name-9483\": dial tcp 10.244.1.5:8444: connect: connection refused"}
2024-02-06T06:53:08Z    error   admission-server    Failed to run validation    {"error": "making HTTP request: Get \"https://10.244.1.5:8444/consumers/consumer-key-auth-name-9483\": dial tcp 10.244.1.5:8444: connect: connection refused"}
2024-02-06T06:53:08Z    error   admission-server    Failed to run validation    {"error": "making HTTP request: Post \"https://10.244.1.5:8444/schemas/plugins/validate\": dial tcp 10.244.1.5:8444: connect: connection refused"}
2024-02-06T06:53:08Z    error   admission-server    Failed to fetch consumer from kong  {"error": "making HTTP request: Get \"https://10.244.1.5:8444/consumers/consumer-key-auth-name-9484\": dial tcp 10.244.1.5:8444: connect: connection refused"}
2024-02-06T06:53:08Z    error   admission-server    Failed to run validation    {"error": "making HTTP request: Get \"https://10.244.1.5:8444/consumers/consumer-key-auth-name-9484\": dial tcp 10.244.1.5:8444: connect: connection refused"}
2024-02-06T06:53:08Z    info    controllers.KongAdminAPIService Reconciling Admin API EndpointSlice {"v": 0, "namespace": "kong", "name": "kong-gateway-admin-648qw"}
2024-02-06T06:53:10Z    info    controllers.KongAdminAPIService Reconciling Admin API EndpointSlice {"v": 0, "namespace": "kong", "name": "kong-gateway-admin-648qw"}
2024-02-06T06:53:37Z    error   Exceeded Kong API timeout, consider increasing --proxy-timeout-seconds  {"url": "https://10.244.1.5:8444"}
2024-02-06T06:53:52Z    info    Successfully synced configuration to Kong   {"url": "https://10.244.1.7:8444", "update_strategy": "InMemory", "v": 0}
2024-02-06T06:53:52Z    error   dataplane-synchronizer  Could not update kong admin {"error": "performing update for https://10.244.1.5:8444 failed: failed posting new config to /config: making HTTP request: Post \"https://10.244.1.5:8444/config?check_hash=1&flatten_errors=1\": context deadline exceeded"}

This error is caused by the abnormal exit of Kong Gateway and the creation of a new Pod.

KIC will automatically coordinate the endpoints of Kong admin API.

$ kubectl  -n kong get pods
NAME                              READY   STATUS                   RESTARTS   AGE
kong-controller-69b6d7865-rqdcx   1/1     Running                  0          114m
kong-gateway-f849f9c59-2p4l6      0/1     Completed                0          61m
kong-gateway-f849f9c59-bdwzb      0/1     Completed                0          66m
kong-gateway-f849f9c59-jz5cf      0/1     Completed                0          64m
kong-gateway-f849f9c59-n4mwj      0/1     ContainerStatusUnknown   1          57m
kong-gateway-f849f9c59-sn85g      1/1     Running                  0          33m
tao12345666333 commented 9 months ago

too many nginx_http_lua_regex_cache_max_entries

10.244.2.2 - - [06/Feb/2024:10:47:29 +0000] "POST /schemas/routes/validate HTTP/2.0" 200 42 "-" "Go-http-client/2.0"
10.244.2.2 - - [06/Feb/2024:10:47:30 +0000] "POST /schemas/plugins/validate HTTP/2.0" 200 42 "-" "Go-http-client/2.0"
10.244.2.2 - - [06/Feb/2024:10:47:30 +0000] "GET /consumers/consumer-key-auth-name-52742 HTTP/2.0" 404 23 "-" "Go-http-client/2.0"
10.244.2.2 - - [06/Feb/2024:10:47:30 +0000] "POST /schemas/routes/validate HTTP/2.0" 200 42 "-" "Go-http-client/2.0"
10.244.2.2 - - [06/Feb/2024:10:47:30 +0000] "POST /schemas/plugins/validate HTTP/2.0" 200 42 "-" "Go-http-client/2.0"
2024/02/06 10:47:30 [warn] 1285#0: *153 [lua] traditional.lua:1521: new(): the 'nginx_http_lua_regex_cache_max_entries' setting is set to 8192 but there are 15112 regex paths configured. This may lead to performance issue due to regex cache trashing. Consider increasing the 'nginx_http_lua_regex_cache_max_entries' to at least 30224, context: ngx.timer
10.244.2.2 - - [06/Feb/2024:10:47:30 +0000] "GET /consumers/consumer-key-auth-name-52743 HTTP/2.0" 404 23 "-" "Go-http-client/2.0"
10.244.2.2 - - [06/Feb/2024:10:47:31 +0000] "POST /schemas/routes/validate HTTP/2.0" 200 42 "-" "Go-http-client/2.0"
10.244.2.2 - - [06/Feb/2024:10:47:31 +0000] "POST /schemas/plugins/validate HTTP/2.0" 200 42 "-" "Go-http-client/2.0"
2024/02/06 10:47:31 [notice] 1285#0: *153 [lua] init.lua:259: purge(): [DB cache] purging (local) cache, context: ngx.timer
2024/02/06 10:47:31 [notice] 1285#0: *153 [lua] init.lua:259: purge(): [DB cache] purging (local) cache, context: ngx.timer
10.244.2.2 - - [06/Feb/2024:10:47:31 +0000] "GET /status HTTP/2.0" 200 1177 "-" "Go-http-client/2.0"
tao12345666333 commented 9 months ago

Under high load conditions, the admission erroneously rejected the request.

When there are nearly 120,000 resources in the cluster, query requests take some time. In my testing environment, I found that in this scenario, admission erroneously denied a series of creation requests for kongconsumer, resulting in a series of consumers unable to be created successfully.

ingress.networking.k8s.io/test-ingress-39999 created         
kongplugin.configuration.konghq.com/auth-plugin-39999 created
secret/consumer-key-auth-secret-39999 created                
Error from server: error when creating "30000.yaml": admission webhook "validations.kong.konghq.com" denied the request: consumer referenced non-existent credentials secret: 
Secret "consumer-key-auth-secret-30000" not found
Error from server: error when creating "30000.yaml": admission webhook "validations.kong.konghq.com" denied the request: consumer referenced non-existent credentials secret: 
Secret "consumer-key-auth-secret-30001" not found
Error from server: error when creating "30000.yaml": admission webhook "validations.kong.konghq.com" denied the request: consumer referenced non-existent credentials secret: 
Secret "consumer-key-auth-secret-30002" not found

In fact, the secret does exist.

$ time kubectl  get secret consumer-key-auth-secret-30000 

NAME                             TYPE     DATA   AGE
consumer-key-auth-secret-30000   Opaque   1      128m

real    2m48.831s
user    0m0.119s
sys     0m0.665s