Kong 3.7.1 Exponential memory growth over time.

muh-kamran commented 3 months ago

Is there an existing issue for this?

[X] I have searched the existing issues

Kong version (`$ kong version`)

3.7.1

Current Behavior

We recently upgraded from 3.3 to 3.71. In last 8 days the memory of proxy grew from 550MB to 1.03GB and eventually got OOM killed. we didn't saw this issue on 3.3.

Nothing changed from our end.. just version upgrade. Please inform me if you need more details. Additionally, I would appreciate any recommendations for debugging steps, areas to investigate, or configuration changes that have successfully addressed similar issues for you.

Expected Behavior

kong 3.3 its much stable:

Steps To Reproduce

No response

Anything else?

only error message in logs i see is: [error] 1437#0: *582289 [lua] http.lua:962: request_uri(): closed, context: ngx.timer

nowNick commented 3 months ago

Hi @muh-kamran ! That looks concerning! We should definitely investigate further. Could you please share a little bit more info about your configuration?

What is your deployment topology? Traditional / declarative / hybrid mode?
How many services / routes / consumers / other entities are defined?
What plugins are enabled? How many plugin instances?

Sharing this data will allow us to create a similar environment to yours and potentially replicate this issue.

muh-kamran commented 3 months ago

we are using kong as ingress as well as gateway. In gateway we push declarative configuration through admin api.The issue is on both but very aggressive on kong ingress. let share the detail of ingress.

Helm chart: https://github.com/Kong/charts/tree/main/charts/kong version: 2.38.0 Max current rps is : 200 values file:

deployment:
  serviceAccount:
    automountServiceAccountToken: true

readinessProbe:
  httpGet:
    path: /status/ready
    port: status
    scheme: HTTP

livenessProbe:
  httpGet:
    path: /status/ready
    port: status
    scheme: HTTP

customEnv:
  MEM_CACHE_SIZE: 256m
  LUA_SOCKET_POOL_SIZE: 200
  nginx_main_worker_rlimit_nofile: 8192
  nginx_events_worker_connections: 16384
  LMDB_MAP_SIZE: 500m
env:
  PROXY_LISTEN: 0.0.0.0:8000 reuseport, 0.0.0.0:8443 reuseport ssl
  TRUSTED_IPS: 0.0.0.0/0,::/0
  REAL_IP_RECURSIVE: "on"
  REAL_IP_HEADER: "X-Forwarded-For"
  nginx_worker_processes: 2
  plugins: "bundled"
  MEM_CACHE_SIZE: 256m
  LUA_SOCKET_POOL_SIZE: 200
  nginx_main_worker_rlimit_nofile: 8192
  nginx_events_worker_connections: 16384
  LMDB_MAP_SIZE: 500m

proxy:
  type: ClusterIP

namespace: "kong-ingress"
image:
  repository: docker-mirror.control.sre.red/library/kong
  tag: "3.7.1"
  effectiveSemver:
  pullPolicy: IfNotPresent  
autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 5

resources:
    requests:
      cpu: 3
      memory: 5Gi
    limits:
      cpu: 3
      memory: 5Gi
ingressController:
    # ingress controller container resources
  resources:
      requests:
        cpu: 100m
        memory: 256Mi
podAnnotations:
  prometheus.io/scrape: 'true'
  prometheus.io/port: '8100'

ingressController:
  image:
    repository: docker-mirror.control.sre.red/kong/kubernetes-ingress-controller
    tag: "3.1"
    effectiveSemver:
  installCRDs: false
  env:
    enable_reverse_sync: false
    update_status: false
    PROXY_SYNC_SECONDS: 600

podDisruptionBudget:
  enabled: true

podSecurityPolicy:
  enabled: false
  spec:
    privileged: false
    fsGroup:
      rule: RunAsAny
    runAsUser:
      rule: RunAsAny
    runAsGroup:
      rule: RunAsAny
    seLinux:
      rule: RunAsAny
    supplementalGroups:
      rule: RunAsAny
    volumes:
      - 'configMap'
      - 'secret'
      - 'emptyDir'
      - 'projected'
      - 'downwardAPI'
      - 'persistentVolumeClaim'
    allowPrivilegeEscalation: false
    hostNetwork: false
    hostIPC: false
    hostPID: false
    readOnlyRootFilesystem: true
    allowedCapabilities:
      - NET_ADMIN
      - NET_RAW

One thing to note this configuration have impact on memory:

ingressController:
  image:
    repository: docker-mirror.control.sre.red/kong/kubernetes-ingress-controller
    tag: "3.1"
    effectiveSemver:
  installCRDs: false
  env:
    enable_reverse_sync: false
    update_status: false
    PROXY_SYNC_SECONDS: 600

when enable_reverse_sync: true and PROXY_SYNC_SECONDS: 30 .. the proxy got OOM killed immediately in few hours. setting it to enable_reverse_sync: false and PROXY_SYNC_SECONDS: 600 or so .. The memory grow slowly and proxy got OOM in days. Could be memory leak on configuration push to admin api.

Plugins enabled:

request-termination
prometheus

services: 70 routes: 119

surenraju-careem commented 3 months ago

We found similar issue kong 3.0, just by frequent GET requests to admin API. But it was fixed in 3.2 https://github.com/Kong/kong/issues/10782

chobits commented 3 months ago

hi @muh-kamran We need some detailed information to help us identify where the memory consumption is happening. You could run the following commands to get the stats .

whether the memory is consumed in kong's lua land from admin_listen
- if you are using hybrid mode to deploy, and you want to get info from Data plane, you need to enable status_listen, and run command curl <status_listen_ip_port>/status

$ curl localhost:8001/status

show us the heap map of kong(nginx worker process)

$ cat /proc/<nginx-worker-process-id>/maps

show the open fd of kong (nginx worker process) to see whether there are some leaked connection

lsof -p <nginx-worker-process-id>

or 

netstat -np|grep nginx

or any information which you think is helpful for troubleshooting

bungle commented 3 months ago

The growth here doesn't look exponential to me, but quite linear. Also at the end it seems to drop (is that the OOM kill). Am I correct? OOM kill seems to happen pretty close to 1.1GB whereas 3.3 you got somewhere around ~1GB? Or do you mean the hikes in graph look like double each time?

muh-kamran commented 3 months ago

Yes, it's experiencing OOM. We increased the memory to 5GB, but it still hits the OOM state in about 2 hours. This time, the memory grow quickly because we changed these two flags on the controller to:

    enable_reverse_sync: true
    proxy_sync_seconds: 2

muh-kamran commented 3 months ago

I'll provide the requested details. However, before doing so, I was testing it on a local kind cluster and observed the same pattern with a single ingress object. Here are the steps to reproduce:

kind create cluster --name kong-debug
helm repo add stable https://charts.helm.sh/stable
helm repo add kong https://charts.konghq.com
helm repo update
kubectl create namespace monitoring
helm install kind-prometheus prometheus-community/kube-prometheus-stack --namespace monitoring --set prometheus.service.nodePort=30000 --set prometheus.service.type=NodePort --set grafana.service.nodePort=31000 --set grafana.service.type=NodePort --set alertmanager.service.nodePort=32000 --set alertmanager.service.type=NodePort --set prometheus-node-exporter.service.nodePort=32001 --set prometheus-node-exporter.service.type=NodePort

values.yaml

deployment:
prefixDir:
sizeLimit: 1Gi
serviceAccount:
automountServiceAccountToken: true
image:
repository: docker-mirror.control.sre.red/library/kong
tag: "3.7.1"
effectiveSemver:
pullPolicy: IfNotPresent
readinessProbe:
httpGet:
path: /status/ready
port: status
scheme: HTTP
livenessProbe:
httpGet:
path: /status/ready
port: status
scheme: HTTP
env:
PROXY_LISTEN: 0.0.0.0:8000 reuseport, 0.0.0.0:8443 reuseport ssl
TRUSTED_IPS: 0.0.0.0/0,::/0
REAL_IP_RECURSIVE: "on"
REAL_IP_HEADER: X-Forwarded-For
nginx_worker_processes: 2
MEM_CACHE_SIZE: 256m
LUA_SOCKET_POOL_SIZE: 200
nginx_main_worker_rlimit_nofile: 8192
nginx_events_worker_connections: 16384
LMDB_MAP_SIZE: 500m
proxy_access_log: 'off'
customenv:
MEM_CACHE_SIZE: 256m
LUA_SOCKET_POOL_SIZE: 200
nginx_main_worker_rlimit_nofile: 8192
nginx_events_worker_connections: 16384
LMDB_MAP_SIZE: 500m
proxy:
type: ClusterIP
namespace: "kong-ingress"
autoscaling:
enabled: true
minReplicas: 1
maxReplicas: 1
resources:
requests:
  cpu: 0.5
  memory: 1Gi
limits:
  cpu: 1
  memory: 2Gi
podAnnotations:
prometheus.io/scrape: 'true'
prometheus.io/port: '8100,10255'
lifecycle:
preStop:
exec:
  command:
    - kong
    - quit
    - '--wait=60'
    - '--timeout=15'
terminationGracePeriodSeconds: 90
ingressController:
image:
repository: docker-mirror.control.sre.red/kong/kubernetes-ingress-controller
tag: "3.1"
effectiveSemver:
installCRDs: false
env:
enable_reverse_sync: true
proxy_sync_seconds: 2
update_status: false
log_level: debug
resources:
requests:
  cpu: 1
  memory: 1Gi
limits:
  cpu: 1
  memory: 1Gi
podDisruptionBudget:
enabled: true
podSecurityPolicy:
enabled: false
spec:
privileged: false
fsGroup:
  rule: RunAsAny
runAsUser:
  rule: RunAsAny
runAsGroup:
  rule: RunAsAny
seLinux:
  rule: RunAsAny
supplementalGroups:
  rule: RunAsAny
volumes:
  - 'configMap'
  - 'secret'
  - 'emptyDir'
  - 'projected'
  - 'downwardAPI'
  - 'persistentVolumeClaim'
allowPrivilegeEscalation: false
hostNetwork: false
hostIPC: false
hostPID: false
readOnlyRootFilesystem: true
allowedCapabilities:
  - NET_ADMIN
  - NET_RAW

helm install --values values.yaml kong/kong --generate-name
helm repo add ealenn https://ealenn.github.io/charts
helm repo update
helm install --set ingress.enable=true --name echoserver ealenn/echo-server

ingress.yaml


apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: echo1
namespace: default
labels:
app.kubernetes.io/managed-by: Helm
annotations:
configuration.konghq.com: echo1
konghq.com/request-buffering: 'false'
konghq.com/response-buffering: 'false'
kubernetes.io/ingress.class: kong
meta.helm.sh/release-name: echo1
meta.helm.sh/release-namespace: echo1
plugins.konghq.com: echo1-request-transformer,echo1-response-transformer
spec:
rules:
- host: echo1.dev.test.com
  http:
    paths:
      - path: /
        pathType: ImplementationSpecific
        backend:
          service:
            name: echoserver-echo-server
            port:
              number: 80
- host: echo1.alfa.dev.test.com
  http:
    paths:
      - path: /
        pathType: ImplementationSpecific
        backend:
          service:
            name: echoserver-echo-server
            port:
              number: 80



- Next i port forward to grafana and run `sum(container_memory_working_set_bytes{container=~"proxy", container!="", image!=""}) by (pod,container)`
<img width="1470" alt="Screenshot 2024-08-27 at 8 29 02 PM" src="https://github.com/user-attachments/assets/5be0e67a-06bc-4e77-9173-e0283e6702c8">

muh-kamran commented 3 months ago

curl localhost:8001/status

{"memory":{"lua_shared_dicts":{"kong_healthchecks":{"allocated_slabs":"0.04 MiB","capacity":"5.00 MiB"},"kong_cluster_events":{"allocated_slabs":"0.04 MiB","capacity":"5.00 MiB"},"kong_rate_limiting_counters":{"allocated_slabs":"0.08 MiB","capacity":"12.00 MiB"},"kong_core_db_cache":{"allocated_slabs":"1.66 MiB","capacity":"256.00 MiB"},"kong_core_db_cache_miss":{"allocated_slabs":"0.08 MiB","capacity":"12.00 MiB"},"kong_db_cache":{"allocated_slabs":"1.50 MiB","capacity":"256.00 MiB"},"kong_db_cache_miss":{"allocated_slabs":"0.08 MiB","capacity":"12.00 MiB"},"kong_secrets":{"allocated_slabs":"0.04 MiB","capacity":"5.00 MiB"},"prometheus_metrics":{"allocated_slabs":"0.07 MiB","capacity":"5.00 MiB"},"kong":{"allocated_slabs":"0.04 MiB","capacity":"5.00 MiB"},"kong_locks":{"allocated_slabs":"0.06 MiB","capacity":"8.00 MiB"}},"workers_lua_vms":[{"http_allocated_gc":"116.97 MiB","pid":1346},{"http_allocated_gc":"167.17 MiB","pid":1347}]},"configuration_hash":"c0b9e5b8a8e08eb4b8d7499b68af3b4d","server":{"connections_reading":0,"total_requests":16216,"connections_writing":3,"connections_handled":4084,"connections_waiting":3,"connections_accepted":4084,"connections_active":6}}

cat /proc/worker1/maps (there are two workers)


5558ab985000-5558ab9c0000 r--p 00000000 103:10 67370767                  /usr/local/openresty/nginx/sbin/nginx
5558ab9c0000-5558abbbe000 r-xp 0003b000 103:10 67370767                  /usr/local/openresty/nginx/sbin/nginx
5558abbbe000-5558abca2000 r--p 00239000 103:10 67370767                  /usr/local/openresty/nginx/sbin/nginx
5558abca2000-5558abca5000 r--p 0031c000 103:10 67370767                  /usr/local/openresty/nginx/sbin/nginx
5558abca5000-5558abcc4000 rw-p 0031f000 103:10 67370767                  /usr/local/openresty/nginx/sbin/nginx
5558abcc4000-5558abcf5000 rw-p 00000000 00:00 0 
5558ad872000-5558adb0a000 rw-p 00000000 00:00 0                          [heap]
5558adb0a000-55593fc02000 rw-p 00000000 00:00 0                          [heap]
7ff60b23f000-7ff60b340000 rw-p 00000000 00:00 0 
7ff60b340000-7ff60ca00000 rw-p 00000000 00:00 0 
7ff60ca00000-7ff62be00000 r--s 00000000 103:10 51481867                  /kong_prefix/dbless.lmdb/data.mdb
7ff62be00000-7ff62be50000 r-xp 00000000 00:00 0 
7ff62be50000-7ff62bf40000 r-xp 00000000 00:00 0 
7ff62bf5e000-7ff62c09e000 rw-p 00000000 00:00 0 
7ff62c09f000-7ff62c43f000 rw-p 00000000 00:00 0 
7ff62c452000-7ff62cb92000 rw-p 00000000 00:00 0 
7ff62cc1e000-7ff62cc9e000 rw-p 00000000 00:00 0 
7ff62cc9e000-7ff62ccae000 r-xp 00000000 00:00 0 
7ff62ccfd000-7ff62cebd000 rw-p 00000000 00:00 0 
7ff62cecc000-7ff62cf4c000 rw-p 00000000 00:00 0 
7ff62cf4c000-7ff62cf5c000 r-xp 00000000 00:00 0 
7ff62cf5d000-7ff62d05d000 rw-p 00000000 00:00 0 
7ff62d05d000-7ff62d09d000 r-xp 00000000 00:00 0 
7ff62d09d000-7ff62d19d000 rw-p 00000000 00:00 0 
7ff62d19e000-7ff62d1de000 r-xp 00000000 00:00 0 
7ff62d1fe000-7ff62d20e000 r-xp 00000000 00:00 0 
7ff62d21d000-7ff62d27d000 rw-p 00000000 00:00 0 
7ff62d27e000-7ff62d28e000 r-xp 00000000 00:00 0 
7ff62d28e000-7ff62d2ae000 rw-p 00000000 00:00 0 
7ff62d2ae000-7ff62d2be000 r-xp 00000000 00:00 0 
7ff62d2be000-7ff62d53e000 rw-p 00000000 00:00 0 
7ff62d53f000-7ff62d55f000 rw-p 00000000 00:00 0 
7ff62d55f000-7ff62d5af000 r-xp 00000000 00:00 0 
7ff62d5af000-7ff62d60f000 rw-p 00000000 00:00 0 
7ff62d610000-7ff62d670000 rw-p 00000000 00:00 0 
7ff62d670000-7ff62d680000 r-xp 00000000 00:00 0 
7ff62d680000-7ff62d6e0000 rw-p 00000000 00:00 0 
7ff62d6e1000-7ff62d761000 rw-p 00000000 00:00 0 
7ff62d762000-7ff62d802000 rw-p 00000000 00:00 0 
7ff62d803000-7ff62d823000 rw-p 00000000 00:00 0 
7ff62d824000-7ff62d844000 rw-p 00000000 00:00 0 
7ff62d844000-7ff62d854000 r-xp 00000000 00:00 0 
7ff62d854000-7ff62d874000 rw-p 00000000 00:00 0 
7ff62d874000-7ff62d894000 r-xp 00000000 00:00 0 
7ff62d895000-7ff62d8a5000 r-xp 00000000 00:00 0 
7ff62d8a5000-7ff62d8e5000 rw-p 00000000 00:00 0 
7ff62d8e5000-7ff62d8f5000 r-xp 00000000 00:00 0 
7ff62d8f5000-7ff62d9b5000 rw-p 00000000 00:00 0 
7ff62d9b5000-7ff62d9c5000 r-xp 00000000 00:00 0 
7ff62d9c5000-7ff62e125000 rw-p 00000000 00:00 0 
7ff62e126000-7ff62e306000 rw-p 00000000 00:00 0 
7ff62e315000-7ff62e355000 r-xp 00000000 00:00 0 
7ff62e355000-7ff62e795000 rw-p 00000000 00:00 0 
7ff62e796000-7ff62e7a6000 r-xp 00000000 00:00 0 
7ff62e7a6000-7ff62f766000 rw-p 00000000 00:00 0 
7ff62f775000-7ff630175000 rw-p 00000000 00:00 0 
7ff630184000-7ff630584000 rw-p 00000000 00:00 0 
7ff630585000-7ff631085000 rw-p 00000000 00:00 0 
7ff631086000-7ff631096000 r-xp 00000000 00:00 0 
7ff631096000-7ff6312f6000 rw-p 00000000 00:00 0 
7ff6312f7000-7ff631307000 r-xp 00000000 00:00 0 
7ff631307000-7ff631567000 rw-p 00000000 00:00 0 
7ff631567000-7ff631577000 r-xp 00000000 00:00 0 
7ff631577000-7ff631f57000 rw-p 00000000 00:00 0 
7ff631f57000-7ff631f67000 r-xp 00000000 00:00 0 
7ff631f67000-7ff6323e7000 rw-p 00000000 00:00 0 
7ff6323f6000-7ff632bd6000 rw-p 00000000 00:00 0 
7ff632bd7000-7ff632be7000 r-xp 00000000 00:00 0 
7ff632be7000-7ff6332e7000 rw-p 00000000 00:00 0 
7ff6332f6000-7ff633316000 r-xp 00000000 00:00 0 
7ff633317000-7ff63561e000 rw-p 00000000 00:00 0 
7ff63561e000-7ff63571e000 rw-s 00000000 00:01 130                        /dev/zero (deleted)
7ff63571e000-7ff63611e000 rw-s 00000000 00:01 129                        /dev/zero (deleted)
7ff63611e000-7ff636ebe000 rw-p 00000000 00:00 0 
7ff636ebf000-7ff636ecf000 r-xp 00000000 00:00 0 
7ff636ecf000-7ff63799e000 rw-p 00000000 00:00 0 
7ff6379ad000-7ff6379ed000 r-xp 00000000 00:00 0 
7ff6379ed000-7ff637c5c000 rw-p 00000000 00:00 0 
7ff637c5d000-7ff637c9d000 rw-p 00000000 00:00 0 
7ff637c9d000-7ff637d9e000 rw-p 00000000 00:00 0 
7ff637d9e000-7ff637dae000 rwxp 00000000 00:00 0 
7ff637dae000-7ff637e0f000 rw-p 00000000 00:00 0 
7ff637e0f000-7ff637e11000 r--p 00000000 103:10 218507063                 /usr/lib/x86_64-linux-gnu/libyaml-0.so.2.0.6
7ff637e11000-7ff637e2a000 r-xp 00002000 103:10 218507063                 /usr/lib/x86_64-linux-gnu/libyaml-0.so.2.0.6
7ff637e2a000-7ff637e2e000 r--p 0001b000 103:10 218507063                 /usr/lib/x86_64-linux-gnu/libyaml-0.so.2.0.6
7ff637e2e000-7ff637e2f000 r--p 0001e000 103:10 218507063                 /usr/lib/x86_64-linux-gnu/libyaml-0.so.2.0.6
7ff637e2f000-7ff637e30000 rw-p 0001f000 103:10 218507063                 /usr/lib/x86_64-linux-gnu/libyaml-0.so.2.0.6
7ff637e30000-7ff638000000 rw-p 00000000 00:00 0 
7ff638000000-7ff648000000 rw-s 00000000 00:01 9456                       /dev/zero (deleted)
7ff648000000-7ff658000000 rw-s 00000000 00:01 9454                       /dev/zero (deleted)
7ff658000000-7ff658021000 rw-p 00000000 00:00 0 
7ff658021000-7ff65c000000 ---p 00000000 00:00 0 
7ff65c00d000-7ff65c00f000 r--p 00000000 103:10 218515797                 /usr/local/lib/lua/5.1/yaml.so
7ff65c00f000-7ff65c013000 r-xp 00002000 103:10 218515797                 /usr/local/lib/lua/5.1/yaml.so
7ff65c013000-7ff65c014000 r--p 00006000 103:10 218515797                 /usr/local/lib/lua/5.1/yaml.so
7ff65c014000-7ff65c015000 r--p 00006000 103:10 218515797                 /usr/local/lib/lua/5.1/yaml.so
7ff65c015000-7ff65c016000 rw-p 00007000 103:10 218515797                 /usr/local/lib/lua/5.1/yaml.so
7ff65c016000-7ff65c196000 rw-p 00000000 00:00 0 
7ff65c1a5000-7ff65c205000 rw-p 00000000 00:00 0 
7ff65c206000-7ff65c456000 rw-p 00000000 00:00 0 
7ff65c45e000-7ff65c463000 r--p 00000000 103:10 252256718                 /usr/local/lib/lua/5.1/socket/core.so
7ff65c463000-7ff65c46b000 r-xp 00005000 103:10 252256718                 /usr/local/lib/lua/5.1/socket/core.so
7ff65c46b000-7ff65c46e000 r--p 0000d000 103:10 252256718                 /usr/local/lib/lua/5.1/socket/core.so
7ff65c46e000-7ff65c46f000 r--p 0000f000 103:10 252256718                 /usr/local/lib/lua/5.1/socket/core.so
7ff65c46f000-7ff65c470000 rw-p 00010000 103:10 252256718                 /usr/local/lib/lua/5.1/socket/core.so
7ff65c470000-7ff65c473000 r--p 00000000 103:10 218515795                 /usr/local/lib/lua/5.1/pb.so
7ff65c473000-7ff65c481000 r-xp 00003000 103:10 218515795                 /usr/local/lib/lua/5.1/pb.so
7ff65c481000-7ff65c485000 r--p 00011000 103:10 218515795                 /usr/local/lib/lua/5.1/pb.so
7ff65c485000-7ff65c486000 r--p 00014000 103:10 218515795                 /usr/local/lib/lua/5.1/pb.so
7ff65c486000-7ff65c487000 rw-p 00015000 103:10 218515795                 /usr/local/lib/lua/5.1/pb.so
7ff65c48a000-7ff65c48b000 r--p 00000000 103:10 218515791                 /usr/local/lib/lua/5.1/lsyslog.so
7ff65c48b000-7ff65c48c000 r-xp 00001000 103:10 218515791                 /usr/local/lib/lua/5.1/lsyslog.so
7ff65c48c000-7ff65c48d000 r--p 00002000 103:10 218515791                 /usr/local/lib/lua/5.1/lsyslog.so
7ff65c48d000-7ff65c48e000 r--p 00002000 103:10 218515791                 /usr/local/lib/lua/5.1/lsyslog.so
7ff65c48e000-7ff65c48f000 rw-p 00003000 103:10 218515791                 /usr/local/lib/lua/5.1/lsyslog.so
7ff65c48f000-7ff65c492000 r--p 00000000 103:10 218515790                 /usr/local/lib/lua/5.1/lpeg.so
7ff65c492000-7ff65c49a000 r-xp 00003000 103:10 218515790                 /usr/local/lib/lua/5.1/lpeg.so
7ff65c49a000-7ff65c49d000 r--p 0000b000 103:10 218515790                 /usr/local/lib/lua/5.1/lpeg.so
7ff65c49d000-7ff65c49e000 r--p 0000d000 103:10 218515790                 /usr/local/lib/lua/5.1/lpeg.so
7ff65c49e000-7ff65c49f000 rw-p 0000e000 103:10 218515790                 /usr/local/lib/lua/5.1/lpeg.so
7ff65c49f000-7ff65c4bf000 rw-p 00000000 00:00 0 
7ff65c4c3000-7ff65c4d3000 r-xp 00000000 00:00 0 
7ff65c4d3000-7ff65c4d4000 r--p 00000000 103:10 218515792                 /usr/local/lib/lua/5.1/lua_pack.so
7ff65c4d4000-7ff65c4d6000 r-xp 00001000 103:10 218515792                 /usr/local/lib/lua/5.1/lua_pack.so
7ff65c4d6000-7ff65c4d7000 r--p 00003000 103:10 218515792                 /usr/local/lib/lua/5.1/lua_pack.so
7ff65c4d7000-7ff65c4d8000 r--p 00003000 103:10 218515792                 /usr/local/lib/lua/5.1/lua_pack.so
7ff65c4d8000-7ff65c4d9000 rw-p 00004000 103:10 218515792                 /usr/local/lib/lua/5.1/lua_pack.so
7ff65c4d9000-7ff65c4db000 r--p 00000000 103:10 218515789                 /usr/local/lib/lua/5.1/lfs.so
7ff65c4db000-7ff65c4dd000 r-xp 00002000 103:10 218515789                 /usr/local/lib/lua/5.1/lfs.so
7ff65c4dd000-7ff65c4de000 r--p 00004000 103:10 218515789                 /usr/local/lib/lua/5.1/lfs.so
7ff65c4de000-7ff65c4df000 r--p 00004000 103:10 218515789                 /usr/local/lib/lua/5.1/lfs.so
7ff65c4df000-7ff65c4e0000 rw-p 00005000 103:10 218515789                 /usr/local/lib/lua/5.1/lfs.so
7ff65c4e0000-7ff65c500000 rw-p 00000000 00:00 0 
7ff65c500000-7ff65ca00000 rw-s 00000000 00:01 9459                       /dev/zero (deleted)
7ff65ca00000-7ff65cf00000 rw-s 00000000 00:01 9458                       /dev/zero (deleted)
7ff65cf00000-7ff65db00000 rw-s 00000000 00:01 9457                       /dev/zero (deleted)
7ff65db00000-7ff65e700000 rw-s 00000000 00:01 9455                       /dev/zero (deleted)
7ff65e700000-7ff65f300000 rw-s 00000000 00:01 9453                       /dev/zero (deleted)
7ff65f300000-7ff65f800000 rw-s 00000000 00:01 9452                       /dev/zero (deleted)
7ff65f800000-7ff660000000 rw-s 00000000 00:01 9450                       /dev/zero (deleted)
7ff660000000-7ff660021000 rw-p 00000000 00:00 0 
7ff660021000-7ff664000000 ---p 00000000 00:00 0 
7ff664003000-7ff664004000 r--p 00000000 103:10 218515793                 /usr/local/lib/lua/5.1/lua_system_constants.so
7ff664004000-7ff664005000 r-xp 00001000 103:10 218515793                 /usr/local/lib/lua/5.1/lua_system_constants.so
7ff664005000-7ff664006000 r--p 00002000 103:10 218515793                 /usr/local/lib/lua/5.1/lua_system_constants.so
7ff664006000-7ff664007000 r--p 00002000 103:10 218515793                 /usr/local/lib/lua/5.1/lua_system_constants.so
7ff664007000-7ff664008000 rw-p 00003000 103:10 218515793                 /usr/local/lib/lua/5.1/lua_system_constants.so
7ff664008000-7ff66400a000 r--p 00000000 103:10 235164598                 /usr/local/openresty/lualib/cjson.so
7ff66400a000-7ff66400f000 r-xp 00002000 103:10 235164598                 /usr/local/openresty/lualib/cjson.so
7ff66400f000-7ff664011000 r--p 00007000 103:10 235164598                 /usr/local/openresty/lualib/cjson.so
7ff664011000-7ff664012000 r--p 00008000 103:10 235164598                 /usr/local/openresty/lualib/cjson.so
7ff664012000-7ff664013000 rw-p 00009000 103:10 235164598                 /usr/local/openresty/lualib/cjson.so
7ff664013000-7ff6640b3000 rw-p 00000000 00:00 0 
7ff6640b3000-7ff6645b3000 rw-s 00000000 00:01 9451                       /dev/zero (deleted)
7ff6645b3000-7ff664ab3000 rw-s 00000000 00:01 9449                       /dev/zero (deleted)
7ff664ab3000-7ff664b13000 rw-p 00000000 00:00 0 
7ff664b13000-7ff664b14000 ---p 00000000 00:00 0 
7ff664b14000-7ff665314000 rw-p 00000000 00:00 0 
7ff665314000-7ff665315000 ---p 00000000 00:00 0 
7ff665315000-7ff665b19000 rw-p 00000000 00:00 0 
7ff665b19000-7ff665b1a000 r--p 00000000 103:10 251901754                 /usr/lib/x86_64-linux-gnu/libpthread.so.0
7ff665b1a000-7ff665b1b000 r-xp 00001000 103:10 251901754                 /usr/lib/x86_64-linux-gnu/libpthread.so.0
7ff665b1b000-7ff665b1c000 r--p 00002000 103:10 251901754                 /usr/lib/x86_64-linux-gnu/libpthread.so.0
7ff665b1c000-7ff665b1d000 r--p 00002000 103:10 251901754                 /usr/lib/x86_64-linux-gnu/libpthread.so.0
7ff665b1d000-7ff665b1e000 rw-p 00003000 103:10 251901754                 /usr/lib/x86_64-linux-gnu/libpthread.so.0
7ff665b1e000-7ff665b22000 r--p 00000000 103:10 167842508                 /opt/dynatrace/oneagent-paas/agent/bin/1.297.53.20240820-121511/linux-x86-64/libelf.so.1
7ff665b22000-7ff665b40000 r-xp 00004000 103:10 167842508                 /opt/dynatrace/oneagent-paas/agent/bin/1.297.53.20240820-121511/linux-x86-64/libelf.so.1
7ff665b40000-7ff665b49000 r--p 00022000 103:10 167842508                 /opt/dynatrace/oneagent-paas/agent/bin/1.297.53.20240820-121511/linux-x86-64/libelf.so.1
7ff665b49000-7ff665b4a000 r--p 0002a000 103:10 167842508                 /opt/dynatrace/oneagent-paas/agent/bin/1.297.53.20240820-121511/linux-x86-64/libelf.so.1
7ff665b4a000-7ff665b4b000 rw-p 0002b000 103:10 167842508                 /opt/dynatrace/oneagent-paas/agent/bin/1.297.53.20240820-121511/linux-x86-64/libelf.so.1
7ff665b4b000-7ff665b5d000 r--p 00000000 103:10 167842431                 /opt/dynatrace/oneagent-paas/agent/bin/1.297.53.20240820-121511/linux-x86-64/libdwarf.so
7ff665b5d000-7ff665bbd000 r-xp 00012000 103:10 167842431                 /opt/dynatrace/oneagent-paas/agent/bin/1.297.53.20240820-121511/linux-x86-64/libdwarf.so
7ff665bbd000-7ff665be5000 r--p 00072000 103:10 167842431                 /opt/dynatrace/oneagent-paas/agent/bin/1.297.53.20240820-121511/linux-x86-64/libdwarf.so
7ff665be5000-7ff665be6000 ---p 0009a000 103:10 167842431                 /opt/dynatrace/oneagent-paas/agent/bin/1.297.53.20240820-121511/linux-x86-64/libdwarf.so
7ff665be6000-7ff665be7000 r--p 0009a000 103:10 167842431                 /opt/dynatrace/oneagent-paas/agent/bin/1.297.53.20240820-121511/linux-x86-64/libdwarf.so
7ff665be7000-7ff665be9000 rw-p 0009b000 103:10 167842431                 /opt/dynatrace/oneagent-paas/agent/bin/1.297.53.20240820-121511/linux-x86-64/libdwarf.so
7ff665be9000-7ff665bea000 rw-p 00000000 00:00 0 
7ff665bea000-7ff665beb000 r--p 00000000 103:10 251901673                 /usr/lib/x86_64-linux-gnu/libdl.so.2
7ff665beb000-7ff665bec000 r-xp 00001000 103:10 251901673                 /usr/lib/x86_64-linux-gnu/libdl.so.2
7ff665bec000-7ff665bed000 r--p 00002000 103:10 251901673                 /usr/lib/x86_64-linux-gnu/libdl.so.2
7ff665bed000-7ff665bee000 r--p 00002000 103:10 251901673                 /usr/lib/x86_64-linux-gnu/libdl.so.2
7ff665bee000-7ff665bef000 rw-p 00003000 103:10 251901673                 /usr/lib/x86_64-linux-gnu/libdl.so.2
7ff665bef000-7ff665bf0000 r--p 00000000 103:10 251901756                 /usr/lib/x86_64-linux-gnu/librt.so.1
7ff665bf0000-7ff665bf1000 r-xp 00001000 103:10 251901756                 /usr/lib/x86_64-linux-gnu/librt.so.1
7ff665bf1000-7ff665bf2000 r--p 00002000 103:10 251901756                 /usr/lib/x86_64-linux-gnu/librt.so.1
7ff665bf2000-7ff665bf3000 r--p 00002000 103:10 251901756                 /usr/lib/x86_64-linux-gnu/librt.so.1
7ff665bf3000-7ff665bf4000 rw-p 00003000 103:10 251901756                 /usr/lib/x86_64-linux-gnu/librt.so.1
7ff665bf4000-7ff665d27000 r--p 00000000 103:10 167842499                 /opt/dynatrace/oneagent-paas/agent/bin/1.297.53.20240820-121511/linux-x86-64/liboneagentnginx.so
7ff665d27000-7ff666bd0000 r-xp 00133000 103:10 167842499                 /opt/dynatrace/oneagent-paas/agent/bin/1.297.53.20240820-121511/linux-x86-64/liboneagentnginx.so
7ff666bd0000-7ff667249000 r--p 00fdc000 103:10 167842499                 /opt/dynatrace/oneagent-paas/agent/bin/1.297.53.20240820-121511/linux-x86-64/liboneagentnginx.so
7ff667249000-7ff66730d000 r--p 01654000 103:10 167842499                 /opt/dynatrace/oneagent-paas/agent/bin/1.297.53.20240820-121511/linux-x86-64/liboneagentnginx.so
7ff66730d000-7ff667316000 rw-p 01718000 103:10 167842499                 /opt/dynatrace/oneagent-paas/agent/bin/1.297.53.20240820-121511/linux-x86-64/liboneagentnginx.so
7ff667316000-7ff6673bf000 rw-p 00000000 00:00 0 
7ff6673bf000-7ff6673c0000 r--p 00000000 00:00 0 
7ff6673c0000-7ff6674c6000 rw-p 00000000 00:00 0 
7ff6674c6000-7ff667560000 r--p 00000000 103:10 251901768                 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30
7ff667560000-7ff667671000 r-xp 0009a000 103:10 251901768                 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30
7ff667671000-7ff6676e0000 r--p 001ab000 103:10 251901768                 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30
7ff6676e0000-7ff6676e1000 ---p 0021a000 103:10 251901768                 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30
7ff6676e1000-7ff6676ec000 r--p 0021a000 103:10 251901768                 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30
7ff6676ec000-7ff6676ef000 rw-p 00225000 103:10 251901768                 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30
7ff6676ef000-7ff6676f2000 rw-p 00000000 00:00 0 
7ff6676f2000-7ff6676f5000 r--p 00000000 103:10 251901684                 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
7ff6676f5000-7ff66770c000 r-xp 00003000 103:10 251901684                 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
7ff66770c000-7ff667710000 r--p 0001a000 103:10 251901684                 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
7ff667710000-7ff667711000 r--p 0001d000 103:10 251901684                 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
7ff667711000-7ff667712000 rw-p 0001e000 103:10 251901684                 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
7ff667712000-7ff66773a000 r--p 00000000 103:10 251901659                 /usr/lib/x86_64-linux-gnu/libc.so.6
7ff66773a000-7ff6678cf000 r-xp 00028000 103:10 251901659                 /usr/lib/x86_64-linux-gnu/libc.so.6
7ff6678cf000-7ff667927000 r--p 001bd000 103:10 251901659                 /usr/lib/x86_64-linux-gnu/libc.so.6
7ff667927000-7ff667928000 ---p 00215000 103:10 251901659                 /usr/lib/x86_64-linux-gnu/libc.so.6
7ff667928000-7ff66792c000 r--p 00215000 103:10 251901659                 /usr/lib/x86_64-linux-gnu/libc.so.6
7ff66792c000-7ff66792e000 rw-p 00219000 103:10 251901659                 /usr/lib/x86_64-linux-gnu/libc.so.6
7ff66792e000-7ff66793b000 rw-p 00000000 00:00 0 
7ff66793b000-7ff66793d000 r--p 00000000 103:10 251901790                 /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
7ff66793d000-7ff66794e000 r-xp 00002000 103:10 251901790                 /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
7ff66794e000-7ff667954000 r--p 00013000 103:10 251901790                 /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
7ff667954000-7ff667955000 ---p 00019000 103:10 251901790                 /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
7ff667955000-7ff667956000 r--p 00019000 103:10 251901790                 /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
7ff667956000-7ff667957000 rw-p 0001a000 103:10 251901790                 /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
7ff667957000-7ff667e40000 r-xp 00000000 103:10 84442479                  /usr/local/kong/lib/libcrypto.so.3
7ff667e40000-7ff667e41000 ---p 004e9000 103:10 84442479                  /usr/local/kong/lib/libcrypto.so.3
7ff667e41000-7ff667ea0000 r--p 004e9000 103:10 84442479                  /usr/local/kong/lib/libcrypto.so.3
7ff667ea0000-7ff667ea3000 rw-p 00548000 103:10 84442479                  /usr/local/kong/lib/libcrypto.so.3
7ff667ea3000-7ff667ea9000 rw-p 00000000 00:00 0 
7ff667ea9000-7ff667f88000 r-xp 00000000 103:10 84442485                  /usr/local/kong/lib/libssl.so.3
7ff667f88000-7ff667f93000 r--p 000de000 103:10 84442485                  /usr/local/kong/lib/libssl.so.3
7ff667f93000-7ff667f97000 rw-p 000e9000 103:10 84442485                  /usr/local/kong/lib/libssl.so.3
7ff667f97000-7ff667fa5000 r--p 00000000 103:10 251901711                 /usr/lib/x86_64-linux-gnu/libm.so.6
7ff667fa5000-7ff668021000 r-xp 0000e000 103:10 251901711                 /usr/lib/x86_64-linux-gnu/libm.so.6
7ff668021000-7ff66807c000 r--p 0008a000 103:10 251901711                 /usr/lib/x86_64-linux-gnu/libm.so.6
7ff66807c000-7ff66807d000 r--p 000e4000 103:10 251901711                 /usr/lib/x86_64-linux-gnu/libm.so.6
7ff66807d000-7ff66807e000 rw-p 000e5000 103:10 251901711                 /usr/lib/x86_64-linux-gnu/libm.so.6
7ff66807e000-7ff668086000 r--p 00000000 103:10 135303769                 /usr/local/openresty/luajit/lib/libluajit-5.1.so.2.1.ROLLING
7ff668086000-7ff6680f3000 r-xp 00008000 103:10 135303769                 /usr/local/openresty/luajit/lib/libluajit-5.1.so.2.1.ROLLING
7ff6680f3000-7ff668108000 r--p 00075000 103:10 135303769                 /usr/local/openresty/luajit/lib/libluajit-5.1.so.2.1.ROLLING
7ff668108000-7ff668109000 ---p 0008a000 103:10 135303769                 /usr/local/openresty/luajit/lib/libluajit-5.1.so.2.1.ROLLING
7ff668109000-7ff66810b000 r--p 0008a000 103:10 135303769                 /usr/local/openresty/luajit/lib/libluajit-5.1.so.2.1.ROLLING
7ff66810b000-7ff66810c000 rw-p 0008c000 103:10 135303769                 /usr/local/openresty/luajit/lib/libluajit-5.1.so.2.1.ROLLING
7ff66810c000-7ff66810e000 r--p 00000000 103:10 251901668                 /usr/lib/x86_64-linux-gnu/libcrypt.so.1.1.0
7ff66810e000-7ff668122000 r-xp 00002000 103:10 251901668                 /usr/lib/x86_64-linux-gnu/libcrypt.so.1.1.0
7ff668122000-7ff66813b000 r--p 00016000 103:10 251901668                 /usr/lib/x86_64-linux-gnu/libcrypt.so.1.1.0
7ff66813b000-7ff66813c000 ---p 0002f000 103:10 251901668                 /usr/lib/x86_64-linux-gnu/libcrypt.so.1.1.0
7ff66813c000-7ff66813d000 r--p 0002f000 103:10 251901668                 /usr/lib/x86_64-linux-gnu/libcrypt.so.1.1.0
7ff66813d000-7ff66813e000 rw-p 00030000 103:10 251901668                 /usr/lib/x86_64-linux-gnu/libcrypt.so.1.1.0
7ff66813e000-7ff668146000 rw-p 00000000 00:00 0 
7ff668146000-7ff668148000 rw-s 00000000 103:10 51481866                  /kong_prefix/dbless.lmdb/lock.mdb
7ff668148000-7ff668153000 r--p 00000000 103:10 185076107                 /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so
7ff668153000-7ff6681f8000 r-xp 0000b000 103:10 185076107                 /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so
7ff6681f8000-7ff668239000 r--p 000b0000 103:10 185076107                 /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so
7ff668239000-7ff66823f000 r--p 000f0000 103:10 185076107                 /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so
7ff66823f000-7ff668240000 rw-p 000f6000 103:10 185076107                 /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so
7ff668240000-7ff66837a000 rw-p 00000000 00:00 0 
7ff66837a000-7ff66837e000 r--p 00000000 00:00 0                          [vvar]
7ff66837e000-7ff668380000 r-xp 00000000 00:00 0                          [vdso]
7ff668380000-7ff668382000 r--p 00000000 103:10 251901641                 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7ff668382000-7ff6683ac000 r-xp 00002000 103:10 251901641                 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7ff6683ac000-7ff6683b7000 r--p 0002c000 103:10 251901641                 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7ff6683b7000-7ff6683b8000 rw-s 00000000 00:01 131                        /dev/zero (deleted)
7ff6683b8000-7ff6683ba000 r--p 00037000 103:10 251901641                 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7ff6683ba000-7ff6683bc000 rw-p 00039000 103:10 251901641                 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7fff211d8000-7fff211fb000 rw-p 00000000 00:00 0                          [stack]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]

pmap -d 1347

00005558adb0a000 1995440 rw--- 0000000000000000 000:00000 [ anon ] The anon memory is high for this worker process its 1995440KB which is 1995.44MB since two worker becomes 3990MB = 4GB and total memory in graph is around 4.89.

Address           Kbytes Mode  Offset           Device    Mapping
00005558ab985000     236 r---- 0000000000000000 103:00010 nginx
00005558ab9c0000    2040 r-x-- 000000000003b000 103:00010 nginx
00005558abbbe000     912 r---- 0000000000239000 103:00010 nginx
00005558abca2000      12 r---- 000000000031c000 103:00010 nginx
00005558abca5000     124 rw--- 000000000031f000 103:00010 nginx
00005558abcc4000     196 rw--- 0000000000000000 000:00000   [ anon ]
00005558ad872000    2656 rw--- 0000000000000000 000:00000   [ anon ]
00005558adb0a000 1995440 rw--- 0000000000000000 000:00000   [ anon ]
00007ff60b375000      68 rw--- 0000000000000000 000:00000   [ anon ]
00007ff60b476000    1664 rw--- 0000000000000000 000:00000   [ anon ]
00007ff60b7e5000    1920 rw--- 0000000000000000 000:00000   [ anon ]
00007ff60b9d4000    1920 rw--- 0000000000000000 000:00000   [ anon ]
00007ff60bbc3000    9600 rw--- 0000000000000000 000:00000   [ anon ]
00007ff60c530000     640 rw--- 0000000000000000 000:00000   [ anon ]
00007ff60c5df000    1792 rw--- 0000000000000000 000:00000   [ anon ]
00007ff60c800000    2048 rw--- 0000000000000000 000:00000   [ anon ]
00007ff60ca00000  512000 r--s- 0000000000000000 103:00010 data.mdb
00007ff62be00000     320 r-x-- 0000000000000000 000:00000   [ anon ]
00007ff62be50000     960 r-x-- 0000000000000000 000:00000   [ anon ]
00007ff62bf4b000      64 r-x-- 0000000000000000 000:00000   [ anon ]
00007ff62bf5b000     640 rw--- 0000000000000000 000:00000   [ anon ]
00007ff62bffc000    1024 rw--- 0000000000000000 000:00000   [ anon ]
00007ff62c0fd000   10624 rw--- 0000000000000000 000:00000   [ anon ]
00007ff62cb5e000    6400 rw--- 0000000000000000 000:00000   [ anon ]
00007ff62d1a0000    4224 rw--- 0000000000000000 000:00000   [ anon ]
00007ff62d5c1000    1024 rw--- 0000000000000000 000:00000   [ anon ]
00007ff62d6c2000     768 rw--- 0000000000000000 000:00000   [ anon ]
00007ff62d78e000     128 r-x-- 0000000000000000 000:00000   [ anon ]
00007ff62d7ae000     896 rw--- 0000000000000000 000:00000   [ anon ]
00007ff62d891000     128 r-x-- 0000000000000000 000:00000   [ anon ]
00007ff62d8b1000    4224 rw--- 0000000000000000 000:00000   [ anon ]
00007ff62dcd2000    2432 rw--- 0000000000000000 000:00000   [ anon ]
00007ff62df34000      64 r-x-- 0000000000000000 000:00000   [ anon ]
00007ff62df44000   15744 rw--- 0000000000000000 000:00000   [ anon ]
00007ff62eeb2000      64 r-x-- 0000000000000000 000:00000   [ anon ]
00007ff62eec2000     128 rw--- 0000000000000000 000:00000   [ anon ]
00007ff62eee3000      64 r-x-- 0000000000000000 000:00000   [ anon ]
00007ff62eef3000    1792 rw--- 0000000000000000 000:00000   [ anon ]
00007ff62f0b4000      64 r-x-- 0000000000000000 000:00000   [ anon ]
00007ff62f0c4000     640 rw--- 0000000000000000 000:00000   [ anon ]
00007ff62f16b000      64 r-x-- 0000000000000000 000:00000   [ anon ]
00007ff62f18a000      64 r-x-- 0000000000000000 000:00000   [ anon ]
00007ff62f19a000    1536 rw--- 0000000000000000 000:00000   [ anon ]
00007ff62f31a000     256 r-x-- 0000000000000000 000:00000   [ anon ]
00007ff62f35a000    1408 rw--- 0000000000000000 000:00000   [ anon ]
00007ff62f4c9000     384 rw--- 0000000000000000 000:00000   [ anon ]
00007ff62f52a000      64 r-x-- 0000000000000000 000:00000   [ anon ]
00007ff62f53a000     768 rw--- 0000000000000000 000:00000   [ anon ]
00007ff62f5fa000     192 r-x-- 0000000000000000 000:00000   [ anon ]
00007ff62f62a000   11264 rw--- 0000000000000000 000:00000   [ anon ]
00007ff63012b000    1024 rw--- 0000000000000000 000:00000   [ anon ]
00007ff63022c000      64 r-x-- 0000000000000000 000:00000   [ anon ]
00007ff63023c000    3456 rw--- 0000000000000000 000:00000   [ anon ]
00007ff63059d000   18816 rw--- 0000000000000000 000:00000   [ anon ]
00007ff6317fd000      64 r-x-- 0000000000000000 000:00000   [ anon ]
00007ff63180d000    6400 rw--- 0000000000000000 000:00000   [ anon ]
00007ff631e4e000      64 r-x-- 0000000000000000 000:00000   [ anon ]
00007ff631e5e000   12160 rw--- 0000000000000000 000:00000   [ anon ]
00007ff632a3f000      64 r-x-- 0000000000000000 000:00000   [ anon ]
00007ff632a4f000    1024 rw--- 0000000000000000 000:00000   [ anon ]
00007ff632b51000      64 r-x-- 0000000000000000 000:00000   [ anon ]
00007ff632b70000     128 rw--- 0000000000000000 000:00000   [ anon ]
00007ff632b90000     128 r-x-- 0000000000000000 000:00000   [ anon ]
00007ff632bb0000     512 rw--- 0000000000000000 000:00000   [ anon ]
00007ff632c31000      64 r-x-- 0000000000000000 000:00000   [ anon ]
00007ff632c41000    2560 rw--- 0000000000000000 000:00000   [ anon ]
00007ff632ec7000     128 rw--- 0000000000000000 000:00000   [ anon ]
00007ff632ef3000     256 rw--- 0000000000000000 000:00000   [ anon ]
00007ff632f34000     384 r-x-- 0000000000000000 000:00000   [ anon ]
00007ff632f94000     128 rw--- 0000000000000000 000:00000   [ anon ]
00007ff632fb4000      64 r-x-- 0000000000000000 000:00000   [ anon ]
00007ff632fc4000     128 rw--- 0000000000000000 000:00000   [ anon ]
00007ff632fe7000    3584 rw--- 0000000000000000 000:00000   [ anon ]
00007ff633367000     320 r-x-- 0000000000000000 000:00000   [ anon ]
00007ff6333b7000   35228 rw--- 0000000000000000 000:00000   [ anon ]
00007ff63561e000    1024 rw-s- 0000000000000000 000:00001 zero (deleted)
00007ff63571e000   10240 rw-s- 0000000000000000 000:00001 zero (deleted)
00007ff63611e000   13952 rw--- 0000000000000000 000:00000   [ anon ]
00007ff636ebf000      64 r-x-- 0000000000000000 000:00000   [ anon ]
00007ff636ecf000   11068 rw--- 0000000000000000 000:00000   [ anon ]
00007ff6379ad000     256 r-x-- 0000000000000000 000:00000   [ anon ]
00007ff6379ed000    2492 rw--- 0000000000000000 000:00000   [ anon ]
00007ff637c5d000     256 rw--- 0000000000000000 000:00000   [ anon ]
00007ff637c9d000    1028 rw--- 0000000000000000 000:00000   [ anon ]
00007ff637d9e000      64 rwx-- 0000000000000000 000:00000   [ anon ]
00007ff637dae000     388 rw--- 0000000000000000 000:00000   [ anon ]
00007ff637e0f000       8 r---- 0000000000000000 103:00010 libyaml-0.so.2.0.6
00007ff637e11000     100 r-x-- 0000000000002000 103:00010 libyaml-0.so.2.0.6
00007ff637e2a000      16 r---- 000000000001b000 103:00010 libyaml-0.so.2.0.6
00007ff637e2e000       4 r---- 000000000001e000 103:00010 libyaml-0.so.2.0.6
00007ff637e2f000       4 rw--- 000000000001f000 103:00010 libyaml-0.so.2.0.6
00007ff637e30000    1856 rw--- 0000000000000000 000:00000   [ anon ]
00007ff638000000  262144 rw-s- 0000000000000000 000:00001 zero (deleted)
00007ff648000000  262144 rw-s- 0000000000000000 000:00001 zero (deleted)
00007ff658000000     132 rw--- 0000000000000000 000:00000   [ anon ]
00007ff658021000   65404 ----- 0000000000000000 000:00000   [ anon ]
00007ff65c00d000       8 r---- 0000000000000000 103:00010 yaml.so
00007ff65c00f000      16 r-x-- 0000000000002000 103:00010 yaml.so
00007ff65c013000       4 r---- 0000000000006000 103:00010 yaml.so
00007ff65c014000       4 r---- 0000000000006000 103:00010 yaml.so
00007ff65c015000       4 rw--- 0000000000007000 103:00010 yaml.so
00007ff65c016000    1536 rw--- 0000000000000000 000:00000   [ anon ]
00007ff65c1a5000     384 rw--- 0000000000000000 000:00000   [ anon ]
00007ff65c206000    2368 rw--- 0000000000000000 000:00000   [ anon ]
00007ff65c45e000      20 r---- 0000000000000000 103:00010 core.so
00007ff65c463000      32 r-x-- 0000000000005000 103:00010 core.so
00007ff65c46b000      12 r---- 000000000000d000 103:00010 core.so
00007ff65c46e000       4 r---- 000000000000f000 103:00010 core.so
00007ff65c46f000       4 rw--- 0000000000010000 103:00010 core.so
00007ff65c470000      12 r---- 0000000000000000 103:00010 pb.so
00007ff65c473000      56 r-x-- 0000000000003000 103:00010 pb.so
00007ff65c481000      16 r---- 0000000000011000 103:00010 pb.so
00007ff65c485000       4 r---- 0000000000014000 103:00010 pb.so
00007ff65c486000       4 rw--- 0000000000015000 103:00010 pb.so
00007ff65c48a000       4 r---- 0000000000000000 103:00010 lsyslog.so
00007ff65c48b000       4 r-x-- 0000000000001000 103:00010 lsyslog.so
00007ff65c48c000       4 r---- 0000000000002000 103:00010 lsyslog.so
00007ff65c48d000       4 r---- 0000000000002000 103:00010 lsyslog.so
00007ff65c48e000       4 rw--- 0000000000003000 103:00010 lsyslog.so
00007ff65c48f000      12 r---- 0000000000000000 103:00010 lpeg.so
00007ff65c492000      32 r-x-- 0000000000003000 103:00010 lpeg.so
00007ff65c49a000      12 r---- 000000000000b000 103:00010 lpeg.so
00007ff65c49d000       4 r---- 000000000000d000 103:00010 lpeg.so
00007ff65c49e000       4 rw--- 000000000000e000 103:00010 lpeg.so
00007ff65c49f000     128 rw--- 0000000000000000 000:00000   [ anon ]
00007ff65c4c3000      64 r-x-- 0000000000000000 000:00000   [ anon ]
00007ff65c4d3000       4 r---- 0000000000000000 103:00010 lua_pack.so
00007ff65c4d4000       8 r-x-- 0000000000001000 103:00010 lua_pack.so
00007ff65c4d6000       4 r---- 0000000000003000 103:00010 lua_pack.so
00007ff65c4d7000       4 r---- 0000000000003000 103:00010 lua_pack.so
00007ff65c4d8000       4 rw--- 0000000000004000 103:00010 lua_pack.so
00007ff65c4d9000       8 r---- 0000000000000000 103:00010 lfs.so
00007ff65c4db000       8 r-x-- 0000000000002000 103:00010 lfs.so
00007ff65c4dd000       4 r---- 0000000000004000 103:00010 lfs.so
00007ff65c4de000       4 r---- 0000000000004000 103:00010 lfs.so
00007ff65c4df000       4 rw--- 0000000000005000 103:00010 lfs.so
00007ff65c4e0000     128 rw--- 0000000000000000 000:00000   [ anon ]
00007ff65c500000    5120 rw-s- 0000000000000000 000:00001 zero (deleted)
00007ff65ca00000    5120 rw-s- 0000000000000000 000:00001 zero (deleted)
00007ff65cf00000   12288 rw-s- 0000000000000000 000:00001 zero (deleted)
00007ff65db00000   12288 rw-s- 0000000000000000 000:00001 zero (deleted)
00007ff65e700000   12288 rw-s- 0000000000000000 000:00001 zero (deleted)
00007ff65f300000    5120 rw-s- 0000000000000000 000:00001 zero (deleted)
00007ff65f800000    8192 rw-s- 0000000000000000 000:00001 zero (deleted)
00007ff660000000     132 rw--- 0000000000000000 000:00000   [ anon ]
00007ff660021000   65404 ----- 0000000000000000 000:00000   [ anon ]
00007ff664003000       4 r---- 0000000000000000 103:00010 lua_system_constants.so
00007ff664004000       4 r-x-- 0000000000001000 103:00010 lua_system_constants.so
00007ff664005000       4 r---- 0000000000002000 103:00010 lua_system_constants.so
00007ff664006000       4 r---- 0000000000002000 103:00010 lua_system_constants.so
00007ff664007000       4 rw--- 0000000000003000 103:00010 lua_system_constants.so
00007ff664008000       8 r---- 0000000000000000 103:00010 cjson.so
00007ff66400a000      20 r-x-- 0000000000002000 103:00010 cjson.so
00007ff66400f000       8 r---- 0000000000007000 103:00010 cjson.so
00007ff664011000       4 r---- 0000000000008000 103:00010 cjson.so
00007ff664012000       4 rw--- 0000000000009000 103:00010 cjson.so
00007ff664013000     640 rw--- 0000000000000000 000:00000   [ anon ]
00007ff6640b3000    5120 rw-s- 0000000000000000 000:00001 zero (deleted)
00007ff6645b3000    5120 rw-s- 0000000000000000 000:00001 zero (deleted)
00007ff664ab3000     384 rw--- 0000000000000000 000:00000   [ anon ]
00007ff664b13000       4 ----- 0000000000000000 000:00000   [ anon ]
00007ff664b14000    8192 rw--- 0000000000000000 000:00000   [ anon ]
00007ff665314000       4 ----- 0000000000000000 000:00000   [ anon ]
00007ff665315000    8208 rw--- 0000000000000000 000:00000   [ anon ]
00007ff665b19000       4 r---- 0000000000000000 103:00010 libpthread.so.0
00007ff665b1a000       4 r-x-- 0000000000001000 103:00010 libpthread.so.0
00007ff665b1b000       4 r---- 0000000000002000 103:00010 libpthread.so.0
00007ff665b1c000       4 r---- 0000000000002000 103:00010 libpthread.so.0
00007ff665b1d000       4 rw--- 0000000000003000 103:00010 libpthread.so.0
00007ff665b1e000      16 r---- 0000000000000000 103:00010 libelf.so.1
00007ff665b22000     120 r-x-- 0000000000004000 103:00010 libelf.so.1
00007ff665b40000      36 r---- 0000000000022000 103:00010 libelf.so.1
00007ff665b49000       4 r---- 000000000002a000 103:00010 libelf.so.1
00007ff665b4a000       4 rw--- 000000000002b000 103:00010 libelf.so.1
00007ff665b4b000      72 r---- 0000000000000000 103:00010 libdwarf.so
00007ff665b5d000     384 r-x-- 0000000000012000 103:00010 libdwarf.so
00007ff665bbd000     160 r---- 0000000000072000 103:00010 libdwarf.so
00007ff665be5000       4 ----- 000000000009a000 103:00010 libdwarf.so
00007ff665be6000       4 r---- 000000000009a000 103:00010 libdwarf.so
00007ff665be7000       8 rw--- 000000000009b000 103:00010 libdwarf.so
00007ff665be9000       4 rw--- 0000000000000000 000:00000   [ anon ]
00007ff665bea000       4 r---- 0000000000000000 103:00010 libdl.so.2
00007ff665beb000       4 r-x-- 0000000000001000 103:00010 libdl.so.2
00007ff665bec000       4 r---- 0000000000002000 103:00010 libdl.so.2
00007ff665bed000       4 r---- 0000000000002000 103:00010 libdl.so.2
00007ff665bee000       4 rw--- 0000000000003000 103:00010 libdl.so.2
00007ff665bef000       4 r---- 0000000000000000 103:00010 librt.so.1
00007ff665bf0000       4 r-x-- 0000000000001000 103:00010 librt.so.1
00007ff665bf1000       4 r---- 0000000000002000 103:00010 librt.so.1
00007ff665bf2000       4 r---- 0000000000002000 103:00010 librt.so.1
00007ff665bf3000       4 rw--- 0000000000003000 103:00010 librt.so.1
00007ff665bf4000    1228 r---- 0000000000000000 103:00010 liboneagentnginx.so
00007ff665d27000   15012 r-x-- 0000000000133000 103:00010 liboneagentnginx.so
00007ff666bd0000    6628 r---- 0000000000fdc000 103:00010 liboneagentnginx.so
00007ff667249000     784 r---- 0000000001654000 103:00010 liboneagentnginx.so
00007ff66730d000      36 rw--- 0000000001718000 103:00010 liboneagentnginx.so
00007ff667316000     676 rw--- 0000000000000000 000:00000   [ anon ]
00007ff6673bf000       4 r---- 0000000000000000 000:00000   [ anon ]
00007ff6673c0000    1048 rw--- 0000000000000000 000:00000   [ anon ]
00007ff6674c6000     616 r---- 0000000000000000 103:00010 libstdc++.so.6.0.30
00007ff667560000    1092 r-x-- 000000000009a000 103:00010 libstdc++.so.6.0.30
00007ff667671000     444 r---- 00000000001ab000 103:00010 libstdc++.so.6.0.30
00007ff6676e0000       4 ----- 000000000021a000 103:00010 libstdc++.so.6.0.30
00007ff6676e1000      44 r---- 000000000021a000 103:00010 libstdc++.so.6.0.30
00007ff6676ec000      12 rw--- 0000000000225000 103:00010 libstdc++.so.6.0.30
00007ff6676ef000      12 rw--- 0000000000000000 000:00000   [ anon ]
00007ff6676f2000      12 r---- 0000000000000000 103:00010 libgcc_s.so.1
00007ff6676f5000      92 r-x-- 0000000000003000 103:00010 libgcc_s.so.1
00007ff66770c000      16 r---- 000000000001a000 103:00010 libgcc_s.so.1
00007ff667710000       4 r---- 000000000001d000 103:00010 libgcc_s.so.1
00007ff667711000       4 rw--- 000000000001e000 103:00010 libgcc_s.so.1
00007ff667712000     160 r---- 0000000000000000 103:00010 libc.so.6
00007ff66773a000    1620 r-x-- 0000000000028000 103:00010 libc.so.6
00007ff6678cf000     352 r---- 00000000001bd000 103:00010 libc.so.6
00007ff667927000       4 ----- 0000000000215000 103:00010 libc.so.6
00007ff667928000      16 r---- 0000000000215000 103:00010 libc.so.6
00007ff66792c000       8 rw--- 0000000000219000 103:00010 libc.so.6
00007ff66792e000      52 rw--- 0000000000000000 000:00000   [ anon ]
00007ff66793b000       8 r---- 0000000000000000 103:00010 libz.so.1.2.11
00007ff66793d000      68 r-x-- 0000000000002000 103:00010 libz.so.1.2.11
00007ff66794e000      24 r---- 0000000000013000 103:00010 libz.so.1.2.11
00007ff667954000       4 ----- 0000000000019000 103:00010 libz.so.1.2.11
00007ff667955000       4 r---- 0000000000019000 103:00010 libz.so.1.2.11
00007ff667956000       4 rw--- 000000000001a000 103:00010 libz.so.1.2.11
00007ff667957000    5028 r-x-- 0000000000000000 103:00010 libcrypto.so.3
00007ff667e40000       4 ----- 00000000004e9000 103:00010 libcrypto.so.3
00007ff667e41000     380 r---- 00000000004e9000 103:00010 libcrypto.so.3
00007ff667ea0000      12 rw--- 0000000000548000 103:00010 libcrypto.so.3
00007ff667ea3000      24 rw--- 0000000000000000 000:00000   [ anon ]
00007ff667ea9000     892 r-x-- 0000000000000000 103:00010 libssl.so.3
00007ff667f88000      44 r---- 00000000000de000 103:00010 libssl.so.3
00007ff667f93000      16 rw--- 00000000000e9000 103:00010 libssl.so.3
00007ff667f97000      56 r---- 0000000000000000 103:00010 libm.so.6
00007ff667fa5000     496 r-x-- 000000000000e000 103:00010 libm.so.6
00007ff668021000     364 r---- 000000000008a000 103:00010 libm.so.6
00007ff66807c000       4 r---- 00000000000e4000 103:00010 libm.so.6
00007ff66807d000       4 rw--- 00000000000e5000 103:00010 libm.so.6
00007ff66807e000      32 r---- 0000000000000000 103:00010 libluajit-5.1.so.2.1.ROLLING
00007ff668086000     436 r-x-- 0000000000008000 103:00010 libluajit-5.1.so.2.1.ROLLING
00007ff6680f3000      84 r---- 0000000000075000 103:00010 libluajit-5.1.so.2.1.ROLLING
00007ff668108000       4 ----- 000000000008a000 103:00010 libluajit-5.1.so.2.1.ROLLING
00007ff668109000       8 r---- 000000000008a000 103:00010 libluajit-5.1.so.2.1.ROLLING
00007ff66810b000       4 rw--- 000000000008c000 103:00010 libluajit-5.1.so.2.1.ROLLING
00007ff66810c000       8 r---- 0000000000000000 103:00010 libcrypt.so.1.1.0
00007ff66810e000      80 r-x-- 0000000000002000 103:00010 libcrypt.so.1.1.0
00007ff668122000     100 r---- 0000000000016000 103:00010 libcrypt.so.1.1.0
00007ff66813b000       4 ----- 000000000002f000 103:00010 libcrypt.so.1.1.0
00007ff66813c000       4 r---- 000000000002f000 103:00010 libcrypt.so.1.1.0
00007ff66813d000       4 rw--- 0000000000030000 103:00010 libcrypt.so.1.1.0
00007ff66813e000      32 rw--- 0000000000000000 000:00000   [ anon ]
00007ff668146000       8 rw-s- 0000000000000000 103:00010 lock.mdb
00007ff668148000      44 r---- 0000000000000000 103:00010 liboneagentproc.so
00007ff668153000     660 r-x-- 000000000000b000 103:00010 liboneagentproc.so
00007ff6681f8000     260 r---- 00000000000b0000 103:00010 liboneagentproc.so
00007ff668239000      24 r---- 00000000000f0000 103:00010 liboneagentproc.so
00007ff66823f000       4 rw--- 00000000000f6000 103:00010 liboneagentproc.so
00007ff668240000    1256 rw--- 0000000000000000 000:00000   [ anon ]
00007ff66837a000      16 r---- 0000000000000000 000:00000   [ anon ]
00007ff66837e000       8 r-x-- 0000000000000000 000:00000   [ anon ]
00007ff668380000       8 r---- 0000000000000000 103:00010 ld-linux-x86-64.so.2
00007ff668382000     168 r-x-- 0000000000002000 103:00010 ld-linux-x86-64.so.2
00007ff6683ac000      44 r---- 000000000002c000 103:00010 ld-linux-x86-64.so.2
00007ff6683b7000       4 rw-s- 0000000000000000 000:00001 zero (deleted)
00007ff6683b8000       8 r---- 0000000000037000 103:00010 ld-linux-x86-64.so.2
00007ff6683ba000       8 rw--- 0000000000039000 103:00010 ld-linux-x86-64.so.2
00007ff668c10000     192 r-x-- 0000000000000000 000:00000   [ anon ]
00007fff211d8000     140 rw--- 0000000000000000 000:00000   [ stack ]
ffffffffff600000       4 --x-- 0000000000000000 000:00000   [ anon ]
mapped: 3521720K    writeable/private: 2226280K    shared: 1118220K

current memory in dashboard in 4..89GB.

muh-kamran commented 2 months ago

I was trying out versions released after 3.3. The memory issue starts from 3.5 onwards.

muh-kamran commented 2 months ago

Hi @nowNick ,I was wondering if you’ve identified the issue or if it’s still unknown.

nowNick commented 2 months ago

Hi @muh-kamran, this is still under investigation.

brad-getpassport commented 2 months ago

We're seeing the same issue albeit smaller leaks of about ~2-22MB for every one of our calls through the KONG gateway. We're on 3.6.1. We are using the grpc plugin and a custom grpc-request-transfomer. If we let the system run for a long time we get the OOMKilled, but only allocating 2Gi for the memory and 2 cpus.

@nowNick if you need some extra logs just let us know as we're seeing the request-transformer is constantly being called before it dies like this:

2024/09/12 14:01:42 [notice] 1438#0: *42074 [kong] handler.lua:71 [grpc_request_transformer] body_filter while reading upstream
2024/09/12 14:01:42 [notice] 1438#0: *42074 [kong] handler.lua:71 [grpc_request_transformer] body_filter while reading upstream
2024/09/12 14:01:42 [notice] 1438#0: *42074 [kong] handler.lua:71 [grpc_request_transformer] body_filter while reading upstream
2024/09/12 14:01:42 [notice] 1438#0: *42074 [kong] handler.lua:71 [grpc_request_transformer] body_filter while reading upstream
2024/09/12 14:01:42 [notice] 1438#0: *42074 [kong] handler.lua:71 [grpc_request_transformer] body_filter while reading upstream
2024/09/12 14:01:42 [notice] 1438#0: *42074 [kong] handler.lua:71 [grpc_request_transformer] body_filter while reading upstream
2024/09/12 14:01:42 [notice] 1438#0: *42074 [kong] handler.lua:71 [grpc_request_transformer] body_filter while reading upstream
2024/09/12 14:01:42 [notice] 1438#0: *42074 [kong] handler.lua:71 [grpc_request_transformer] body_filter while reading upstream
2024/09/12 14:01:42 [notice] 1438#0: *42074 [kong] handler.lua:71 [grpc_request_transformer] body_filter while reading upstream
2024/09/12 14:01:42 [notice] 1438#0: *42074 [kong] handler.lua:71 [grpc_request_transformer] body_filter while reading upstream
2024/09/12 14:01:42 [notice] 1438#0: *42074 [kong] handler.lua:71 [grpc_request_transformer] body_filter while reading upstream
2024/09/12 14:01:42 [notice] 1438#0: *42074 [kong] handler.lua:71 [grpc_request_transformer] body_filter while reading upstream
2024/09/12 14:01:42 [notice] 1438#0: *42074 [kong] handler.lua:71 [grpc_request_transformer] body_filter while reading upstream
2024/09/12 14:01:42 [notice] 1438#0: *42074 [kong] handler.lua:71 [grpc_request_transformer] body_filter while reading upstream

kikito commented 2 months ago

Internal tracking code: KAG-5409

brad-getpassport commented 2 months ago

Just to add more information, I setup my system last night on my desktop machine using minikube with 8 CPUs and 16GB of memory and just installed kong, our instrumentation system and one additional pod. I took a memory snapshot before I went to bed, and just now. System was idle all night. Kong version 3.6.1

% date; kubectl top pods -n develop | grep kong
Thu Sep 12 21:46:53 PDT 2024
kong-58f5b88c59-ml2nj                                57m          631Mi           
% date; kubectl top pods -n develop | grep kong
Fri Sep 13 05:31:33 PDT 2024
kong-58f5b88c59-ml2nj                                46m          1044Mi

chobits commented 2 months ago

From your provided information int this comment: https://github.com/Kong/kong/issues/13553?notification_referrer_id=NT_kwDOAAmB1rIxMjA2MjA2MDk0Mzo2MjMwNjI#issuecomment-2320593736, I guess there might be some memory leaks inside nginx core. The memory usage from lua VM appears to be quite small.

pmap -d 1347 00005558adb0a000 1995440 rw--- 0000000000000000 000:00000 [ anon ] The anon memory is high for this worker process its 1995440KB which is 1995.44MB since two worker becomes 3990MB = 4GB and total memory in graph is around 4.89.

Could you dump this segment of memory as some strings , for example, you could use gdb like this:

gdb -p 1347
(gdb) x/1000s 0x00005558adb0a000

^--- 1000 is the potential string number you want to decode, you could modify it to observe the results.

If you're fortunate enough that a specific string is leaking, the information provided by the string can help us further deduce which module is causing the leak.

muh-kamran commented 2 months ago

I have updated kong to 3.8.0 and memory looks good so far. I will monitor it for a while.

FYI: the memory issue is from 3.5.0 to 3.7.1.

Kong / kong