help request: memory leakage problem in Apisix GW

[sdd-el]

Description

Current behavior: After running stress tests, memory usage increases and doesn’t decrease after the tests are finished (unlike CPU usage, which decreases as expected). The configured resources are: resources: limits: cpu: 2000m memory: 5Gi requests: cpu: 1500m memory: 4Gi The minimum replicas are set to 1, and the maximum replicas are set to 3. The plugins in use are proxy-rewrite, serverless-pre-function, loki-logger, and prometheus. Any idea what could be causing this? Why isn’t memory being “cleaned” after the traffic load is gone?

Expected behavior: Memory usage should decrease after the traffic load is gone, similar to how CPU usage decreases.

Environment

• APISIX version: 3.9.1-debian
• Operating system (run uname -a):
• OpenResty / Nginx version (run openresty -V or nginx -V): nginx version: openresty/1.25.3.1

[wklken]

did you set the prometheus.expire in config.yaml? https://github.com/apache/apisix/blob/release/3.9/conf/config-default.yaml#L605

and also you can change the default_buckets

try and do the test again.

[sdd-el]

@wklken thx for your comment. I tried what you have proposed, i.e.,
prometheus:

export_addr:

  ip: 0.0.0.0

  port: 9091

export_uri: /apisix/prometheus/metrics

metric_prefix: apisix_

expire: 600

default_buckets:

  - 10

  - 50

but I still get memory increase which is constant, i.e., memory is not released after running load test(s). Do you have any other possible solution in mind?

[wklken]

curl http://0.0.0.0:9091/apisix/prometheus/metrics > a

get the metrics

• file size
• line count
• the metrics labels which is not enum(like customer)

---

you can see some more tips here: https://github.com/apache/apisix/issues/10618

and https://github.com/apache/apisix/issues/10618#issuecomment-1870936375

[sdd-el]

@wklken thanks for your feedback.

Here are my findings:

• line count: 249 lines
• snippet of the file (some inputs are left out due to confidentiality):

  HELP apisix_bandwidth Total bandwidth in bytes consumed per service in APISIX

  TYPE apisix_bandwidth counter

  apisix_bandwidth{type="egress",route="16607a",service="",consumer="",node="10.244.121.127"} 3060 apisix_bandwidth{type="egress",route="16607a",service="",consumer="",node="10.244.121.127"} 3060 apisix_bandwidth{type="egress",route="16607a",service="",consumer="",node="10.244.121.127"} 3060 apisix_bandwidth{type="egress",route="16607a",service="",consumer="",node="10.244.121.127"} 3060 apisix_bandwidth{type="egress",route="drg4025-dev_digint-documentation-dev_public-routes-rule",service="",consumer="",node="10.244.121.127"} 3060 apisix_bandwidth{type="ingress",route="16607a",service="",consumer="",node="10.244.121.127"} 9600 apisix_bandwidth{type="ingress",route="16607a",service="",consumer="",node="10.244.121.127"} 9600 apisix_bandwidth{type="ingress",route="16607a",service="",consumer="",node="10.244.121.127"} 9600 apisix_bandwidth{type="ingress",route="16607a",service="",consumer="",node="10.244.121.127"} 9600 apisix_bandwidth{type="ingress",route="public-routes-rule",service="",consumer="",node="10.244.121.127"} 9600 apisix_bandwidth{type="ingress",route="public-routes-rule",service="",consumer="",node="10.244.121.127"} 9600 apisix_bandwidth{type="ingress",route="public-routes-rule",service="",consumer="",node="10.244.121.127"} 9600 apisix_bandwidth{type="ingress",route="public-routes-rule",service="",consumer="",node="10.244.121.127"} 9600 apisix_batch_process_entries{name="loki logger",route_id="13628b87",server_addr="10.244.52.40"} 0 apisix_batch_process_entries{name="loki logger",route_id="16607a",server_addr="10.244.52.40"} 0 apisix_batch_process_entries{name="loki logger",route_id="1b024cba",server_addr="10.244.52.40"} 0 apisix_batch_process_entries{name="loki logger",route_id="1c189a16",server_addr="10.244.52.40"} 0 apisix_batch_process_entries{name="loki logger",route_id="2ab1ba1b",server_addr="10.244.52.40"} 0 apisix_batch_process_entries{name="loki logger",route_id="2f222445",server_addr="10.244.52.40"} 0 apisix_batch_process_entries{name="loki logger",route_id="34af2003",server_addr="10.244.52.40"} 0 apisix_batch_process_entries{name="loki logger",route_id="5787afbe",server_addr="10.244.52.40"} 0 apisix_batch_process_entries{name="loki logger",route_id="65d6160b",server_addr="10.244.52.40"} 0 apisix_batch_process_entries{name="loki logger",route_id="6e312e6f",server_addr="10.244.52.40"} 0 apisix_batch_process_entries{name="loki logger",route_id="6e8ca755",server_addr="10.244.52.40"} 0 apisix_batch_process_entries{name="loki logger",route_id="747b5",server_addr="10.244.52.40"} 0 apisix_batch_process_entries{name="loki logger",route_id="761056d5",server_addr="10.244.52.40"} 0 apisix_batch_process_entries{name="loki logger",route_id="8b25f851",server_addr="10.244.52.40"} 0 apisix_batch_process_entries{name="loki logger",route_id="b4d36336",server_addr="10.244.52.40"} 0 apisix_batch_process_entries{name="loki logger",route_id="be7c0e57",server_addr="10.244.52.40"} 0 apisix_batch_process_entries{name="loki logger",route_id="d07bcaff",server_addr="10.244.52.40"} 0 apisix_batch_process_entries{name="loki logger",route_id="e0ef6cc0",server_addr="10.244.52.40"} 0 apisix_batch_process_entries{name="loki logger",route_id="e467cf29",server_addr="10.244.52.40"} 0 apisix_batch_process_entries{name="loki logger",route_id="f4f8913c",server_addr="10.244.52.40"} 0 apisix_batch_process_entries{name="loki logger",route_id="fa62605f",server_addr="10.244.52.40"} 0

  HELP apisix_etcd_modify_indexes Etcd modify index for APISIX keys

  TYPE apisix_etcd_modify_indexes gauge

  apisix_etcd_modify_indexes{key="consumers"} 105099 apisix_etcd_modify_indexes{key="global_rules"} 105086 apisix_etcd_modify_indexes{key="max_modify_index"} 105207 apisix_etcd_modify_indexes{key="prev_index"} 105086 apisix_etcd_modify_indexes{key="protos"} 0 apisix_etcd_modify_indexes{key="routes"} 105207 apisix_etcd_modify_indexes{key="services"} 0 apisix_etcd_modify_indexes{key="ssls"} 105089 apisix_etcd_modify_indexes{key="stream_routes"} 0 apisix_etcd_modify_indexes{key="upstreams"} 105170 apisix_etcd_modify_indexes{key="x_etcd_index"} 105208

  HELP apisix_etcd_reachable Config server etcd reachable from APISIX, 0 is unreachable

  TYPE apisix_etcd_reachable gauge

  apisix_etcd_reachable 1

  HELP apisix_http_latency HTTP request latency in milliseconds per service in APISIX

  TYPE apisix_http_latency histogram

  apisix_http_latency_bucket{type="apisix",route="16607a",service="",consumer="",node="10.244.121.127",le="10"} 12 apisix_http_latency_bucket{type="apisix",route="16607a",service="",consumer="",node="10.244.121.127",le="10"} 12 apisix_http_latency_bucket{type="apisix",route="16607a",service="",consumer="",node="10.244.121.127",le="10"} 12 apisix_http_latency_bucket{type="apisix",route="16607a",service="",consumer="",node="10.244.121.127",le="10"} 12 apisix_http_latency_bucket{type="apisix",route="16607a",service="",consumer="",node="10.244.121.127",le="50"} 12 apisix_http_latency_bucket{type="apisix",route="16607a",service="",consumer="",node="10.244.121.127",le="50"} 12 apisix_http_latency_bucket{type="apisix",route="16607a",service="",consumer="",node="10.244.121.127",le="50"} 12 apisix_http_latency_bucket{type="apisix",route="16607a",service="",consumer="",node="10.244.121.127",le="50"} 12 apisix_http_latency_bucket{type="apisix",route="16607a",service="",consumer="",node="10.244.121.127",le="+Inf"} 12 apisix_http_latency_bucket{type="apisix",route="16607a",service="",consumer="",node="10.244.121.127",le="+Inf"} 12 apisix_http_latency_bucket{type="apisix",route="16607a",service="",consumer="",node="10.244.121.127",le="+Inf"} 12 apisix_http_latency_bucket{type="apisix",route="16607a",service="",consumer="",node="10.244.121.127",le="+Inf"} 12 ...

  HELP apisix_nginx_http_current_connections Number of HTTP connections

  TYPE apisix_nginx_http_current_connections gauge

  apisix_nginx_http_current_connections{state="accepted"} 29504 apisix_nginx_http_current_connections{state="active"} 76 apisix_nginx_http_current_connections{state="handled"} 29504 apisix_nginx_http_current_connections{state="reading"} 0 apisix_nginx_http_current_connections{state="waiting"} 2 apisix_nginx_http_current_connections{state="writing"} 74

  HELP apisix_nginx_metric_errors_total Number of nginx-lua-prometheus errors

  TYPE apisix_nginx_metric_errors_total counter

  apisix_nginx_metric_errors_total 0

  HELP apisix_node_info Info of APISIX node

  TYPE apisix_node_info gauge

  apisix_node_info{hostname="apisix-6b54f445f5-hbmns"} 1

  HELP apisix_shared_dict_capacity_bytes The capacity of each nginx shared DICT since APISIX start

  TYPE apisix_shared_dict_capacity_bytes gauge

  apisix_shared_dict_capacity_bytes{name="access-tokens"} 1048576 apisix_shared_dict_capacity_bytes{name="balancer-ewma"} 10485760 apisix_shared_dict_capacity_bytes{name="balancer-ewma-last-touched-at"} 10485760 apisix_shared_dict_capacity_bytes{name="balancer-ewma-locks"} 10485760 apisix_shared_dict_capacity_bytes{name="discovery"} 1048576 apisix_shared_dict_capacity_bytes{name="etcd-cluster-health-check"} 10485760 apisix_shared_dict_capacity_bytes{name="ext-plugin"} 1048576 apisix_shared_dict_capacity_bytes{name="internal-status"} 10485760 apisix_shared_dict_capacity_bytes{name="introspection"} 10485760 apisix_shared_dict_capacity_bytes{name="jwks"} 1048576 apisix_shared_dict_capacity_bytes{name="lrucache-lock"} 10485760 apisix_shared_dict_capacity_bytes{name="plugin-api-breaker"} 10485760 apisix_shared_dict_capacity_bytes{name="plugin-limit-conn"} 10485760 apisix_shared_dict_capacity_bytes{name="plugin-limit-conn-redis-cluster-slot-lock"} 1048576 apisix_shared_dict_capacity_bytes{name="plugin-limit-count"} 10485760 apisix_shared_dict_capacity_bytes{name="plugin-limit-count-redis-cluster-slot-lock"} 1048576 apisix_shared_dict_capacity_bytes{name="plugin-limit-count-reset-header"} 10485760 apisix_shared_dict_capacity_bytes{name="plugin-limit-req"} 10485760 apisix_shared_dict_capacity_bytes{name="plugin-limit-req-redis-cluster-slot-lock"} 1048576 apisix_shared_dict_capacity_bytes{name="prometheus-metrics"} 10485760 apisix_shared_dict_capacity_bytes{name="upstream-healthcheck"} 10485760 apisix_shared_dict_capacity_bytes{name="worker-events"} 10485760

  HELP apisix_shared_dict_free_space_bytes The free space of each nginx shared DICT since APISIX start

  TYPE apisix_shared_dict_free_space_bytes gauge

  apisix_shared_dict_free_space_bytes{name="access-tokens"} 1032192 apisix_shared_dict_free_space_bytes{name="balancer-ewma"} 10412032 apisix_shared_dict_free_space_bytes{name="balancer-ewma-last-touched-at"} 10412032 apisix_shared_dict_free_space_bytes{name="balancer-ewma-locks"} 10412032 apisix_shared_dict_free_space_bytes{name="discovery"} 1032192 apisix_shared_dict_free_space_bytes{name="etcd-cluster-health-check"} 10412032 apisix_shared_dict_free_space_bytes{name="ext-plugin"} 1032192 apisix_shared_dict_free_space_bytes{name="internal-status"} 10412032 apisix_shared_dict_free_space_bytes{name="introspection"} 10412032 apisix_shared_dict_free_space_bytes{name="jwks"} 1032192 apisix_shared_dict_free_space_bytes{name="lrucache-lock"} 10412032 apisix_shared_dict_free_space_bytes{name="plugin-api-breaker"} 10412032 apisix_shared_dict_free_space_bytes{name="plugin-limit-conn"} 10412032 apisix_shared_dict_free_space_bytes{name="plugin-limit-conn-redis-cluster-slot-lock"} 1036288 apisix_shared_dict_free_space_bytes{name="plugin-limit-count"} 10412032 apisix_shared_dict_free_space_bytes{name="plugin-limit-count-redis-cluster-slot-lock"} 1036288 apisix_shared_dict_free_space_bytes{name="plugin-limit-count-reset-header"} 10412032 apisix_shared_dict_free_space_bytes{name="plugin-limit-req"} 10412032 apisix_shared_dict_free_space_bytes{name="plugin-limit-req-redis-cluster-slot-lock"} 1036288 apisix_shared_dict_free_space_bytes{name="prometheus-metrics"} 9166848 apisix_shared_dict_free_space_bytes{name="upstream-healthcheck"} 10412032 apisix_shared_dict_free_space_bytes{name="worker-events"} 10412032