bug: Memory leak when logger log the traffic with large (about 128K) response body

caogw279 commented 1 month ago

Current Behavior

When configured kafka-logger in global rules with include_resp_body set to true and upstream server response 128K body, the memory will increase during the test and never get down, if the test continue, the memory usage will continue increase until the OOM happen.

Expected Behavior

The memory usage will keep in a range during the test and will return to normal after the stress stop.

Error Logs

No error logs, please have look at output of "free" and the RES of very worker process

Steps to Reproduce

Run apisix in CentOS
APISIX Config route with Admin API
APISIX Config kafka-logger to globale_rules with include_resp_body set to true
Stress the APISIX (with Avalanche, server response 128K )
Monitor the system memory usage (such as via order free -h)
Check the memory usage of every worker process (such as via top)

Environment

APISIX version (run apisix version): apisix 3.8.0
Operating system (run uname -a): Linux localhost.localdomain 3.10.0-1160.105.1.el7.x86_64 #1 SMP Thu Dec 7 15:39:45 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
OpenResty / Nginx version (run openresty -V or nginx -V): openresty/1.21.4.2
etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): 3.5.0
APISIX Dashboard version, if relevant: No APISIX Dashboard used.
Plugin runner version, for issues related to plugin runners: no external plugin is used
LuaRocks version, for installation issues (run luarocks --version): No luarocks is installed.

caogw279 commented 1 month ago

Addition: I test it with wrk as client and take the nginx as the upstream server, and change it's index.html with 512k bytes characters, can't reproduce the issue, resident memory of every openresty process will keep at 1 - 1.5 G no matter how long wrk stress it.

caogw279 commented 1 month ago

From the document [gc public API] (https://apisix.apache.org/docs/apisix/control-api/) I found a method which will trigger a full garbage collection in the HTTP subsystem, it's works, the memory not leak ? just hold by GC ? but why doesn't GC collection work by itself, why the memory is exhausted and GC still not work ?

apache / apisix