help request: send Put method admin request frequently costs apisix cpu high(about 100%) when more than 15 thousand routes exist in apisix

[ranxuxin001]

Description

Hello everyone, My apisix 2.9 has more than 15 thousand routes currently. And when I use the script to update large amount of routes frequently(sleep 0.5 to 5 seconds), my apisix costs cpu near 100%(user). It leads to request apisix slowly.

I've read some source code of apisix. When update a route, the apisix firstly search relevant node in radix tree. And secondly insert node in radix tree. But most of my routes has the same uri "/*" . The radix tree should has a few nodes, but it has an array with many routes which has the same uri. After that it updates the configuration of the matched route according to host, uri, header etc. So I doubt finding matched route in a long array causes much more cpu time than updating the node in the radix tree. By the way, it there any cpu performance tool suggusted? How about openresty xray?

Is there any strategy to improve the apisx's performance for updating or creating routes frequently while large amount of routes already exist in the apisix, please? Any suggest is appreciated.

Best Regards, Ranxuxin

Environment

• APISIX version (run apisix version): apisix-2.9-0.el7.x86_64
• Operating system (run uname -a): Linux 3.10.0-862.el7.x86_64 #1 SMP Fri Apr 20 16:44:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• OpenResty / Nginx version (run openresty -V or nginx -V): nginx version: openresty/1.19.9.1 built by gcc 9.3.1 20200408 (Red Hat 9.3.1-2) (GCC) built with OpenSSL 1.1.1k 25 Mar 2021 TLS SNI support enabled configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/zlib/include -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl111/include' --add-module=../ngx_devel_kit-0.3.1 --add-module=../echo-nginx-module-0.62 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.32 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.08 --add-module=../srcache-nginx-module-0.32 --add-module=../ngx_lua-0.10.20 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.7 --add-module=../ngx_stream_lua-0.0.10 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -L/usr/local/openresty/zlib/lib -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl111/lib -Wl,-rpath,/usr/local/openresty/zlib/lib:/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl111/lib' --with-cc='ccache gcc -fdiagnostics-color=always' --with-pcre-jit --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_v2_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_auth_request_module --with-http_secure_link_module --with-http_random_index_module --with-http_gzip_static_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --with-compat --with-stream --with-http_ssl_module
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): 3.5
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[ranxuxin001]

@membphis @soulbird hi membphis, soulbird, Could you help me to suggest this issue to any adminstrator who is responsible for solving performance problem of apisix? #9157

[soulbird]

Creating a new route does trigger a rebuild of the route radix tree, which involves a lot of table creation and gc, which usually consume CPU, which is currently unavoidable. Can you consider reducing the frequency of routing updates to reduce CPU consumption? In addition, you can observe CPU performance problems through flame graphs. For APISIX, you can refer to the script here: https://github.com/apache/apisix/blob/master/ci/performance_test.

[ranxuxin001]

Thank to your suggestion.