upstream_response_time and upstream_header_time takes too long

[highwayliu]

Summary

• Kong version ($ kong version)
• 2.4.1
• A concise summary of the bug.
• I use kong for grpc. for some cases, upstream_response_time and upstream_header_time over 15s, but the upstream_connect_time is 0.000s. and the upstream server cost only 0.05s. the nginx log format is '$remote_addr - $remote_user [$time_local]"$request" $status "$upstream_status" $body_bytes_sent "$http_referer" "$http_user_agent" "$request_time" "$upstream_response_time" "$upstream_connect_time" "$upstream_header_time"'; the nginx log is 8.17.206.83 - - [26/Jun/2021:03:38:31 +0000] "POST /feed.FeedService/ExposureFeed HTTP/2.0" 200 "200" 7 "-" "dart-grpc/2.0.0" "15.057" "15.056" "0.000" "15.056"

Steps To Reproduce

1. 2. 3. 4.

Additional Details & Logs

• Kong debug-level startup logs ($ kong start --vv)
• 2021/06/26 05:09:20 [verbose] Kong: 2.4.1 2021/06/26 05:09:20 [debug] ngx_lua: 10019 2021/06/26 05:09:20 [debug] nginx: 1019003 2021/06/26 05:09:20 [debug] Lua: LuaJIT 2.1.0-beta3 2021/06/26 05:09:20 [verbose] no config file found at /etc/kong/kong.conf 2021/06/26 05:09:20 [verbose] no config file found at /etc/kong.conf 2021/06/26 05:09:20 [verbose] no config file, skip loading 2021/06/26 05:09:20 [debug] reading environment variables 2021/06/26 05:09:20 [debug] KONG_NGINX_PROXY_CLIENT_BODY_BUFFER_SIZE ENV found with "8m" 2021/06/26 05:09:20 [debug] KONG_NGINX_PROXY_CLIENT_MAX_BODY_SIZE ENV found with "8m" 2021/06/26 05:09:20 [debug] KONG_NGINX_PROXY_KEEPALIVE_TIMEOUT ENV found with "60s" 2021/06/26 05:09:20 [debug] KONG_NGINX_PROXY_PROXY_BUFFERS ENV found with "4 256k" 2021/06/26 05:09:20 [debug] KONG_NGINX_PROXY_PROXY_BUFFER_SIZE ENV found with "128k" 2021/06/26 05:09:20 [debug] KONG_NGINX_PROXY_PROXY_BUSY_BUFFERS_SIZE ENV found with "256k" 2021/06/26 05:09:20 [debug] KONG_LUA_PACKAGE_PATH ENV found with "./?.lua;./?/init.lua;/usr/local/custom/?.lua;;" 2021/06/26 05:09:20 [debug] KONG_SSL_CERT ENV found with "/usr/local/custom/kong/ssl/server.pem" 2021/06/26 05:09:20 [debug] KONG_PROXY_LISTEN ENV found with "0.0.0.0:8000, 0.0.0.0:8443 ssl, 0.0.0.0:9080 http2, 0.0.0.0:9081 http2 ssl" 2021/06/26 05:09:20 [debug] KONG_ADMIN_LISTEN ENV found with "0.0.0.0:8001" 2021/06/26 05:09:20 [debug] KONG_NGINX_HTTP_CLIENT_MAX_BODY_SIZE ENV found with "40m" 2021/06/26 05:09:20 [debug] KONG_DATABASE ENV found with "postgres" 2021/06/26 05:09:20 [debug] KONG_PG_PORT ENV found with "5432" 2021/06/26 05:09:20 [debug] KONG_PG_PASSWORD ENV found with "**" 2021/06/26 05:09:20 [debug] KONG_NGINX_HTTP_LOG_FORMAT ENV found with "main '$remote_addr - $remote_user [$time_local] "$request" $status "$upstream_status" $body_bytes_sent "$http_referer" "$http_user_agent" "$request_time" "$upstream_response_time" "$upstream_connect_time" "$upstream_header_time"'" 2021/06/26 05:09:20 [debug] KONG_SSL_CERT_KEY ENV found with "/usr/local/custom/kong/ssl/server.key" 2021/06/26 05:09:20 [debug] KONG_PG_HOST ENV found with "172.26.0.45" 2021/06/26 05:09:20 [debug] KONG_PG_DATABASE ENV found with "kong_pre_outer" 2021/06/26 05:09:20 [debug] KONG_PG_USER ENV found with "kong" 2021/06/26 05:09:20 [debug] KONG_PROXY_ERROR_LOG ENV found with "/usr/local/kong/logs/proxy-error.log" 2021/06/26 05:09:20 [debug] KONG_PROXY_ACCESS_LOG ENV found with "/usr/local/kong/logs/proxy-access.log main" 2021/06/26 05:09:20 [debug] KONG_ADMIN_ACCESS_LOG ENV found with "/usr/local/kong/logs/admin-access.log main" 2021/06/26 05:09:20 [debug] KONG_ADMIN_ERROR_LOG ENV found with "/usr/local/kong/logs/admin-error.log" 2021/06/26 05:09:20 [debug] KONG_LOG_LEVEL ENV found with "info" 2021/06/26 05:09:20 [debug] KONG_PLUGINS ENV found with "bundled,grpc-itop-verify,grpc-api-log,grpc-error-handler,p-prometheus,correlationid,itop-verify,g-gateway,short-link" 2021/06/26 05:09:20 [debug] admin_access_log = "/usr/local/kong/logs/admin-access.log main" 2021/06/26 05:09:20 [debug] admin_error_log = "/usr/local/kong/logs/admin-error.log" 2021/06/26 05:09:20 [debug] admin_listen = {"0.0.0.0:8001"} 2021/06/26 05:09:20 [debug] admin_ssl_cert = {} 2021/06/26 05:09:20 [debug] admin_ssl_cert_key = {} 2021/06/26 05:09:20 [debug] anonymous_reports = true 2021/06/26 05:09:20 [debug] cassandra_contact_points = {"127.0.0.1"} 2021/06/26 05:09:20 [debug] cassandra_data_centers = {"dc1:2","dc2:3"} 2021/06/26 05:09:20 [debug] cassandra_keyspace = "kong" 2021/06/26 05:09:20 [debug] cassandra_lb_policy = "RequestRoundRobin" 2021/06/26 05:09:20 [debug] cassandra_port = 9042 2021/06/26 05:09:20 [debug] cassandra_read_consistency = "ONE" 2021/06/26 05:09:20 [debug] cassandra_refresh_frequency = 60 2021/06/26 05:09:20 [debug] cassandra_repl_factor = 1 2021/06/26 05:09:20 [debug] cassandra_repl_strategy = "SimpleStrategy" 2021/06/26 05:09:20 [debug] cassandra_schema_consensus_timeout = 10000 2021/06/26 05:09:20 [debug] cassandra_ssl = false 2021/06/26 05:09:20 [debug] cassandra_ssl_verify = false 2021/06/26 05:09:20 [debug] cassandra_timeout = 5000 2021/06/26 05:09:20 [debug] cassandra_username = "kong" 2021/06/26 05:09:20 [debug] cassandra_write_consistency = "ONE" 2021/06/26 05:09:20 [debug] client_body_buffer_size = "8k" 2021/06/26 05:09:20 [debug] client_max_body_size = "0" 2021/06/26 05:09:20 [debug] client_ssl = false 2021/06/26 05:09:20 [debug] cluster_control_plane = "127.0.0.1:8005" 2021/06/26 05:09:20 [debug] cluster_data_plane_purge_delay = 1209600 2021/06/26 05:09:20 [debug] cluster_listen = {"0.0.0.0:8005"} 2021/06/26 05:09:20 [debug] cluster_mtls = "shared" 2021/06/26 05:09:20 [debug] cluster_ocsp = "off" 2021/06/26 05:09:20 [debug] database = "postgres" 2021/06/26 05:09:20 [debug] db_cache_ttl = 0 2021/06/26 05:09:20 [debug] db_cache_warmup_entities = {"services"} 2021/06/26 05:09:20 [debug] db_resurrect_ttl = 30 2021/06/26 05:09:20 [debug] db_update_frequency = 5 2021/06/26 05:09:20 [debug] db_update_propagation = 0 2021/06/26 05:09:20 [debug] dns_error_ttl = 1 2021/06/26 05:09:20 [debug] dns_hostsfile = "/etc/hosts" 2021/06/26 05:09:20 [debug] dns_no_sync = false 2021/06/26 05:09:20 [debug] dns_not_found_ttl = 30 2021/06/26 05:09:20 [debug] dns_order = {"LAST","SRV","A","CNAME"} 2021/06/26 05:09:20 [debug] dns_resolver = {} 2021/06/26 05:09:20 [debug] dns_stale_ttl = 4 2021/06/26 05:09:20 [debug] error_default_type = "text/plain" 2021/06/26 05:09:20 [debug] go_plugins_dir = "off" 2021/06/26 05:09:20 [debug] go_pluginserver_exe = "/usr/local/bin/go-pluginserver" 2021/06/26 05:09:20 [debug] headers = {"server_tokens","latency_tokens"} 2021/06/26 05:09:20 [debug] host_ports = {} 2021/06/26 05:09:20 [debug] kic = false 2021/06/26 05:09:20 [debug] log_level = "info" 2021/06/26 05:09:20 [debug] lua_package_cpath = "" 2021/06/26 05:09:20 [debug] lua_package_path = "./?.lua;./?/init.lua;/usr/local/custom/?.lua;;" 2021/06/26 05:09:20 [debug] lua_socket_pool_size = 30 2021/06/26 05:09:20 [debug] lua_ssl_protocols = "TLSv1.1 TLSv1.2 TLSv1.3" 2021/06/26 05:09:20 [debug] lua_ssl_trusted_certificate = {} 2021/06/26 05:09:20 [debug] lua_ssl_verify_depth = 1 2021/06/26 05:09:20 [debug] mem_cache_size = "128m" 2021/06/26 05:09:20 [debug] nginx_admin_client_body_buffer_size = "10m" 2021/06/26 05:09:20 [debug] nginx_admin_client_max_body_size = "10m" 2021/06/26 05:09:20 [debug] nginx_admin_directives = {{name="client_max_body_size",value="10m"},{name="client_body_buffer_size",value="10m"}} 2021/06/26 05:09:20 [debug] nginx_daemon = "on" 2021/06/26 05:09:20 [debug] nginx_events_directives = {{name="multi_accept",value="on"},{name="worker_connections",value="auto"}} 2021/06/26 05:09:20 [debug] nginx_events_multi_accept = "on" 2021/06/26 05:09:20 [debug] nginx_events_worker_connections = "auto" 2021/06/26 05:09:20 [debug] nginx_http_client_body_buffer_size = "8k" 2021/06/26 05:09:20 [debug] nginx_http_client_max_body_size = "40m" 2021/06/26 05:09:20 [debug] nginx_http_directives = {{name="lua_ssl_protocols",value="TLSv1.1 TLSv1.2 TLSv1.3"},{name="client_max_body_size",value="40m"},{name="client_body_buffer_size",value="8k"},{name="ssl_dhparam",value="ffdhe2048"},{name="log_format",value="main '$remote_addr - $remote_user [$time_local] \"$request\" $status \"$upstream_status\" $body_bytes_sent \"$http_referer\" \"$http_user_agent\" \"$request_time\" \"$upstream_response_time\" \"$upstream_connect_time\" \"$upstream_header_time\"'"},{name="ssl_prefer_server_ciphers",value="off"},{name="ssl_session_tickets",value="on"},{name="ssl_session_timeout",value="1d"},{name="ssl_protocols",value="TLSv1.2 TLSv1.3"}} 2021/06/26 05:09:20 [debug] nginx_http_log_format = "main '$remote_addr - $remote_user [$time_local] \"$request\" $status \"$upstream_status\" $body_bytes_sent \"$http_referer\" \"$http_user_agent\" \"$request_time\" \"$upstream_response_time\" \"$upstream_connect_time\" \"$upstream_header_time\"'" 2021/06/26 05:09:20 [debug] nginx_http_lua_ssl_protocols = "TLSv1.1 TLSv1.2 TLSv1.3" 2021/06/26 05:09:20 [debug] nginx_http_ssl_dhparam = "ffdhe2048" 2021/06/26 05:09:20 [debug] nginx_http_ssl_prefer_server_ciphers = "off" 2021/06/26 05:09:20 [debug] nginx_http_ssl_protocols = "TLSv1.2 TLSv1.3" 2021/06/26 05:09:20 [debug] nginx_http_ssl_session_tickets = "on" 2021/06/26 05:09:20 [debug] nginx_http_ssl_session_timeout = "1d" 2021/06/26 05:09:20 [debug] nginx_http_status_directives = {} 2021/06/26 05:09:20 [debug] nginx_http_upstream_directives = {} 2021/06/26 05:09:20 [debug] nginx_main_daemon = "on" 2021/06/26 05:09:20 [debug] nginx_main_directives = {{name="worker_rlimit_nofile",value="auto"},{name="daemon",value="on"},{name="worker_processes",value="auto"}} 2021/06/26 05:09:20 [debug] nginx_main_worker_processes = "auto" 2021/06/26 05:09:20 [debug] nginx_main_worker_rlimit_nofile = "auto" 2021/06/26 05:09:20 [debug] nginx_optimizations = true 2021/06/26 05:09:20 [debug] nginx_proxy_client_body_buffer_size = "8m" 2021/06/26 05:09:20 [debug] nginx_proxy_client_max_body_size = "8m" 2021/06/26 05:09:20 [debug] nginx_proxy_directives = {{name="client_body_buffer_size",value="8m"},{name="client_max_body_size",value="8m"},{name="keepalive_timeout",value="60s"},{name="proxy_buffers",value="4 256k"},{name="proxy_buffer_size",value="128k"},{name="proxy_busy_buffers_size",value="256k"},{name="real_ip_header",value="X-Real-IP"},{name="real_ip_recursive",value="off"}} 2021/06/26 05:09:20 [debug] nginx_proxy_keepalive_timeout = "60s" 2021/06/26 05:09:20 [debug] nginx_proxy_proxy_buffer_size = "128k" 2021/06/26 05:09:20 [debug] nginx_proxy_proxy_buffers = "4 256k" 2021/06/26 05:09:20 [debug] nginx_proxy_proxy_busy_buffers_size = "256k" 2021/06/26 05:09:20 [debug] nginx_proxy_real_ip_header = "X-Real-IP" 2021/06/26 05:09:20 [debug] nginx_proxy_real_ip_recursive = "off" 2021/06/26 05:09:20 [debug] nginx_sproxy_directives = {} 2021/06/26 05:09:20 [debug] nginx_status_directives = {} 2021/06/26 05:09:20 [debug] nginx_stream_directives = {{name="lua_ssl_protocols",value="TLSv1.1 TLSv1.2 TLSv1.3"},{name="ssl_dhparam",value="ffdhe2048"},{name="ssl_protocols",value="TLSv1.2 TLSv1.3"},{name="ssl_prefer_server_ciphers",value="off"},{name="ssl_session_tickets",value="on"},{name="ssl_session_timeout",value="1d"}} 2021/06/26 05:09:20 [debug] nginx_stream_lua_ssl_protocols = "TLSv1.1 TLSv1.2 TLSv1.3" 2021/06/26 05:09:20 [debug] nginx_stream_ssl_dhparam = "ffdhe2048" 2021/06/26 05:09:20 [debug] nginx_stream_ssl_prefer_server_ciphers = "off" 2021/06/26 05:09:20 [debug] nginx_stream_ssl_protocols = "TLSv1.2 TLSv1.3" 2021/06/26 05:09:20 [debug] nginx_stream_ssl_session_tickets = "on" 2021/06/26 05:09:20 [debug] nginx_stream_ssl_session_timeout = "1d" 2021/06/26 05:09:20 [debug] nginx_supstream_directives = {} 2021/06/26 05:09:20 [debug] nginx_upstream_directives = {} 2021/06/26 05:09:20 [debug] nginx_worker_processes = "auto" 2021/06/26 05:09:20 [debug] pg_database = "kong_pre_outer" 2021/06/26 05:09:20 [debug] pg_host = "172.26.0.45" 2021/06/26 05:09:20 [debug] pg_max_concurrent_queries = 0 2021/06/26 05:09:20 [debug] pg_password = "**" 2021/06/26 05:09:20 [debug] pg_port = 5432 2021/06/26 05:09:20 [debug] pg_ro_ssl = false 2021/06/26 05:09:20 [debug] pg_ro_ssl_verify = false 2021/06/26 05:09:20 [debug] pg_semaphore_timeout = 60000 2021/06/26 05:09:20 [debug] pg_ssl = false 2021/06/26 05:09:20 [debug] pg_ssl_verify = false 2021/06/26 05:09:20 [debug] pg_timeout = 5000 2021/06/26 05:09:20 [debug] pg_user = "kong" 2021/06/26 05:09:20 [debug] plugins = {"bundled","grpc-itop-verify","grpc-api-log","grpc-error-handler","p-prometheus","correlationid","itop-verify","g-gateway","short-link"} 2021/06/26 05:09:20 [debug] pluginserver_names = {} 2021/06/26 05:09:20 [debug] port_maps = {} 2021/06/26 05:09:20 [debug] prefix = "/usr/local/kong/" 2021/06/26 05:09:20 [debug] proxy_access_log = "/usr/local/kong/logs/proxy-access.log main" 2021/06/26 05:09:20 [debug] proxy_error_log = "/usr/local/kong/logs/proxy-error.log" 2021/06/26 05:09:20 [debug] proxy_listen = {"0.0.0.0:8000","0.0.0.0:8443 ssl","0.0.0.0:9080 http2","0.0.0.0:9081 http2 ssl"} 2021/06/26 05:09:20 [debug] proxy_stream_access_log = "logs/access.log basic" 2021/06/26 05:09:20 [debug] proxy_stream_error_log = "logs/error.log" 2021/06/26 05:09:20 [debug] real_ip_header = "X-Real-IP" 2021/06/26 05:09:20 [debug] real_ip_recursive = "off" 2021/06/26 05:09:20 [debug] role = "traditional" 2021/06/26 05:09:20 [debug] ssl_cert = {"/usr/local/custom/kong/ssl/server.pem"} 2021/06/26 05:09:20 [debug] ssl_cert_key = {"/usr/local/custom/kong/ssl/server.key"} 2021/06/26 05:09:20 [debug] ssl_cipher_suite = "intermediate" 2021/06/26 05:09:20 [debug] ssl_ciphers = "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384" 2021/06/26 05:09:20 [debug] ssl_dhparam = "ffdhe2048" 2021/06/26 05:09:20 [debug] ssl_prefer_server_ciphers = "on" 2021/06/26 05:09:20 [debug] ssl_protocols = "TLSv1.1 TLSv1.2 TLSv1.3" 2021/06/26 05:09:20 [debug] ssl_session_tickets = "on" 2021/06/26 05:09:20 [debug] ssl_session_timeout = "1d" 2021/06/26 05:09:20 [debug] status_access_log = "off" 2021/06/26 05:09:20 [debug] status_error_log = "logs/status_error.log" 2021/06/26 05:09:20 [debug] status_listen = {"off"} 2021/06/26 05:09:20 [debug] status_ssl_cert = {} 2021/06/26 05:09:20 [debug] status_ssl_cert_key = {} 2021/06/26 05:09:20 [debug] stream_listen = {"off"} 2021/06/26 05:09:20 [debug] trusted_ips = {} 2021/06/26 05:09:20 [debug] untrusted_lua = "sandbox" 2021/06/26 05:09:20 [debug] untrusted_lua_sandbox_environment = {} 2021/06/26 05:09:20 [debug] untrusted_lua_sandbox_requires = {} 2021/06/26 05:09:20 [debug] upstream_keepalive_idle_timeout = 60 2021/06/26 05:09:20 [debug] upstream_keepalive_max_requests = 100 2021/06/26 05:09:20 [debug] upstream_keepalive_pool_size = 60 2021/06/26 05:09:20 [debug] worker_consistency = "strict" 2021/06/26 05:09:20 [debug] worker_state_update_frequency = 5 2021/06/26 05:09:20 [verbose] prefix in use: /usr/local/kong 2021/06/26 05:09:20 [verbose] preparing nginx prefix directory at /usr/local/kong 2021/06/26 05:09:20 [debug] searching for OpenResty 'nginx' executable 2021/06/26 05:09:20 [debug] /usr/local/openresty/nginx/sbin/nginx -v: 'nginx version: openresty/1.19.3.1' 2021/06/26 05:09:20 [debug] found OpenResty 'nginx' executable at /usr/local/openresty/nginx/sbin/nginx 2021/06/26 05:09:20 [debug] testing nginx configuration: KONG_NGINX_CONF_CHECK=true /usr/local/openresty/nginx/sbin/nginx -t -p /usr/local/kong -c nginx.conf 2021/06/26 05:09:20 [debug] sending signal to pid at: /usr/local/kong/pids/nginx.pid 2021/06/26 05:09:20 [debug] kill -0 cat /usr/local/kong/pids/nginx.pid >/dev/null 2>&1 Error: /usr/local/share/lua/5.1/kong/cmd/start.lua:26: Kong is already running in /usr/local/kong stack traceback: [C]: in function 'assert' /usr/local/share/lua/5.1/kong/cmd/start.lua:26: in function 'cmd_exec' /usr/local/share/lua/5.1/kong/cmd/init.lua:88: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:88> [C]: in function 'xpcall' /usr/local/share/lua/5.1/kong/cmd/init.lua:88: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:45> /usr/local/bin/kong:9: in function 'file_gen' init_worker_by_lua:47: in function [C]: in function 'xpcall' init_worker_by_lua:54: in function
• Kong error logs (<KONG_PREFIX>/logs/error.log)
• 2021/06/26 03:38:31 [warn] 23#0: 9951633 a client request body is buffered to a temporary file /usr/local/kong/client_body_temp/0000113895, client: 8.17.206.83, server: kong, request: "POST /feed.FeedService/ExposureFeed HTTP/2.0", host: "meemo" 2021/06/26 03:38:31 [warn] 23#0: 9951633 a client request body is buffered to a temporary file /usr/local/kong/client_body_temp/0000113896, client: 8.17.206.83, server: kong, request: "POST /feed.FeedService/ExposureFeed HTTP/2.0", host: "meemo" 2021/06/26 03:38:31 [warn] 23#0: 9951633 a client request body is buffered to a temporary file /usr/local/kong/client_body_temp/0000113897, client: 8.17.206.83, server: kong, request: "POST /feed.FeedService/ExposureFeed HTTP/2.0", host: "meemo" 2021/06/26 03:38:31 [warn] 23#0: 9951633 a client request body is buffered to a temporary file /usr/local/kong/client_body_temp/0000113898, client: 8.17.206.83, server: kong, request: "POST /feed.FeedService/ExposureFeed HTTP/2.0", host: "meemo" 2021/06/26 03:38:31 [warn] 23#0: 9951633 a client request body is buffered to a temporary file /usr/local/kong/client_body_temp/0000113899, client: 8.17.206.83, server: kong, request: "POST /feed.FeedService/ExposureFeed HTTP/2.0", host: "meemo" 2021/06/26 03:38:31 [warn] 23#0: 9951633 a client request body is buffered to a temporary file /usr/local/kong/client_body_temp/0000113900, client: 8.17.206.83, server: kong, request: "POST /feed.FeedService/ExposureFeed HTTP/2.0", host: "meemo" 2021/06/26 03:38:31 [warn] 23#0: 9951633 a client request body is buffered to a temporary file /usr/local/kong/client_body_temp/0000113901, client: 8.17.206.83, server: kong, request: "POST /feed.FeedService/ExposureFeed HTTP/2.0", host: "meemo" 2021/06/26 03:38:31 [warn] 23#0: 9951633 a client request body is buffered to a temporary file /usr/local/kong/client_body_temp/0000113902, client: 8.17.206.83, server: kong, request: "POST /feed.FeedService/ExposureFeed HTTP/2.0", host: "meemo" 2021/06/26 03:38:31 [warn] 23#0: *9951633 a client request body is buffered to a temporary file /usr/local/kong/client_body_temp/0000113903, client: 8.17.206.83, server: kong, request: "POST /feed.FeedService/ExposureFeed HTTP/2.0", host: "meemo"
• Kong configuration (the output of a GET request to Kong's Admin port - see https://docs.konghq.com/latest/admin-api/#retrieve-node-information) {"plugins":{"available_on_server":{"jwt":true,"acl":true,"correlation-id":true,"cors":true,"oauth2":true,"tcp-log":true,"udp-log":true,"file-log":true,"http-log":true,"key-auth":true,"hmac-auth":true,"basic-auth":true,"ip-restriction":true,"request-transformer":true,"response-transformer":true,"request-size-limiting":true,"rate-limiting":true,"response-ratelimiting":true,"syslog":true,"loggly":true,"datadog":true,"ldap-auth":true,"statsd":true,"bot-detection":true,"aws-lambda":true,"request-termination":true,"azure-functions":true,"zipkin":true,"pre-function":true,"post-function":true,"prometheus":true,"proxy-cache":true,"session":true,"acme":true,"grpc-web":true,"grpc-gateway":true,"short-link":true,"g-gateway":true,"itop-verify":true,"correlationid":true,"p-prometheus":true,"grpc-error-handler":true,"grpc-api-log":true,"grpc-itop-verify":true},"enabled_in_cluster":["grpc-itop-verify","grpc-error-handler","correlationid","short-link","itop-verify","prometheus"]},"node_id":"c70d009f-46fb-4a65-8133-3f021eb6dc53","tagline":"Welcome to kong","configuration":{"nginx_http_ssl_prefer_server_ciphers":"off","nginx_stream_ssl_prefer_server_ciphers":"off","ssl_dhparam":"ffdhe2048","nginx_http_ssl_dhparam":"ffdhe2048","nginx_stream_ssl_dhparam":"ffdhe2048","ssl_session_tickets":"on","nginx_http_ssl_session_tickets":"on","nginx_stream_ssl_session_tickets":"on","ssl_session_timeout":"1d","nginx_http_ssl_session_timeout":"1d","nginx_stream_ssl_session_timeout":"1d","proxy_access_log":"/usr/local/kong/logs/proxy-access.log main","proxy_error_log":"/usr/local/kong/logs/proxy-error.log","nginx_proxy_client_max_body_size":"8m","error_default_type":"text/plain","nginx_proxy_proxy_buffers":"4 256k","admin_error_log":"/usr/local/kong/logs/admin-error.log","status_access_log":"off","status_error_log":"logs/status_error.log","log_level":"info","nginx_proxy_proxy_buffer_size":"128k","nginx_optimizations":true,"lua_ssl_trusted_certificate":{},"lua_ssl_verify_depth":1,"lua_ssl_protocols":"TLSv1.1 TLSv1.2 TLSv1.3","nginx_http_lua_ssl_protocols":"TLSv1.1 TLSv1.2 TLSv1.3","nginx_stream_lua_ssl_protocols":"TLSv1.1 TLSv1.2 TLSv1.3","client_ssl":false,"stream_proxy_ssl_enabled":false,"enabled_headers":{"X-Kong-Upstream-Latency":true,"X-Kong-Upstream-Status":false,"server_tokens":true,"latency_tokens":true,"X-Kong-Proxy-Latency":true,"Via":true,"X-Kong-Response-Latency":true,"X-Kong-Admin-Latency":true,"Server":true},"status_ssl_enabled":false,"cluster_control_plane":"127.0.0.1:8005","admin_listeners":[{"ip":"0.0.0.0","ssl":false,"http2":false,"proxy_protocol":false,"deferred":false,"reuseport":false,"backlog=%d+":false,"bind":false,"port":8001,"listener":"0.0.0.0:8001"}],"proxy_listeners":[{"ip":"0.0.0.0","ssl":false,"http2":false,"proxy_protocol":false,"deferred":false,"reuseport":false,"backlog=%d+":false,"bind":false,"port":8000,"listener":"0.0.0.0:8000"},{"ip":"0.0.0.0","ssl":true,"http2":false,"proxy_protocol":false,"deferred":false,"reuseport":false,"backlog=%d+":false,"bind":false,"port":8443,"listener":"0.0.0.0:8443 ssl"},{"ip":"0.0.0.0","ssl":false,"http2":true,"proxy_protocol":false,"deferred":false,"reuseport":false,"backlog=%d+":false,"bind":false,"port":9080,"listener":"0.0.0.0:9080 http2"},{"ip":"0.0.0.0","ssl":true,"http2":true,"proxy_protocol":false,"deferred":false,"reuseport":false,"backlog=%d+":false,"bind":false,"port":9081,"listener":"0.0.0.0:9081 ssl http2"}],"stream_listeners":{},"headers":["server_tokens","latency_tokens"],"database":"postgres","pluginserver_names":{},"worker_consistency":"strict","nginx_main_directives":[{"value":"off","name":"daemon"},{"value":"auto","name":"worker_processes"},{"value":"auto","name":"worker_rlimit_nofile"}],"nginx_events_directives":[{"value":"on","name":"multi_accept"},{"value":"auto","name":"worker_connections"}],"db_update_frequency":5,"nginx_http_directives":[{"value":"8k","name":"client_body_buffer_size"},{"value":"40m","name":"client_max_body_size"},{"value":"main '$remote_addr - $remote_user [$time_local] \"$request\" $status \"$upstream_status\" $body_bytes_sent \"$http_referer\" \"$http_user_agent\" \"$request_time\" \"$upstream_response_time\" \"$upstream_connect_time\" \"$upstream_header_time\"'","name":"log_format"},{"value":"prometheus_metrics 5m","name":"lua_shared_dict"},{"value":"TLSv1.1 TLSv1.2 TLSv1.3","name":"lua_ssl_protocols"},{"value":"/usr/local/kong/ssl/ffdhe2048.pem","name":"ssl_dhparam"},{"value":"off","name":"ssl_prefer_server_ciphers"},{"value":"TLSv1.2 TLSv1.3","name":"ssl_protocols"},{"value":"on","name":"ssl_session_tickets"},{"value":"1d","name":"ssl_session_timeout"}],"db_update_propagation":0,"dns_resolver":{},"dns_hostsfile":"/etc/hosts","dns_error_ttl":1,"dns_not_found_ttl":30,"dns_stale_ttl":4,"dns_order":["LAST","SRV","A","CNAME"],"dns_no_sync":false,"nginx_admin_directives":[{"value":"10m","name":"client_body_buffer_size"},{"value":"10m","name":"client_max_body_size"}],"nginx_stream_directives":[{"value":"stream_prometheus_metrics 5m","name":"lua_shared_dict"},{"value":"TLSv1.1 TLSv1.2 TLSv1.3","name":"lua_ssl_protocols"},{"value":"/usr/local/kong/ssl/ffdhe2048.pem","name":"ssl_dhparam"},{"value":"off","name":"ssl_prefer_server_ciphers"},{"value":"TLSv1.2 TLSv1.3","name":"ssl_protocols"},{"value":"on","name":"ssl_session_tickets"},{"value":"1d","name":"ssl_session_timeout"}],"nginx_main_worker_rlimit_nofile":"auto","status_listeners":{},"nginx_events_worker_connections":"auto","cluster_listeners":[{"ip":"0.0.0.0","ssl":false,"http2":false,"proxy_protocol":false,"deferred":false,"reuseport":false,"backlog=%d+":false,"bind":false,"port":8005,"listener":"0.0.0.0:8005"}],"nginx_events_multi_accept":"on","nginx_http_upstream_directives":{},"nginx_http_status_directives":{},"nginx_pid":"/usr/local/kong/pids/nginx.pid","nginx_err_logs":"/usr/local/kong/logs/error.log","nginx_acc_logs":"/usr/local/kong/logs/access.log","admin_acc_logs":"/usr/local/kong/logs/admin_access.log","nginx_conf":"/usr/local/kong/nginx.conf","anonymous_reports":true,"nginx_kong_conf":"/usr/local/kong/nginx-kong.conf","nginx_kong_stream_conf":"/usr/local/kong/nginx-kong-stream.conf","nginx_admin_client_body_buffer_size":"10m","kong_env":"/usr/local/kong/.kong_env","ssl_cert_csr_default":"/usr/local/kong/ssl/kong-default.csr","cassandra_timeout":5000,"ssl_cert_default":"/usr/local/kong/ssl/kong-default.crt","mem_cache_size":"128m","ssl_cert_key_default":"/usr/local/kong/ssl/kong-default.key","ssl_cert_default_ecdsa":"/usr/local/kong/ssl/kong-default-ecdsa.crt","proxy_ssl_enabled":true,"ssl_cert_key_default_ecdsa":"/usr/local/kong/ssl/kong-default-ecdsa.key","plugins":["bundled","grpc-itop-verify","grpc-api-log","grpc-error-handler","p-prometheus","correlationid","itop-verify","g-gateway","short-link"],"client_ssl_cert_default":"/usr/local/kong/ssl/kong-default.crt","client_ssl_cert_key_default":"/usr/local/kong/ssl/kong-default.key","admin_ssl_cert_default":"/usr/local/kong/ssl/admin-kong-default.crt","admin_ssl_cert_key_default":"/usr/local/kong/ssl/admin-kong-default.key","untrusted_lua":"sandbox","admin_ssl_cert_default_ecdsa":"/usr/local/kong/ssl/admin-kong-default-ecdsa.crt","untrusted_lua_sandbox_environment":{},"prefix":"/usr/local/kong","status_ssl_cert_default":"/usr/local/kong/ssl/status-kong-default.crt","pg_user":"kong","status_ssl_cert_key_default":"/usr/local/kong/ssl/status-kong-default.key","status_ssl_cert_default_ecdsa":"/usr/local/kong/ssl/status-kong-default-ecdsa.crt","go_plugins_dir":"off","status_ssl_cert_key_default_ecdsa":"/usr/local/kong/ssl/status-kong-default-ecdsa.key","admin_ssl_enabled":false,"cluster_mtls":"shared","admin_ssl_cert_key_default_ecdsa":"/usr/local/kong/ssl/admin-kong-default-ecdsa.key","untrusted_lua_sandbox_requires":{},"go_pluginserver_exe":"/usr/local/bin/go-pluginserver","ssl_cipher_suite":"intermediate","cassandra_ssl":false,"cassandra_ssl_verify":false,"worker_state_update_frequency":5,"nginx_http_log_format":"main '$remote_addr - $remote_user [$time_local] \"$request\" $status \"$upstream_status\" $body_bytes_sent \"$http_referer\" \"$http_user_agent\" \"$request_time\" \"$upstream_response_time\" \"$upstream_connect_time\" \"$upstream_header_time\"'","nginx_supstream_directives":{},"port_maps":{},"proxy_listen":["0.0.0.0:8000","0.0.0.0:8443 ssl","0.0.0.0:9080 http2","0.0.0.0:9081 http2 ssl"],"admin_listen":["0.0.0.0:8001"],"status_listen":["off"],"stream_listen":["off"],"cluster_listen":["0.0.0.0:8005"],"admin_ssl_cert":{},"admin_ssl_cert_key":{},"status_ssl_cert":{},"status_ssl_cert_key":{},"db_resurrect_ttl":30,"proxy_stream_access_log":"logs/access.log basic","proxy_stream_error_log":"logs/error.log","nginx_daemon":"off","cassandra_username":"kong","nginx_main_daemon":"off","nginx_worker_processes":"auto","nginx_main_worker_processes":"auto","host_ports":{},"trusted_ips":{},"real_ip_header":"X-Real-IP","nginx_proxy_real_ip_header":"X-Real-IP","real_ip_recursive":"off","nginx_proxy_real_ip_recursive":"off","client_max_body_size":"0","nginx_http_client_max_body_size":"40m","client_body_buffer_size":"8k","nginx_http_client_body_buffer_size":"8k","pg_port":5432,"pg_password":"**","pg_ssl":false,"pg_ssl_verify":false,"pg_max_concurrent_queries":0,"pg_semaphore_timeout":60000,"loaded_plugins":{"jwt":true,"acl":true,"correlation-id":true,"cors":true,"oauth2":true,"tcp-log":true,"udp-log":true,"file-log":true,"http-log":true,"key-auth":true,"hmac-auth":true,"basic-auth":true,"ip-restriction":true,"request-transformer":true,"response-transformer":true,"request-size-limiting":true,"rate-limiting":true,"response-ratelimiting":true,"syslog":true,"loggly":true,"datadog":true,"ldap-auth":true,"statsd":true,"bot-detection":true,"aws-lambda":true,"request-termination":true,"azure-functions":true,"zipkin":true,"pre-function":true,"post-function":true,"prometheus":true,"proxy-cache":true,"session":true,"acme":true,"grpc-web":true,"grpc-gateway":true,"grpc-itop-verify":true,"grpc-api-log":true,"grpc-error-handler":true,"p-prometheus":true,"correlationid":true,"itop-verify":true,"g-gateway":true,"short-link":true},"ssl_ciphers":"ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384","db_cache_ttl":0,"pg_ro_ssl":false,"pg_ro_ssl_verify":false,"nginx_proxy_directives":[{"value":"8m","name":"client_body_buffer_size"},{"value":"8m","name":"client_max_body_size"},{"value":"60s","name":"keepalive_timeout"},{"value":"128k","name":"proxy_buffer_size"},{"value":"4 256k","name":"proxy_buffers"},{"value":"256k","name":"proxy_busy_buffers_size"},{"value":"X-Real-IP","name":"real_ip_header"},{"value":"off","name":"real_ip_recursive"}],"pg_host":"172.26.0.45","cassandra_contact_points":["127.0.0.1"],"cassandra_port":9042,"admin_access_log":"/usr/local/kong/logs/admin-access.log main","lua_package_cpath":"","ssl_cert_key":["/usr/local/custom/kong/ssl/server.key"],"cassandra_write_consistency":"ONE","pg_database":"kong_pre_outer","cassandra_keyspace":"kong","db_cache_warmup_entities":["services"],"pg_timeout":5000,"lua_package_path":"./?.lua;./?/init.lua;/usr/local/custom/?.lua;;","upstream_keepalive_idle_timeout":60,"cluster_data_plane_purge_delay":1209600,"nginx_proxy_proxy_busy_buffers_size":"256k","cassandra_read_consistency":"ONE","upstream_keepalive_max_requests":100,"kic":false,"cassandra_lb_policy":"RequestRoundRobin","ssl_cert":["/usr/local/custom/kong/ssl/server.pem"],"nginx_proxy_keepalive_timeout":"60s","nginx_upstream_directives":{},"upstream_keepalive_pool_size":60,"nginx_status_directives":{},"cassandra_refresh_frequency":60,"cassandra_repl_strategy":"SimpleStrategy","nginx_admin_client_max_body_size":"10m","nginx_sproxy_directives":{},"cassandra_repl_factor":1,"cassandra_data_centers":["dc1:2","dc2:3"],"cassandra_schema_consensus_timeout":10000,"cluster_ocsp":"off","lua_socket_pool_size":30,"role":"traditional","ssl_protocols":"TLSv1.1 TLSv1.2 TLSv1.3","nginx_proxy_client_body_buffer_size":"8m","nginx_http_ssl_protocols":"TLSv1.2 TLSv1.3","nginx_stream_ssl_protocols":"TLSv1.2 TLSv1.3","ssl_prefer_server_ciphers":"on"},"timers":{"pending":9,"running":1},"pids":{"master":1,"workers":[23,24,25,26]},"lua_version":"LuaJIT 2.1.0-beta3","version":"2.4.1","hostname":"kong-outer-699c44c559-frtd8"}
• Operating system centos 7

[oldthreefeng]

any update ? same as kong 2.3.3

[hanshuebner]

Please update to a newer version of Kong Gateway and let us know if the problem still exists.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions[bot]]

Dear contributor,

We are automatically closing this issue because it has not seen any activity for three weeks. We're sorry that your issue could not be resolved. If any new information comes up that could help resolving it, please feel free to reopen it.

Your contribution is greatly appreciated!

Please have a look our pledge to the community for more information.

Sincerely, Your Kong Gateway team