stream_api support breaks exporter

samsk commented 3 years ago

stream_api support introduced by d125bf35ce2b39426609ce09c94ef21771279314 breaks the plugin.

Without kong streaming enabled it results in:

kong                | stack traceback:
kong                |   [C]: in function 'assert'
kong                |   /usr/local/share/lua/5.1/kong/tools/stream_api.lua:47: in function 'request'
kong                |   ...local/share/lua/5.1/kong/plugins/prometheus/exporter.lua:322: in function 'collect'
kong                |   /usr/local/share/lua/5.1/kong/plugins/prometheus/api.lua:6: in function 'fn'
kong                |   /usr/local/share/lua/5.1/kong/api/api_helpers.lua:258: in function </usr/local/share/lua/5.1/kong/api/api_helpers.lua:241>
kong                |
kong                | stack traceback:
kong                |   [C]: in function 'error'
kong                |   /usr/local/share/lua/5.1/lapis/application.lua:401: in function 'handler'
kong                |   /usr/local/share/lua/5.1/lapis/application.lua:130: in function 'resolve'
kong                |   /usr/local/share/lua/5.1/lapis/application.lua:167: in function </usr/local/share/lua/5.1/lapis/application.lua:165>
kong                |   [C]: in function 'xpcall'
kong                |   /usr/local/share/lua/5.1/lapis/application.lua:173: in function 'dispatch'
kong                |   /usr/local/share/lua/5.1/lapis/nginx.lua:230: in function 'serve'
kong                |   /usr/local/share/lua/5.1/kong/init.lua:1319: in function 'admin_content'
kong                |   content_by_lua(nginx-kong.conf:311):2: in main chunk, client: 10.17.6.1, server: kong_admin, request: "GET /metrics HTTP/1.1", host: "localhost:8001"
kong                | 2021/02/15 16:13:44 [error] 22#0: *1481647 lua entry thread aborted: runtime error: /usr/local/share/lua/5.1/lapis/application.lua:141: headers have already been sent
kong                | stack traceback:
kong                | coroutine 0:
kong                |   [C]: in function 'error'
kong                |   /usr/local/share/lua/5.1/kong/pdk/response.lua:785: in function 'handle_error'
kong                |   /usr/local/share/lua/5.1/lapis/application.lua:141: in function 'render_error_request'
kong                |   /usr/local/share/lua/5.1/lapis/application.lua:177: in function 'dispatch'
kong                |   /usr/local/share/lua/5.1/lapis/nginx.lua:230: in function 'serve'
kong                |   /usr/local/share/lua/5.1/kong/init.lua:1319: in function 'admin_content'
kong                |   content_by_lua(nginx-kong.conf:311):2: in main chunk, client: 10.17.6.1, server: kong_admin, request: "GET /metrics HTTP/1.1", host: "localhost:8001"

and curl connection to /metrics is broken.

With streaming enabled

kong                | 2021/02/15 16:28:03 [error] 26#0: *433 lua entry thread aborted: runtime error: ...local/share/lua/5.1/kong/plugins/prometheus/exporter.lua:240: attempt to index field 'location' (a nil value)
kong                | stack traceback:
kong                | coroutine 0:
kong                |   ...local/share/lua/5.1/kong/plugins/prometheus/exporter.lua: in function 'f'
kong                |   /usr/local/share/lua/5.1/kong/tools/stream_api.lua:89: in function 'handle'
kong                |   /usr/local/share/lua/5.1/kong/init.lua:1401: in function 'stream_api'
kong                |   content_by_lua(nginx-kong-stream.conf:92):2: in main chunk, udp client: unix:, server: unix:/usr/local/kong/stream_rpc.sock
kong                | 10.17.6.1 - - [15/Feb/2021:16:29:03 +0000] "GET /metrics HTTP/1.1" 200 3028 "-" "curl/7.68.0"

and curl connection is stalled for few seconds, finishing with niltimeout non-metric text.

PS: I wanted to use HEAD to add some custom features, therefore I'm not using last release, but HEAD is now broken, so this issue is here to track it. I'll possibly use some previous commit for my work, but that will make the possibility to upstream it harder...

kikito commented 3 years ago

Thank you for reporting this. Could you tell us anything else about your system that might help us replicate (for example your Kong version or any other plugins you have installed)?

samsk commented 3 years ago

Sorry, I've forgotten. I've latest docker kong 2.3.2 and HEAD of the prometheus plugin. All other plugins are disabled.

docker config:

      - LC_CTYPE=en_US.UTF-8
      - LC_ALL=en_US.UTF-8
      - KONG_LUA_PACKAGE_PATH=/plugins/?.lua
      - KONG_PLUGINS=bundled,${KONG_CUSTOM_PLUGINS}
      - KONG_LOG_LEVEL
      - KONG_CASSANDRA_CONTACT_POINTS=kong-database
      - KONG_PROXY_ACCESS_LOG=/dev/stdout
      - KONG_ADMIN_ACCESS_LOG=/dev/stdout
      - KONG_PROXY_ERROR_LOG=/dev/stderr
      - KONG_ADMIN_ERROR_LOG=/dev/stderr
      - KONG_ADMIN_LISTEN=0.0.0.0:8001, 0.0.0.0:8444 ssl
      - KONG_TRUSTED_IPS=0.0.0.0/0,::/0
      - KONG_ANONYMOUS_REPORTS=off
#      - KONG_STREAM_LISTEN=0.0.0.0:8888
      - KONG_HEADERS=
      - KONG_PG_HOST
      - KONG_PG_USER
      - KONG_PG_PASSWORD
      - KONG_PG_DATABASE

Maybe for clarification, curl to /metrics is broken after printing out metric_data(), and in log appears error stack trace.

Commenting out line 307 in exporter.lua - https://github.com/Kong/kong-plugin-prometheus/commit/d125bf35ce2b39426609ce09c94ef21771279314#diff-f73ec5229d22e2e8cfafdca76e8663e0b04f960a1e2a2cac361ea2e6f07285e7R307 fixes the problem.

fffonion commented 3 years ago

link to PR that will fix this https://github.com/Kong/kong-plugin-prometheus/pull/108/files#diff-f73ec5229d22e2e8cfafdca76e8663e0b04f960a1e2a2cac361ea2e6f07285e7R237

samsk commented 3 years ago

Fixed in 1.1.0. Thanks

Kong / kong-plugin-prometheus

stream_api support breaks exporter #111