search

Kong / kong-python-pdk

Write Kong plugins in Python (Experimental)
Apache License 2.0
40 stars 14 forks source link

Python plugin occassionally has error "starting instance: no data", when continuously sending requests #142

Closed martijndxc closed 2 months ago

martijndxc commented 2 months ago

We have created a custom plugin in Python using the kong pdk. The plugin works and runs as expected. When only executing a few requests or in intervals it runs without problems. However when running continous requests with a 0.1 or 0.5 second interval most requests succeed. However after a while ocassionally a request fails with an error "starting instance: no data"

2024/05/06 12:18:11 [notice] 2906#0: *937 [kong] process.lua:223 Starting xsdvalidate, context: ngx.timer
2024/05/06 12:18:11 [error] 2906#0: *782 [kong] init.lua:249 [xsdvalidate] starting instance: no data, client: 172.18.0.1, server: kong, request: "POST /request HTTP/1.1", host: "127.0.0.1:8000", request_id: "8f8f58376334e8617313882aed69434d"
2024/05/06 12:18:11 [error] 2906#0: *782 [kong] init.lua:417 [xsdvalidate] ...local/share/lua/5.1/kong/runloop/plugin_servers/init.lua:252: no data, client: 172.18.0.1, server: kong, request: "POST /request HTTP/1.1", host: "127.0.0.1:8000", request_id: "8f8f58376334e8617313882aed69434d"
172.18.0.1 - - [06/May/2024:12:18:11 +0000] "POST /request HTTP/1.1" 500 97 "-" "insomnia/2023.5.8" kong_request_id: "8f8f58376334e8617313882aed69434d"

If i scroll back in the log it appears one of the nginx worker processes has crashed which may be related:

` 024/05/06 12:18:11 [alert] 1#0: worker process 2468 exited on signal 11 (core dumped) 2024/05/06 12:18:11 [error] 2471#0: 768 [kong] mp_rpc.lua:348 [xsdvalidate] no data, client: 172.18.0.1, server: kong, request: "POST /request HTTP/1.1", host: "127.0.0.1:8000", request_id: "1a9ead33819e56740999cc1a76f44034" 2024/05/06 12:18:11 [error] 2789#0: 766 [kong] init.lua:249 [xsdvalidate] starting instance: no data, client: 172.18.0.1, server: kong, request: "POST /request HTTP/1.1", host: "127.0.0.1:8000", request_id: "d6702020d94d6a1325fdb72310aa944e" 2024/05/06 12:18:11 [error] 2789#0: *766 [kong] init.lua:417 [xsdvalidate] ...local/share/lua/5.1/kong/runloop/plugin_servers/init.lua:252: no data, client: 172.18.0.1, server: kong, request: "POST /request HTTP/1.1", host: "127.0.0.1:8000", request_id: "d6702020d94d6a1325fdb72310aa944e" 2024/05/06 12:18:11 [notice] 1#0: start worker process 2906 2024/05/06 12:18:11 [notice] 1#0: signal 29 (SIGIO) received `

Kong image:

Kong pdk version: kong-pdk==0.36

The plugin server commands used are shown below.

      KONG_PLUGINS: "bundled,xsdvalidate" # Plugin name is your Python file name
      KONG_PLUGINSERVER_NAMES: "xsdvalidate"
      KONG_PLUGINSERVER_XSDVALIDATE_SOCKET: "/usr/local/kong/xsdvalidate.sock"
      KONG_PLUGINSERVER_XSDVALIDATE_START_CMD: "/opt/conf/kong-py-plugins/xsdvalidate.py -v --no-lua-style"
      KONG_PLUGINSERVER_XSDVALIDATE_QUERY_CMD: "/opt/conf/kong-py-plugins/xsdvalidate.py --dump --no-lua-style"

Plugin is run using dedicated server using following code:

#add below section to allow this plugin optionally be running in a dedicated process
if __name__ == "__main__":
   from kong_pdk.cli import start_dedicated_server
   start_dedicated_server(pluginName, Plugin, version, priority, Schema)

How can this be prevented ? It is important that our gateway is able to handle incoming requests reliably without crashing.