kong.response.{error,exit} never returns

[Carlotronics]

Kong version: 2.6.0 kong-python-pdk version: 0.2.7

When calling kong.response.error or kong.response.exit in Plugin.access, they exit the client request as expected but don't return, resulting in a ghost thread. It seems the problem could come from Kong upstream as pdk doesn't see any response after the method submission on the rpc channel (https://github.com/Kong/kong-python-pdk/blob/master/kong_pdk/pdk/__init__.py#L22).

Steps to reproduce

Create a python plugin with this access function and enable it:

class Plugin(object):
    def __init__(self, config):
        self.config = config

    def access(self, kong: kong.kong):
        return kong.response.error(403, "Access forbidden")

Start querying your kong instance in an infinite loop: the threads started and not freed by kong-python-pdk won't stop growing.

[jdomag]

Hi, I'm affected by this as well, @fffonion did you have a chance to take a look at this?

[mpromny]

Hi @fffonion , We are also affected by this bug. Unfortunately in our case it ends with: mp_rpc.lua:308 [py-plugin] can't start new thread, and pod restarts.

We are using: Kong version: 2.5.0 kong-python-pdk version: 0.2.7 Phase: Rewrite

Steps to reproduce

import requests
import kong_pdk.pdk.kong as kong

Schema = (
    {"message": {"type": "string"}},
)

version = '0.1.0'
priority = 1100

def url_ok(url):
    r = requests.head(url)
    return r.status_code == 200

class Plugin(object):
    def __init__(self, config):
        self.config = config

    def rewrite(self, kong: kong):
        uri, err_uri = kong.request.get_path()
        if url_ok("http://127.0.0.1:8100/status"):
          kong.response.exit(200, "Kong is working")

[abhisheksharma2805]

I am facing this same issue. In my case I tried with both single threaded and multiprocessing mode

Normal Single threaded mode: I see increasingly high memory consumption because of ghost threads and after some time Kong pods memory exceeds the pods limit and then the plugin server restarts. This plugin server restart has also not been handled correctly. In file https://github.com/Kong/kong/blob/master/kong/runloop/plugin_servers/init.lua, variable running_instances is not being reset when the python plugin server restarts. It is still maintaining the old mapping of plugin instances. I have already created an issue related to this in Kong repo https://github.com/Kong/kong/issues/8370

Multiprocessing mode When I run the server in multiprocessing mode, one by one each process becomes ghost because of kong.response.error or kong.response.exit and all the incoming requests are getting timed out after 60sec and responding with 499 errors

2022/02/10 11:24:50 [error] 37#0: *34244 [kong] mp_rpc.lua:308 [auth_plugin] no data, client: 46.19.86.5, server: kong, request: "GET /v1/users HTTP/1.1", host: "zzz.yyy.com"
...
...
2022/02/10 11:24:58 [info] 37#0: *34482 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while connecting to upstream, client: 100.36.63.49, server: kong, request: "GET /v1/users HTTP/1.1", upstream: "https://35.164.117.120:443/v1/users ", host: "zzz.yyy.com"

46.19.86.5 - - [10/Feb/2022:11:24:58 +0000] "GET /v1/users 499 0 "-" "abc"

For now I am thinking of scheduling daily restart for Kong pods (in normal single threaded mode) to avoid pods reaching memory limit. Please resolve this at high priority. I don't want to rewrite the plugins in Lua/Go.

[fffonion]

@mpromny @abhisheksharma2805 That's very good observation! I have pushed a fix #39. Will release after I fixes several other issues.

[abhisheksharma2805]

@fffonion The issue does not seem to be resolved. I am still seeing ghost threads and total active thread count is also increasing continuously

[mpromny]

@fffonion Thank you for fixing this. We are still testing and monitoring resource usage (especially memory), but it seems to be working.

Current image (Kong 2.7, python-pdk: 0.3) 
Number of requests: ~2000
Threads:        6

Old image     (Kong 2.5, python-pdk: 0.27) 
Number of requests: ~2000
Threads:        2002

[abhisheksharma2805]

@mpromny Can you please share if you are still experiencing any ghost threads or memory issues or is everything working fine. I am also using Kong 2.7 and pdk 0.3, but I am still seeing ghost threads and after some time it is unable to create more threads @Carlotronics @jdomag Have you guys tried out the latest pdk yet?

[fffonion]

@abhisheksharma2805 could you share your plugin code that can reproduce this problem? we can discuss in a a new issue.

[mpromny]

@abhisheksharma2805 After we applied fix (pdk in 0.3) the problem has been solved. Memory usage has also decreased.

[abhisheksharma2805]

@fffonion Sorry for replying this late, there was some issue on my side which I resolved. Certainly there are no ghost threads now. But memory usage is still the problem. Raised a separate issue: https://github.com/Kong/kong-python-pdk/issues/59