Open teocns opened 3 years ago
My suggestion to you is to use wsgi standard and use uwsgi to serve it. I'll look into this to see where is the leak come from.
Thanks for the suggestion. WSGI seems an unnecessarily complex solution in my case, whereas a docker with maxrss
could easily solve the issue.
But in either way, I am looking forward to identify the memory leak
wsgi is just a callable that accepts two arguments env
dict and start_response
function (to indicate response code and headers) and then return the body iterable, like this
# web.py
def application(env, start_response):
start_response('200 OK', [('Content-Type','text/html')])
return [b"Hello World"]
the above two lines function is a complete wsgi
application
you can serve it in million ways, my favorite is cherrypy and uwsgi. for example you can run it with (different distros might have different names/version for python plugin, different package names)
uwsgi --plugin python3 -w web:application --http-socket=0.0.0.0:8080
I took a look at your code, I don't see any obvious leak, are you sure that self.handle_request(js)
does not have a leak
def __init__(self, *args, **kwargs):
self.handle_request = kwargs['handle_request']
# ...
response = self.handle_request(js)
BTW uwsgi has the following options, even if your handle_request(js)
has a leak, it can configured to start fresh after number of requests or passing time.
-R|--max-requests reload workers after the specified amount of managed requests
--min-worker-lifetime number of seconds worker must run before being reloaded (default is 60)
--max-worker-lifetime reload workers after the specified amount of seconds (default is disabled)
Will definitely give a try to cherrypy.
response = self.handle_request(js)
Just invokes a further request
def handle_request(**kwargs):
return requests.post(**kwargs)
It is substantially a scraper service. I have defined handle_request in a very broad way.
After around 24 hours of running and accommodating 100.000s of requests, the script never frees memory and builds up to even 32GB.