search

encode / apistar

The Web API toolkit. 🛠
https://docs.apistar.com
BSD 3-Clause "New" or "Revised" License
5.57k stars 411 forks source link

Memory Leak #606

Closed leetreveil closed 5 years ago

leetreveil commented 6 years ago

There's a memory leak in APIStar that is resulting in the allocation of around 3kB (YMMV) of mem per second that is not being collected by the GC.

mem-graph-leak

Repro:

from apistar import App, Route, http

def welcome():
    return {'message': 'Welcome to API Star!'}

class LeakingHook:
    def on_request(self, req: http.Request):
        pass

routes = [
    Route('/', method='GET', handler=welcome)
]

app = App(routes=routes, event_hooks=[LeakingHook])

if __name__ == '__main__':
    app.serve('127.0.0.1', 5000)

To reproduce this start the above script and hit the welcome endpoint with apache bench:

while; do ab -n 100000 -c 100 127.0.0.1:5000/; done

I believe the memory leak is coming from the inspect call here: https://github.com/encode/apistar/blob/efc084421ab01c9cd7388463ff5c0ace6287e7ca/apistar/server/components.py#L36

More on that soon once I instrument the app with tracemalloc.

leetreveil commented 6 years ago

And here's the results from tracemalloc:

[ Top 10 differences ]
/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/inspect.py:2760: size=1802 KiB (+1802 KiB), count=10979 (+10979), average=168 B
/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/inspect.py:2724: size=1801 KiB (+1801 KiB), count=10978 (+10978), average=168 B
/usr/local/lib/python3.6/site-packages/apistar/server/injector.py:81: size=1175 KiB (+1175 KiB), count=12536 (+12536), average=96 B
<frozen importlib._bootstrap_external>:487: size=1166 KiB (+1166 KiB), count=12589 (+12589), average=95 B
/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/inspect.py:510: size=772 KiB (+772 KiB), count=10985 (+10985), average=72 B
/usr/local/lib/python3.6/site-packages/apistar/server/injector.py:30: size=772 KiB (+772 KiB), count=10977 (+10977), average=72 B
/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/linecache.py:137: size=502 KiB (+502 KiB), count=4992 (+4992), average=103 B
/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/inspect.py:3037: size=490 KiB (+490 KiB), count=7836 (+7836), average=64 B
/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/tracemalloc.py:449: size=442 KiB (+442 KiB), count=4399 (+4399), average=103 B
/usr/local/lib/python3.6/site-packages/apistar/server/components.py:14: size=416 KiB (+416 KiB), count=7840 (+7840), average=54 B
127.0.0.1 - - [13/Aug/2018 14:43:54] "GET / HTTP/1.0" 200 -

And a full traceback from one of the inspect.signature calls that's leaking:

328 memory blocks: 53.8 KiB
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/inspect.py", line 2760
    for param in parameters))
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/inspect.py", line 2173
    __validate_parameters__=is_duck_function)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/inspect.py", line 2262
    return _signature_from_function(sigcls, obj)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/inspect.py", line 2195
    sigcls=sigcls)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/inspect.py", line 2787
    follow_wrapper_chains=follow_wrapped)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/inspect.py", line 3037
    return Signature.from_callable(obj, follow_wrapped=follow_wrapped)
  File "/usr/local/lib/python3.6/site-packages/apistar/server/injector.py", line 32
    signature = inspect.signature(func)
  File "/usr/local/lib/python3.6/site-packages/apistar/server/injector.py", line 69
    parent_parameter=parameter
  File "/usr/local/lib/python3.6/site-packages/apistar/server/injector.py", line 69
    parent_parameter=parameter
  File "/usr/local/lib/python3.6/site-packages/apistar/server/injector.py", line 89
    func_steps = self.resolve_function(func, seen_state=seen_state, set_return=True)
  File "/usr/local/lib/python3.6/site-packages/apistar/server/injector.py", line 100
    steps = self.resolve_functions(funcs)
  File "/usr/local/lib/python3.6/site-packages/apistar/server/app.py", line 227
    return self.injector.run(funcs, state)
  File "/usr/local/lib/python3.6/site-packages/werkzeug/serving.py", line 258
    application_iter = app(environ, start_response)
  File "/usr/local/lib/python3.6/site-packages/werkzeug/serving.py", line 270
    execute(self.server.app)
  File "/usr/local/lib/python3.6/site-packages/werkzeug/serving.py", line 328
    return self.run_wsgi()
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/server.py", line 418
    self.handle_one_request()
  File "/usr/local/lib/python3.6/site-packages/werkzeug/serving.py", line 293
    rv = BaseHTTPRequestHandler.handle(self)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/socketserver.py", line 696
    self.handle()
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/socketserver.py", line 361
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/socketserver.py", line 348
    self.finish_request(request, client_address)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/socketserver.py", line 317
    self.process_request(request, client_address)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/socketserver.py", line 238
    self._handle_request_noblock()
  File "/usr/local/lib/python3.6/site-packages/werkzeug/serving.py", line 612
    HTTPServer.serve_forever(self)
  File "/usr/local/lib/python3.6/site-packages/werkzeug/serving.py", line 777
    srv.serve_forever()
  File "/usr/local/lib/python3.6/site-packages/werkzeug/serving.py", line 814
    inner()

Repro scripts: https://gist.github.com/leetreveil/f36e3ff998e64cce44ee0bb21b7cbcce

leetreveil commented 6 years ago

I can also confirm that using old-style event hooks:

https://github.com/encode/apistar/blob/efc084421ab01c9cd7388463ff5c0ace6287e7ca/apistar/server/app.py#L132

Do not leak. So it's highly likely that the introduction of new style event hooks that get instantiated on every request are the cause:

https://github.com/encode/apistar/pull/475/files

leetreveil commented 6 years ago

Found it with the help of objgraph.

chain

New style event hooks get put into the resolver cache on every request:

https://github.com/encode/apistar/blob/efc084421ab01c9cd7388463ff5c0ace6287e7ca/apistar/server/injector.py#L96

tomchristie commented 5 years ago

Closing this off given that 0.6 is moving to a framework-agnostic suite of API tools, and will no longer include the server. See https://discuss.apistar.org/t/api-star-as-a-framework-independant-tool/614 and #624.