search

Benoss / django-elasticsearch-debug-toolbar

A Django Debug Toolbar panel for Elasticsearch
https://pypi.python.org/pypi/django-elasticsearch-debug-toolbar/
MIT License
29 stars 19 forks source link

Elasticsearch debug toolbar causes memory leak #7

Open igo opened 7 years ago

igo commented 7 years ago

Using Elasticsearch debug toolbar causes memory leak and thus crash when elasticsearch is extensively used. Not sure if this is general debug-toolbar problem or specific for elasticsearch-debug-toolbar but I did not notice this problem with other panels. Seems like after each elastic query an elastic_panel.panel.ElasticQueryInfo instance is created which resides in memory forever. I used http://mg.pov.lt/objgraph/objgraph.html#objgraph.show_growth to find out what objects are kept in memory. During tests I ended up with more than 100.000 of ElasticQueryInfo instances.

Benoss commented 7 years ago

I am not surprised, Django itself as leaks in DEBUG mode because it is keeping some references. But in this case those objects should be cleared when collector.clear_collection() is called. And this is done for every request. This is probably a django debug toolbar issue.

igo commented 7 years ago

It leaks even with DEBUG=False. Question is why is this leak relevant only for django-elasticsearch-debug-toolbar and not for every debug-toolbar plugin?

Benoss commented 7 years ago

I'will try to investigate, but if you have any idea do not hesitate :)

ggaughan commented 7 years ago

I think the patched_log_request_success keeps a history for every thread (DEBUG or not) (invoked by Connections), but suspect that the collector.clear_collection() only runs for the thread in charge of the panel (and only when DEBUG), leaving the other thread histories to leak.

Benoss commented 7 years ago

Good catch, The collector is part of core debug toolbar, This plugin should not be the only one leaking :(