As the meliae library keeps giving us trouble, we can switch over to the built-in gc library for object leak inspection. The following snippet will already do the dirty work:

import gc, sys
obj_dump = gc.get_objects()
out = {}
for obj in obj_dump:
    out[type(obj)] = out.get(type(obj), 0) + 1
print >> sys.stderr, out

This will produce something like:

We can also use gc for checking where objects (including built-ins) were created:

import gc

a = list()
b = ""
c = {'a': 2}

def get_referring_files(obj):
    refs = gc.get_referrers(a)
    files = set()
    for ref in refs:
        if isinstance(ref, dict):
            for v in ref.values():
                if isinstance(v, basestring) and v != '':
                    files |= {v}
    return files

def collect_referring_files(*objs):
    return reduce(lambda a, b: (get_referring_files(b) | a), objs, set())

print collect_referring_files(a, b, c)

$ python test.py 
set(['test.py', '__main__'])

Related to #3598

Some observations:

• The gc.get_objects() does not report all of the objects in memory. You need to recurse over their references to find the remaining objects/structs.
• This is slow.
• We could still just parse the top-level objects to get a quick impression of the application state.
• We need some way to detect a memory leak automatically. I don't know of a good way to detect this. We should be able to detect slow increases as well as fast increases in dict/list size.

Since I don't see a good solution, for now we will have to accept that meliae randomly segfaults some times I guess.

I noticed that meliae does not work/is not available in the repos so I guess we have to move to another memory dumper soon.

I stumbled upon this: https://gist.github.com/mgeeky/6a8fa7814efb6c8ad783c3c76c791c4c . Whether or not this is fast or works at all, I do not know. Generally though, I think a dump-first-analyze-later approach would be good.

As an option: https://pythonhosted.org/Pympler/index.html

Example of diff:

                                                      types |   # objects |   total size
=========================================================== | =========== | ============
                                                       list |          16 |     84.18 KB
                                                        str |         182 |     16.19 KB
                                     traceback.FrameSummary |         141 |      9.91 KB
                               function (trackback_wrapper) |          55 |      7.30 KB
                                     traceback.StackSummary |          55 |      4.71 KB
                                                      tuple |          82 |      4.46 KB
                                                       cell |         110 |      4.30 KB
  tribler_gui.tribler_request_manager.TriblerNetworkRequest |          31 |      4.12 KB
                                                     method |          52 |      3.25 KB
                                    PyQt5.QtWidgets.QAction |          17 |      2.26 KB
                                                       code |           3 |      1.87 KB
                                                      float |          56 |      1.31 KB
                                                        int |          45 |      1.23 KB
                                    PyQt5.QtWidgets.QWidget |           6 |    816     B
                                     PyQt5.QtWidgets.QLabel |           6 |    816     B

related to #5203