Possible memory leak?

[JonnyKong]

While developing ndn-python-repo, we noticed excessive memory use that might be caused by the python-ndn library.

I managed to reproduce the issue based on the example consumer and producer: producer-consumer.tar.gz

Ways to reproduce:

1. Run producer.py.

2. Run consumer.py. The consumer repeatedly sends interests to the producer.

3. Monitor the memory usage of producer.py, which keeps increasing. Judging by the memory usage, the producer seems to save a copy of every data packet in the memory. Even after the consumer leaves, the producer doesn't free the memory.

[zjkmxy]

All memory allocation and free are managed by Python GC. We have no control on when the Data packet will be freed. Also, I don't think I will keep any packet if the callback does not need it. (But if the callbacks keeps the memoryview instead of making a copy, then the whole packet will be alive) To show this is a real memory leak, you can try the following:

1. Let the producer run for a while after the consumer leaves, so we are sure that no one needs these data.
2. Call gc.collect()
3. See how much memory are freed.

[JonnyKong]

• Let the producer run for a while after the consumer leaves, so we are sure that no one needs these data.
• Call gc.collect()
• See how much memory are freed.

I ran the above steps a few times, specifically:

1. Let the producer and consumer run simultaneously. The consumer retrieved ~15000 packets from the producer. At this time, the producer uses around 150 MB of memory.
2. Stop the consumer, let the producer run for a few minutes.
3. Have the producer run gc.collect(). This function returns around 40, meaning the GC found about 40 unreachable objects and collected them. After the call, the memory usage remains at roughly the same level.

[zjkmxy]

OK, I'll check this.

[zjkmxy]

I tried to use objgraph to investigate but could not find any memory leaks in my program. Can you please set a hard memory limit (like using prlimit on Linux) and try again?

[JonnyKong]

It seems that this problem is reproducible on MacOS only. I'm using python3.7 on macOS Catalina 10.15.5 (19F101).

I used objgraph to profile the memory. This is the result on the producer side, after serving 10000 data packets, and calling gc.collect():

>>> import objgraph
>>> objgraph.show_most_common_types(limit=50)
memoryview                 10003
managedbuffer              10001
function                   7321
dict                       4213
tuple                      3283
list                       2754
weakref                    1972
builtin_function_or_method 1488
...

Here, function and dict should not be the cause, because their object count does not increase with time. It seems that the problem comes from some memoryview and managedbuffer not being freed.

They seem to be referenced by some dict objects:

>>> obj = objgraph.by_type('memoryview')[1000]
>>> objgraph.show_backrefs(obj, max_depth=10)

>>> obj = objgraph.by_type('managedbuffer')[1000]
>>> objgraph.show_backrefs(obj, max_depth=10)

[zjkmxy]

We concluded that the memory leak may be a problem with specific Python version and there is nothing we can do for the library.