Possible memory leak when using NDB

[luis5tb]

After some investigation I think I hit some memory leak when using pyroute2.NDB. When executing the next https://github.com/luis5tb/bgp-agent/blob/main/bgp_agent/platform/utils/linux_net.py#L34-L43, on every execution I see that the memory allocated to the process increases: http://paste.openstack.org/show/804952/

The next code was used to estimate that consumption per process after each time the above function got executed: proc = psutil.Process(os.getpid()) LOG.info("MEMORY USAGE PER PROCESS: {}".format(proc.memory_info().rss))

[svinota]

Thanks, investigating

[svinota]

I confirm the RSS increase, tracking the cause.

[luis5tb]

Any news on this? The next can probably be used to reproduce the problem http://paste.openstack.org/show/806232/. I'm going to test if it is just with kind=vrf, or also happens with other kind (like dummy)

[svinota]

@luis5tb sorry, I was totally busy with changing the package layout, and thank you for pinging me about the issue.

Now switching to this task.

[luis5tb]

Thanks! Using this as reproducer now: http://paste.openstack.org/show/806237/. Seems to not be dependent on the kind, it happens with both vrf and dummy types

[svinota]

I found so far that some string objects are leaking, ~15 strings per interface creation. Most probably in the RTNL decoding. Though not tracked the source yet.

It has nothing to do with interfaces per se, I believe it's a parser bug.

And I'm not sure with the strategy, since the parser is going to be rewritten anyways.

[luis5tb]

Note though, with the reproducer the creation of the interface leaks more memory (first iteration), but the second iteration will not create anything, just check the interface is there and already up, and do nothing (at most the set up, as the status will be UNKNOWN). So there is no interface creation in the follow up executions, though the leak is still there

[svinota]

Yep, that's the parser. Same RSS increase with raw IPRoute.get_links().

[svinota]

… Or not. Anyways, debugging now.

[svinota]

Still not perfect, but a bit better.

The code:

import gc
import objgraph
from pyroute2 import NDB

objgraph.show_most_common_types(limit=10)

for _ in range(1000):
    ndb = NDB()
    ndb.close()

gc.collect()

print('8<--------------------------------')
objgraph.show_most_common_types(limit=10)

Before the change:

{(HEAD frånkopplat vid 5aaf9f72)} $ python e789.py 
function                   6620
tuple                      4950
dict                       3776
weakref                    2422
type                       1804
wrapper_descriptor         1403
getset_descriptor          1215
method_descriptor          1195
builtin_function_or_method 1121
cell                       650
8<--------------------------------
list               27400
dict               20848
Logger             13341
function           6624
tuple              4413
weakref            2530
PlaceHolder        1913
type               1804
wrapper_descriptor 1403
getset_descriptor  1217

After the change:

{master} $ python e789.py 
function                   6621
tuple                      4560
dict                       3776
weakref                    2423
type                       1805
wrapper_descriptor         1403
getset_descriptor          1215
method_descriptor          1195
builtin_function_or_method 1121
cell                       650
8<--------------------------------
function                   6625
dict                       4741
tuple                      4413
list                       2794
weakref                    2530
type                       1804
wrapper_descriptor         1403
getset_descriptor          1217
method_descriptor          1195
builtin_function_or_method 1136

[luis5tb]

Nice!! Thanks!

[svinota]

I believe that more changes will follow, even this one might be reworked. Thus not closing the ticket.

[svinota]

RSS growth before https://github.com/svinota/pyroute2/commit/97d7d4f2281a1aef4b1808f23b72716b53a0e9c1, 1000 probes

RSS growth after https://github.com/svinota/pyroute2/commit/97d7d4f2281a1aef4b1808f23b72716b53a0e9c1, 1000 probes

Something is still leaking, I will hunt it down.