Memory Leak in ExtractConnectedComponents

xiangtaoxu commented 2 years ago

Hi, I think there is a memory leak problem in the cc.ExtractConnectedComponents function.

I attach a simple script to illustrate the problem. Basically, the script read a point cloud (roughly 52 MB) and called the ExtractConnectedComponents function 5 times. After each call, roughly 45MB is added to the memory. This is becoming an issue for our codes which calls ExtractConnectedComponents hundreds to thousands of times for some big dataset.

(CloudComPy39) xiangtao@Xu-SLS:~/projects/TLS$ python memory_test.py
QSocketNotifier: Can only be used with threads started with QThread
/home/paul/projets/CloudComPy/CloudComPy/CloudCompare/libs/qCC_db/src/ccLog.cpp [53] : trace OFF
JsonRPCPlugin::JsonRPCPlugin
input cloud memory usage: 4636672
iteration 0, f added memory:  0
iteration 0, ExtractConnectedComponents added memory:  7491584
iteration 1, f added memory:  0
iteration 1, ExtractConnectedComponents added memory:  2097152
iteration 2, f added memory:  0
iteration 2, ExtractConnectedComponents added memory:  2097152
iteration 3, f added memory:  0
iteration 3, ExtractConnectedComponents added memory:  2097152
iteration 4, f added memory:  0
iteration 4, ExtractConnectedComponents added memory:  2097152

memory_test.zip

I know memory bugs are usually very tricky... So hopefully you can help to look into this. Thanks in advance!

prascle commented 2 years ago

Hi, With CloudComPy, the Python garbage collector is not connected to CloudCompare C++ objects. You should explicitly delete the CloudCompare entities you no longer need with cc.deleteEntity(entity).

import sys, os, gc, psutil
# initialize CCP
os.environ["_CCTRACE_"]="OFF" # this line has to be called before import cloudComPy

# load CloudCompy
import cloudComPy as cc

def f(cloud):
    # do nothing, return a tuple
    return (0,[cloud])

cc.initCC()

process = psutil.Process(os.getpid())
mem_start = process.memory_full_info().rss
cloud = cc.loadPointCloud('./memory_test.pcd')
print('input cloud memory usage:', process.memory_full_info().rss - mem_start)
for i in range(100):
    mem_start = process.memory_full_info().rss
    res = f(cloud)
    print(f'iteration {i}, f added memory: ',process.memory_full_info().rss - mem_start)

    mem_start = process.memory_full_info().rss
    res = cc.ExtractConnectedComponents(
            clouds=[cloud],
            octreeLevel=10,
            minComponentSize=1000,
            maxNumberComponents=100)
    # -----------------------------------------------------------
    # do something with the components,
    # then delete the components when you don't need them any more.
    components = res[1]
    for comp in components:
        cc.deleteEntity(comp)
    # be sure to have no more Python variable referencing the deleted items
    components = None
    res = None
    # -----------------------------------------------------------

    print(f'iteration {i}, ExtractConnectedComponents added memory: ',process.memory_full_info().rss - mem_start)

After 3 iterations, memory is stable. Regards,

Paul

xiangtaoxu commented 2 years ago

That makes perfect sense. Thanks so much!

CloudCompare / CloudComPy

Memory Leak in ExtractConnectedComponents #62