How to release RAM

[Pokemer]

When I test an unconstrained optimization algorithm ，I found that reuse pycutest.import_problem will lead the RAM increase fastly.

import pycutest

uncon = pycutest.find_problems(constraints='unconstrained', userN=True)
for prob in uncon:
    p = pycutest.import_problem(prob)

Only exit Python can release RAM

[jfowkes]

This is a good question: Python should automatically free memory for objects that are no longer in scope. Your observation suggests that there may be a memory leak in the C interface (I have been working on modernising it in #33).

I will try and run your example through a Python debug build and see if it is able to detect any memory leaks.

[Pokemer]

Thank you

[jfowkes]

I can confirm that this behaviour is due to memory leaks, valgrind detects multiple memory leaks in CUTEst itself:

839,988 bytes in 1 blocks are still reachable in loss record 3,062 of 3,064
    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
    by 0x473D9CD7: __cutest_MOD_cutest_allocate_array_integer (in /home/jfowkes/pycutest_cache_holder/MODBEALE/_pycutestitf.cpython-310-x86_64-linux-gnu.so)
    by 0x473DA009: __cutest_MOD_cutest_sparse_hessian_by_rows (in /home/jfowkes/pycutest_cache_holder/MODBEALE/_pycutestitf.cpython-310-x86_64-linux-gnu.so)
    by 0x473DEAB5: __cutest_MOD_cutest_size_sparse_hessian (in /home/jfowkes/pycutest_cache_holder/MODBEALE/_pycutestitf.cpython-310-x86_64-linux-gnu.so)
    by 0x473B5717: cutest_udimsh_threadsafe_ (in /home/jfowkes/pycutest_cache_holder/MODBEALE/_pycutestitf.cpython-310-x86_64-linux-gnu.so)
    by 0x473B5763: cutest_udimsh_ (in /home/jfowkes/pycutest_cache_holder/MODBEALE/_pycutestitf.cpython-310-x86_64-linux-gnu.so)
    by 0x473ACD94: cutest__setup (cutestitf.c:460)

959,976 bytes in 1 blocks are still reachable in loss record 3,063 of 3,064
    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
    by 0x473B1441: cutest_usetup_threadsafe_ (in /home/jfowkes/pycutest_cache_holder/MODBEALE/_pycutestitf.cpython-310-x86_64-linux-gnu.so)
    by 0x473B4DF6: cutest_usetup_ (in /home/jfowkes/pycutest_cache_holder/MODBEALE/_pycutestitf.cpython-310-x86_64-linux-gnu.so)
    by 0x473ACD09: cutest__setup (cutestitf.c:425)

2,399,992 bytes in 1 blocks are still reachable in loss record 3,064 of 3,064
    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
    by 0x473E4850: __cutest_MOD_cutest_initialize_workspace (in /home/jfowkes/pycutest_cache_holder/MODBEALE/_pycutestitf.cpython-310-x86_64-linux-gnu.so)
    by 0x473B3608: cutest_usetup_threadsafe_ (in /home/jfowkes/pycutest_cache_holder/MODBEALE/_pycutestitf.cpython-310-x86_64-linux-gnu.so)
    by 0x473B4DF6: cutest_usetup_ (in /home/jfowkes/pycutest_cache_holder/MODBEALE/_pycutestitf.cpython-310-x86_64-linux-gnu.so)
    by 0x473ACD09: cutest__setup (cutestitf.c:425)

and many more such leaks...

I will open an issue upstream and hopefully this will get fixed in the next CUTEst release.

[Pokemer]

Thank you. Python is very efficient in the development of unconstrained optimization algorithms, and pycutest provides a large number of test sets. Thank you very much for developing this interface.

[jfowkes]

Upstream have pointed out that our C interface is not calling the requisite terminate functions:

• CUTEST_uterminate for unconstrained problems
• CUTEST_cterminate for constrained problems

that are required on exit to clean up all the allocated memory! We will have to fix this at our end asap.