Possible memory leak in 1.7.0 (nuget, windows)

[Marluwe]

We notice a memory leak, using the MIP solver in some .net applications (tested on .net7 and .net8).

A simple test program would be to run multiple MIPs with integer variables (e.g. knapsack) one after another. (see the attached cs file HighsTester_LessDependencies.cs.txt). Frankly, the leak only appears when using integer constraints on the variables.

The unmanaged memory keeps rising and analysing it shows some log strings from the MIP solver. We can see that the remaining unmanaged memory contains strings which look like logs of the MIP.

We use the following when working with the solver:

HighsLpSolver solver = new HighsLpSolver();
try
{
    var status = solver.passMip(model);
    status = solver.run();
    if (status == HighsStatus.kError)
    {
        return (null, 0, HighsModelStatus.kUnknown);
    }
    HighsSolution sol = solver.getSolution();
    var info = solver.getInfo();
    HighsModelStatus modelStatus = solver.GetModelStatus();
    return (sol.colvalue, info.ObjectiveValue, modelStatus);
}
finally
{
    solver.clearSolver();
    solver.clearModel();
    solver.Dispose();
}

We haven't been able to test the same in pure C++.

[jajhall]

This is odd. Perhaps you've seen other strings that correspond to MIP output, but they aren't listed here. The strings listed are values assigned as part of the records that are created when the run-time options are set up. It would appear that these records are not cleared when a Highs instance is destroyed

[Marluwe]

Here are some further screenshots for strings in unmanaged memory blocks after all solvers have finished. The program itself requires about 20MB RAM. The test program iterates over 50.000 small knapsack problems (100 variables) and the unmanaged heap blocks 2GB RAM. This test takes just a few seconds. Going up to 1000 variables takes about 5 minutes and increases the leaked memory to 8GB of RAM.

Maybe we are missing some calls to clean up the memory?

Here are the string lists from VMMap of the first three unmanaged memory block of the screenshot above:

[jajhall]

Yes, these are all name and description strings in runtime option records. These are created when the HighsOptions class is instantiated. The destructor for the HighsOptions class calls deleteRecords();,

void deleteRecords() { for (size_t i = 0; i < records.size(); i++) delete records[i]; }

which loops through all the records. Putting a print statement after delete records[i]; sometimes gives the values for the strings that were defined. I don't know how delete acts - I didn't write deleteRecords();, - and it's beyond my C++ knowledge.

Naively, given the declaration std::vector<OptionRecord*> records;, I'm surprised that there's no records.clear(). I don't see what harm it could do, so I can add it, but it would be good if someone who understands C++ better than me were to give an explanation and fix for what's going on.

[jajhall]

Running valgrind (without records.clear();)gives

==35818== ==35818== HEAP SUMMARY: ==35818== in use at exit: 5,252,312 bytes in 52 blocks ==35818== total heap usage: 6,419 allocs, 6,367 frees, 10,063,469 bytes allocated ==35818== ==35818== LEAK SUMMARY: ==35818== definitely lost: 0 bytes in 0 blocks ==35818== indirectly lost: 0 bytes in 0 blocks ==35818== possibly lost: 5,251,784 bytes in 33 blocks ==35818== still reachable: 528 bytes in 19 blocks ==35818== suppressed: 0 bytes in 0 blocks ==35818== Rerun with --leak-check=full to see details of leaked memory ==35818== ==35818== For lists of detected and suppressed errors, rerun with: -s ==35818== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

When running

valgrind --leak-check=full bin/highs

the leaks correspond to the parallel event scheduler, and I remember a discussion with @gottwald to the effect that they may not be real memory leaks and "don't matter"

If I add the call records.clear();. then valgrind still gives

==35818== still reachable: 528 bytes in 19 blocks

which makes sense if the "memory leaks" on Linux are due to the parallel event scheduler. So, could the issue only relate to Windows?

[Marluwe]

We tested our knapsack test application (.Net8) on Ubuntu in wsl.

Running the test application with 500 knapsack problems with 10_000 variables each, the application (console) keeps about 3 GB RAM after all solvers are done, cleared and disposed.

Tracking with valgrind takes ages, so I used a smaller data set with 100 problems and 1000 variables each.

==67673== LEAK SUMMARY: ==67673== definitely lost: 66,472 bytes in 180 blocks ==67673== indirectly lost: 0 bytes in 0 blocks ==67673== possibly lost: 19,682 bytes in 58 blocks ==67673== still reachable: 10,602,507 bytes in 5,864 blocks ==67673== suppressed: 0 bytes in 0 blocks ==67673== Reachable blocks (those to which a pointer was found) are not shown. ==67673== ERROR SUMMARY: 3398 errors from 31 contexts (suppressed: 0 from 0)

The same with only one problem to solve with just one variable gives the following

==70616== HEAP SUMMARY: ==70616== in use at exit: 15,443,039 bytes in 5,349 blocks ==70616== total heap usage: 32,346 allocs, 26,997 frees, 26,152,403 bytes allocated ==70616== ==70616== LEAK SUMMARY: ==70616== definitely lost: 37,276 bytes in 90 blocks ==70616== indirectly lost: 0 bytes in 0 blocks ==70616== possibly lost: 5,275,316 bytes in 105 blocks ==70616== still reachable: 10,130,447 bytes in 5,154 blocks ==70616== suppressed: 0 bytes in 0 blocks

I looks like .net8 and valgrind might not work together that well, at least running in wsl.

@galabovaa Can we provide additional information that might help?

[jajhall]

Are these results with https://github.com/ERGO-Code/HiGHS/tree/latest ?

[Marluwe]

The tests use the official nuget package version 1.7.0.

[jajhall]

Sorry, yes, of course.

Since I've found an anomaly relating to the records for run-time options, it's worth seeing whether my fix - that @galabovaa (or someone with better C++ skills than me) will need a view on - clears the issue. However, this will require a new nuget package to be created. This is something that only @galabovaa can do, and she's on holiday until the end of next week.

[Marluwe]

I checked our test application on a vm with ubuntu (64 bit), to see if it is a windows related issue.

Running 1000 problems with 1000 variables (weight 0.4, see our example file) starts with 70MB before the first solver is initialized and it ends with 677 MB RAM remaining after all solvers have been disposed. So, the issue seems to exist on Linux systems as well. It looks like increasing the knapsack complexity also increases the amount of leaking memory.

It still might be an issue related to the mix of .net and the native highs library.

I took a memory dump and checked for remaining strings after the application disposed all solvers and ran a final aggressive garbage collection (see below).

gcore -o /tmp/core_dump PID

Here are some lines from the command

strings /tmp/core_dump.PID | grep "MIP"

the MIP pent for MIP heuristics simplex", "choose", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored the MIP g MIP solutions should be reported in sparse format ows in the MIP solver cutpool before they are deleted f observations before MIP solver pseudo costs are considered reliable ber of improving solutions found to stop the MIP solver prematurely it on the number of rows in the MIP solver cutpool for dynamic age adjustment f observations before MIP solver pseudo costs are considered rel ng improving MIP solutions: not reported for an empty string "" f entries in the MIP solver cliquetable before neighbourhood queries of the conflict graph use parallel processi tion of the MIP solv MIP MIP solver cliquetable before neighbourhood queries of the conflict graph use parallel processi er when solving LPs, but not subproblems in the MIP solver ative gap, |ub-lb|/|ub|, to determine whether optimality has been reached for a MIP instance simplex", "choose", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored e number of rows in the MIP solver cutpool for dynamic age adjustment the MIP for termination of the MIP solv the MIP simplex", "choose", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored ynamic LP rows before they are removed from the LP relaxation in the MIP solver the MIP MIP solutions should b! MIP in olute gap of MIP, |ub-lb|, to determine whether optimality has been reached for a MIP instance the MIP the MIP f entries in the MIP solver cliquetable before neighbourhood queries of the conflict graph use parallel processi the MIP the MIP b|/|ub|, to determine whether optimality has been reached for a MIP instance lve reductions on restart in MIP solver -1 => no limit, otherwise, must be positive t in MIP ber of improving solutions found to stop the MIP solver prematurely g MIP solutions should be saved the number of improving solutions found to stop the MIP solver prematur Minimal number of entries in the MIP solver cliquetable before neighbourhood queries of the conflict graph use parallel processi b|/|ub|, to determine whether optimality has been reached for a MIP instance e on absolute gap of MIP, |ub-lb|, to determine whether optimality has been reached for a MIP instance number of further presolve reductions on restart in MIP solver -1 => no limit, otherwise, must be positive simplex", "choose", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored e target for termination of the MIP solver improving MIP solutions should be saved improving MIP solutions should b d for a MIP instQ een reached for a MIP in* the MIP f observations before MIP solver pseudo costs are considered reliable for simplex solver when solving LPs, but not subproblems in the MIP solver ng MIP MIP symmetry should be d MIP restart is permitted ng improving MIP solutions: not reported for an empty string "" reporting improving MIP solutions: not reported for an empty string "" e on relative gap, |ub-lb|/|ub|, to determine whether optimality has been reached for a MIP instance ng improving MIP solutions: not reported for an empty string "" ber of improving solutions found to stop the MIP solver prematur the MIP( the MIP e number of rows in the MIP solver cutpool for dynamic age adjusP b|/|ub|, to determine whether optimality has been reached for a MIP instance e on absolute gap of MIP, |ub-lb|, to determine whether optimality has been reached for a MIP instance the number of improving solutions found to stop the MIP solver prematur@ simplex", "choose", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored improving MIP solutions should be saved improving MIP solutions should be reported in sparse format the MIP e on relative gap, |ub-lb|/|ub|, to determine whether optimality has been reached for a MIP instance simplex", "choose", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored lve reductions on restart in MIP solver -1 => no limit, otherwise, must be positive for simplex solver when solving LPs, but not subproblems in the MIP solver the MIP n in the MIP1 improving MIP solutions should be saved improving MIP solutions should be reported in sparse for of further presolve reductions on restart in MIP solver -1 => no limit, otherwise, must be positive MIP so! pent for MIP heuristics e on absolute gap of MIP, |ub-lb|, to determine whether optimality has been reached for a MIP instance f entries in the MIP solver cliquetable before neighbourhood queries of the conflict graph use parallel processi simplex", "choose", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored e", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored pent for MIP heuristics ynamic LP rows before they are removed from the LP relaxation in the MIP solver age of rows in the MIP solver cutpool before they are de number of observations before MIP solver pseudo costs are considered reliable MIP restart is permitted improving MIP solutions should be reported in sparse format MIP symmetry should be detected MIP restart is permitted for simplex solver when solving LPs, but not subproblems in the MIP solver ving MIP MIP solver cliquetable before neighbourhood queries of the conflict graph use parallel processi simplex", "choose", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored for termination of the MIP solv@ the MIP improving MIP solutions should be saved pent for MIP heuristics number of observations before MIP solver pseudo costs are considered rel6 age of rows in the MIP solver cutpool before they are de W g MIP solutions should be reported in sparse forP not subproblems in the MIP solv for termination of the MIP solv e on relative gap, |ub-lb|/|ub|, to determine whether optimality has been reached for a MIP instance e on absolute gap of MIP, |ub-lb|, to determine whether optimality has been reached for a MIP instance the MIP s chosen then, for a MIP (QP) the integr age of rows in the MIP solver cutpool before they are de simplex", "choose", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored it on the number of rows in the MIP solver cutpool for dynamic age adjustment ber of improving solutions found to stop the MIP solver prematurely age of rows in the MIP solver cutpool before they are depE e number of rows in the MIP solver cutpool for dynamic age adjus MIP solver cliquetable before neighbour f entries in the MIP solver cliquetable before neighbourhood queries of the conflict graph use parallel processi rom the LP relaxation in the MIP! the MIP e on absolute gap of MIP, |ub-lb|, to determine whether optimality has been reached for a MIP instance g MIP solutions should be reported in sparse format the MIP age of dynamic LP rows before they are removed from the LP relaxation in the MIP solver e number of rows in the MIP solver cutpool for dynamic age adjus the MIP ng improving MIP solutions: not reported for an empty string "" the MIP the MIP e number of rows in the MIP solver cutpool for dynamic age adjustment ows in the MIP solver cutpool before they are de0 reporting improving MIP solutions: not reported for an empty string "" it on the number of rows in the MIP solver cutpool for dynamic age adjus the MIP the MIP ynamic LP rows before they are removed from the LP relaxation in the MIP solver the MIPC improving MIP solutions should be saved ber of improving solutions found to stop the MIP solver prematurZ g MIP solutions should be reported in sparse forP MIP MIP symmetry should be detected the MIP ative gap, |ub-lb|/|ub|, to determine whether optimality has been reached for a MIP instance olute gap of MIP, |ub-lb|, to determine whether optimality has been reached for a MIP instance the MIP the MIP age of rows in the MIP solver cutpool before they are deleted simplex", "choose", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored simplex", "choose", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored for termination of the MIP solver f entries in the MIP solver cliquetable before neighbourhood queries of the conflict graph use parallel processi for simplex solver when solving LPs, but not subproblems in the MIP solver the MIP MIP heuristics pent for MIP heuristics the MIP the MIP of further presolve reductions on restart in MIP solver -1 => no limit, otherwise, must be positive number of observations before MIP solver pseudo costs are considered rel MIP solver cliquetable before neighbourhood queries of the conflict graph use parallel processi the MIP lve reductions on restart in MIP solver -1 => no limit, otherwise, must be positive the MIP the MIPT the MIPU for simplex solver when solving LPs, but not subproblems in the MIP solver er when solving LPs, but not subproblems in the MIP solver Minimal number of entries in the MIP solver cliquetable before neighbourhood queries of the conflict graph use parallel processi age of rows in the MIP solver cutpool before they are dep ative gap, |ub-lb|/|ub|, to determine whether optimality has been reached for a MIP instance olute gap of MIP, |ub-lb|, to determine whether optimality has been reached for a MIP instance MIP soi improving MIP solutions should be reported in sparse format for termination of the MIP solver the number of improving solutions found to stop the MIP solver prematurely age of rows in the MIP solver cutpool before they are deleted MIP restart is permitted er when solving LPs, but not subproblems in the MIP solver oving MIP soA the MIP the MIP g MIP solutions should be saved ynamic LP rows before they are removed from the LP relaxation in the MIP solver MIP symmetry should be d improving MIP solutions should be saved ng improving MIP solutions: not reported for an empty string "" number of further presolve reductions on restart in MIP solver -1 => no limit, otherwise, must be positive f entries in the MIP solver cliquetable before neighbourhood queries of the conflict graph use parallel processi pent for MIP heuristics simplex", "choose", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored e on relative gap, |ub-lb|/|ub|, to determine whether optimality has been reached for a MIP instance reporting improving MIP solutions: not reported for an empty string "" g MIP solutions should be saved e on relative gap, |ub-lb|/|ub|, to determine whether optimality has been reached for a MIP instance efore they are removed from the LP relaxation in the MIP solver ving MIP0 for simplex solver when solving LPs, but not subproblems in the MIP solver number of observations before MIP solver pseudo costs are considered rel number of observations before MIP solver pseudo costs are considered rel ynamic LP rows before they are removed from the LP relaxation in the MIP solver it on the number of rows in the MIP solver cutpool for dynamic age adjus number of observations before MIP solver pseudo costs are considered rel Minimal number of entries in the MIP solver cliquetable before neighbourhood queries of the conflict graph use parallel processi it on the number of rows in the MIP solver cutpool for dynamic age adjus MIP symmetry should be d MIP restart is permitted improving MIP solutions should be reported in sparse for MIP restart is permittedp e target for termination of the MIP solv of further presolve reductions on restart in MIP solver -1 => no limit, otherwise, must be positive improving MIP solutions should be saved MIP symmetry should be d the MIPl the MIPm age of dynamic LP rows before they are removed from the LP relaxation in the MIP solver , |ub-lb|, to determine whether optimality has been reached for a MIP instance e on absolute gap of MIP, |ub-lb|, to determine whether optimality has been reached for a MIP instance pent for MIP heuristics simplex", "choose", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored MIP e", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored number of further presolve reductions on restart in MIP solver -1 => no limit, otherwise, must be positive MIP symmetry should be d MIP restart is permitted improving MIP solutions should be saved of further presolve reductions on restart in MIP solver -1 => no limit, otherwise, must be positive Solver option: "simplex", "choose", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored ber of improving solutions found to stop the MIP solver prematur the MIP the MIP , |ub-lb|, to determine whether optimality has been reached for a MIP instance simplex", "choose", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored e", "ipm" or "pdlp". If "simplex"/"ipm"/"pdlp" is chosen then, for a MIP (QP) the integrality constraint (quadratic term) will be ignored MIP symmetry sho number of further presolve reductions on restart in MIP solver -1 => no limit, otherwise, must be positive ...

[jajhall]

Yes, as before, all these are from the records for run-time options, that may now be cleared as a consequence of my fix yesterday, but we can't confirm until a new NuGet package is created. This can't happen until @galabovaa is back

Thanks for your patience

[galabovaa]

We have now created a new release on NuGet, containing all of our latest fixes. @Marluwe, would you please give it a go, and let us know if you are still getting memory issues?

[Marluwe]

The new version doesn't show any leaks in our tests. Thank you for the fix!