PatrickKinnear
commented
2 years ago On generating the random SL_2(Z) matrices (this happens in the function generate_raw_data, when generate mode is invoked), the program does fine for a while but is eventually killed by the OOM (out of memory) killer.
Have tried modularising [turn the subroutine within a for loop into a function which is called] as much as possible so that function cleanup routines manage memory. I noticed one for loop that had not been modularised, which was the loop in get_dim_estimate_empty. This has now been modularised into compute_corank. This has increased the length of time the program runs for but it is still killed by the oom killer.
Tracing memory usage with the tracemalloc module, it seems that the program is always killed at the line where the relations returned by get_relations_empty are turned into a Sage matrix:
A = matrix(QQ['q'].fraction_field(), relations)
It is not clear why this line causes an oom issue. There is no clear link to the number of relations: sometimes the process is killed when there are ~50 relations even though it has previously done computations for ~100 relations. Moreover, at the point when the killing happens, tracemalloc indicates that the current memory usage is not problematically high: it is normally below peak usage and is below usages which have been acceptable before.
This suggests that the problem is not build-up of redundant data in the program, but that the above displayed line of code is itself demanding, in some circumstances, too much memory. It is not clear why.
One option may be the sparse matrix implementation Issue #3 .
Investigate and fix the memory errors the program is experiencing in generating random SL_2(Z) matrices.