MakeInteger: No memory while allocating NumberMalloc

[msloechner]

Dear all,

Sadly in several instances I keep running into a bug when calling MakeInteger on a large number of terms, at some point getting back the message No memory while allocating NumberMalloc. From my idea of the inner workings of FORM MakeInteger should be basically cost-free in terms of memory, which makes me quite curious about this issue.

I suspect there is some memory leak in *MakeInteger in src/argument.c. It happens whenever I have factors of the function g and reach a certain number of terms with g that makeinteger acts upon.

With the standard FORM buffer sizes, I can replicate the error with the following simple FORM code. The number of terms here is set to the minimum for which the programme aborts using FORM 4.3.0:

cf f g;
s N;
on highfirst;
l F = 
#do i=1,524284
+ f(`i')*g(1+N)*g(N-1)
#enddo
;
makeinteger g;
.end

I would be very thankful for some insight.

Cheers, Markus

[jodavies]

This commented NumberFree causes the problem: https://github.com/vermaseren/form/blob/8923c77090fcc41e6c5fba0c29564a1c1138f510/sources/argument.c#L442-L453

[msloechner]

Hi Josh, As far as I see, GCDbuffer is still needed for r3, but later freed in https://github.com/vermaseren/form/blob/8923c77090fcc41e6c5fba0c29564a1c1138f510/sources/argument.c#L1740 So I'm not sure this is the issue. Cheers, Markus

Edit: this was actually changed to the current status in https://github.com/vermaseren/form/commit/4cc67db1bfa780ab5dc383c7122382cf921e8879

[jodavies]

Right, but the NumberAlloc is called twice as many times as the NumberFree in your case. When the space for numbers is exhausted FORM extends the buffer, and eventually it asks for more than your OS will allocate, hence the crash.

[jodavies]

It would be good to have a test which reproduces whatever prompted https://github.com/vermaseren/form/commit/4cc67db1bfa780ab5dc383c7122382cf921e8879 , but I haven't found anything relevant in the github issues.

[tueda]

For reference, here are benchmark results with FORM 4.1 and 4.2 for Markus's example on my desktop PC:

FORM 4.1 (Oct 25 2013) 64-bits	Mean (SD), n = 10
Elapsed (wall clock) time (seconds)	1.358 (0.069)
Maximum resident set size (kbytes)	296350 (2136)

FORM 4.2.0 (Jul 6 2017, v4.2.0) 64-bits	Mean (SD), n = 10
Elapsed (wall clock) time (seconds)	6.755 (1.258)
Maximum resident set size (kbytes)	15482278 (19629)

Clearly, the results show a performance regression.

[jodavies]

The test runs a bit faster and is more suitable for the CI tests if you have fewer terms, but leak more per-term:

Off statistics;
cf f g;
s N;
on highfirst;
l F =
#do i=1,11000
+ f(`i')*(g(1+N)*g(N-1))^25
#enddo
;
.sort
makeinteger g;
.end

Edit: @tueda , if I trim this test down so that it runs on my laptop (i runs to 5000), and make sure the different form versions are using the same setup parameters, I see no performance difference between 4.1, 4.2.0, 4.2.1, 4.3.0, 4.3.1, master. But with the fix it seems about 40% faster.

Another edit: I need to amend that statement, that was for vorm, not form. For 20 runs: 4.1: 10.5s 4.2.0: 22s fix: 12.6s

[msloechner]

@jodavies @tueda Thanks a lot for the quick fix!