Problems when calling external programs

[tueda]

This is probably an OS-related issue rather than a FORM bug, and maybe we cannot make any improvements, but still I would like to put a note about it. I am using a Desktop PC with Scientific Linux 6.6, x86_64, 7804MB Mem + 7967MB Swap. The problem I hit is, as a simple example,

#:MaxTermSize 2M
#:WorkSpace 2G
#system echo OK
.end

fails at use of #system:

foo.frm Line 4 ==> System call returned with error condition

If I remove the settings of MaxTermSize and WorkSpace, the program works well. A similar situation was already described in this post in the forum. But now the origin of the problem looks like: if a program allocates big memory blocks, then the system call for system() fails.

Note that the above program consumes only a small memory space (as you can see, for example, by top). GNU time utility reports the maximum resident set size is 1720KB. But actually the program allocates a very large memory space. valgrind reports 14GB memory blocks are still reachable (bigger than Mem+Swap).

[vermaseren]

Hi Takahiro,

When I run this program on my laptop (OS X), on drome and zaira (scientific linux) in all cases it runs fine. Question: are you running in 32-bit or in 64-bit mode?

Jos

On 25 jun. 2015, at 19:17, Takahiro Ueda notifications@github.com wrote:

:MaxTermSize 2M

:WorkSpace 2G

system echo OK

.end

[tueda]

@vermaseren It was in 64-bit mode. On drome, try

#:MaxTermSize 160M
#:WorkSpace   160G

(Actually I first hit it on drome, with very large polynomial manipulations.)

[vermaseren]

OK, I tried it on my OSX laptop with 16 Gbytes and with 4M 4G it crashes as well. I checked that this is the only very large memory allocation. All others are still ‘normal’. At least according to “on setup;”. Next I turned on MALLOCDEBUG in tools.c This shows what is happening. The culprit is not the workspace but maxtermsize. I still have to figure out how it comes to a malloc of 12 Gbytes for its sort buffer with maxtermsize at 4M. Problem is that it needs many times maxtermsize for all kinds of buffers, like each of the patches in the large size should be at least a few times maxtermsize, etc. You could try to look at it yourself, and see whether you see anything ‘suspect’. But making maxtermsize extremely large is not what you are supposed to do. More than 1M is already gigantic.

Cheers

Jos

On 25 jun. 2015, at 20:27, Takahiro Ueda notifications@github.com wrote:

@vermaseren https://github.com/vermaseren It was in 64-bit mode. On drome, try

:MaxTermSize 160M

:WorkSpace 160G

(Actually I first hit it on drome, with very large polynomial manipulations.)

— Reply to this email directly or view it on GitHub https://github.com/vermaseren/form/issues/36#issuecomment-115354949.

[vermaseren]

Hi Takahiro,

If you go to AllocSort in setfile.c there is the statement:

        LargeSize  = MaxFpatches*
         ((IObuffersize+COMPINC)*sizeof(WORD)+2*AM.Maxter)
            - SmallEsize;

When there are 256 patches in the large buffer, each at least two times AM.MaxTer this can run out of hand. Things are even worse in tform, because the merging of the threads needs even more times MaxTer. It would be very nice to have a version of FORM without this limitation, but I never saw a decent solution that would preserve speed.

Result: you cannot create arbitrariy large rational polynomials.

Cheers

Jos

On 25 jun. 2015, at 20:27, Takahiro Ueda notifications@github.com wrote:

@vermaseren https://github.com/vermaseren It was in 64-bit mode. On drome, try

:MaxTermSize 160M

:WorkSpace 160G

(Actually I first hit it on drome, with very large polynomial manipulations.)

— Reply to this email directly or view it on GitHub https://github.com/vermaseren/form/issues/36#issuecomment-115354949.

[tueda]

Yes, actually relations among buffer sizes are very complicated to users. Ben told me that increasing MaxTermSize is not enough when FORM suggests "Increase MaxTermSize" and other buffer sizes are needed to be increased. At least, it would be nice if FORM could warn (or warn for "allwarnings") when some parameters are overwritten because of some minimum conditions.

[vermaseren]

Hi Takahiro,

If it would have been trivial, it would have been done of course. Maybe, after some thinking, it can become trivial, but for now I do not see it. For instance: how much workspace you need, depends on the complexity of your modules. Example: L F = f(1000000); repeat id f(x?pos_) = f(x-1); would probably crash on nearly any system. And it could be because it reaches the WorkSpace limits, or because of stack overflow or what else. Maybe we can modify Malloc1/M_free such that it keeps track of the total amount of allocated memory (like putting the size of the buffer in the first 8 bytes and returning a shifted address (Malloc1) and reshift when freeing. That way the program can warn when the total allocations reach a given limit. But that still does not solve the dependencies between buffer sizes. There is something about it in the manual though. Very few people really read that part…. Oh well. The main reason for the manual may well be that I can say: “it is in the manual” :-)

It seems that my code is working more or less. Maybe we can discuss tomorrow how to incorporate it in the main version and get Bens improvements as well.

Cheers

Jos

On 25 jun. 2015, at 21:16, Takahiro Ueda notifications@github.com wrote:

Yes, actually relations among buffer sizes are very complicated to users. Ben told me that increasing MaxTermSize is not enough when FORM suggests "Increase MaxTermSize" and other buffer sizes are needed to be increased. At least, it would be nice if FORM could warn (or warn for "allwarnings") when some parameters are overwritten because of some minimum conditions.

— Reply to this email directly or view it on GitHub https://github.com/vermaseren/form/issues/36#issuecomment-115367922.