search

vermaseren / form

The FORM project for symbolic manipulation of very big expressions
GNU General Public License v3.0
982 stars 118 forks source link

Changing maximum RAM #371

Closed MA0R closed 1 year ago

MA0R commented 3 years ago

Hey! Is it possible to change the maximum amount of RAM allocated to a computation? It seems that there is a cap at 2G (64bit machine), which I found in the documentation here, but wasn't so sure how to change it.

vermaseren commented 3 years ago

There is no cap currently beyond what the computer is willing/allowed to give you. In 32-bit machines there was this cap, but currently we only develop for 64-bit machines. Precise details are in the chapter on the setup in the manual. For instance: LargeSize 10G allocates 10 Gigabytes for the large buffer. Etc.

On most batch systems to have to specify how much RAM you are going to use and if you go beyond that, the system will not allow you and the program will crash. Similarly, on your laptop, if you do not have swap space, allocations beyond what the computer has are not possible. Hence, if you have a 64-bit laptop with 2Gbytes of RAM and no swapspace, you will be limited to 2 Gbytes.

I hope this answers your question.

On 11 Dec 2020, at 15:39, Maor Ben-Shahar notifications@github.com wrote:

Hey! Is it possible to change the maximum amount of RAM allocated to a computation? It seems that there is a cap at 2G (64bit machine), which I found in the documentation here https://www.nikhef.nl/~form/maindir/documentation/reference/online/online.html#469, but wasn't so sure how to change it.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/vermaseren/form/issues/371, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABJPCERTRDCPI74FWHCN7GLSUIVL5ANCNFSM4UWXEGLQ.

tueda commented 3 years ago

For information, the cap of 2G on 64-bit machines in the section of the manual that @MA0R mentioned is: FORM allows one to use only 2,000,000,000 identifiers in a program.

MA0R commented 3 years ago

Thanks for the clarification @tueda, I misinterpreted this section.

I will clarify how I came to the thought there is a cap: I have 32G of ram on my computer, but my form program never uses more than 2G while running, and it takes a really long time. When I terminate it, it says "Max. space for expressions 8G" (or more). So I thought some large expressions are being expanded and saved somewhere along the way, since mathematica tends to do that when reaching max allocated ram.

I tried playing around with the setup as you were saying @vermaseren, and indeed my choices there seem to affect how much physical RAM the program uses, but it is smaller than the max size the expressions required along the way.

Maybe this is not something to be worried about?

vermaseren commented 3 years ago

When expressions use more space than RAM, they go to disk. Hence the size of your disk is the limitation on the size of expressions. Because Form knows what it has where and when to use it, this usage of the disk hardly affects performance. It is of course most efficient when expressions can stay in memory entirely, but as mentioned, it is not very much more efficient. Hence I would not worry. @tueda has a little program that can determine a reasonable setting for your computer, making it use most of its RAM. Alternatively you have to experiment a little bit to see what works best for your calculations. For that you have to understand this chapter on the setup rather well though.

On 11 Dec 2020, at 18:31, Maor Ben-Shahar notifications@github.com wrote:

Thanks for the clarification @tueda https://github.com/tueda, I misinterpreted this section.

I will clarify how I came to the thought there is a cap: I have 32G of ram on my computer, but my form program never uses more than 2G while running, and it takes a really long time. When I terminate it, it says "Max. space for expressions 8G" (or more). So I thought some large expressions are being expanded and saved somewhere along the way, since mathematica tends to do that when reaching max allocated ram.

I tried playing around with the setup as you were saying @vermaseren https://github.com/vermaseren, and indeed my choices there seem to affect how much physical RAM the program uses, but it is smaller than the max size the expressions required along the way.

Maybe this is not something to be worried about?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/vermaseren/form/issues/371#issuecomment-743326250, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABJPCERHHLHFKU45JGQIHZLSUJJO3ANCNFSM4UWXEGLQ.

tueda commented 3 years ago

My program to adjust sort buffer sizes etc. is here: https://github.com/tueda/formset. Still, documentation is missing and I should write it, but usage and discussion can be found in https://github.com/vermaseren/form/issues/369.

MA0R commented 3 years ago

Thank you for your support and suggestions! After playing around with the set-up and also using formset.py I think that I determined that my complete calculation it too time consuming for my desktop computer (and my patience). I still managed to do some pieces using FORM so I am quite happy with it, the rest is probably for the cluster.

Thanks again!