no error when trying to allocate too much memory with -memsize

[karlmsmith]

Reported by @AndrewWittenberg on 28 Aug 2006 16:43 UTC Finally entering this into Bugzilla. (Hope I'm not duplicating another bug, but a query didn't turn up anything...)

Date: Fri, 19 Sep 2003 12:09:20 -0400 (EDT) From: Andrew Wittenberg To: Kevin O'Brien Subject: allocating huge memory

Hi Kevin,

This is strange: if I try to allocate an impossibly huge amount of memory, Ferret doesn't issue an error...

ferret_v553beta5 -memsize 400 NOAA/PMEL TMAP FERRET v5.53 (beta5) IRIX 6.5 - 09/10/03 19-Sep-03 12:05

yes? quit !ok -- I have at least 400MB to work with

ferret_v553beta5 -memsize 1000 Unable to allocate the requested 1000.000000 Mwords of memory. !ok -- I don't have 1000MB

ferret_v553beta5 -memsize 10000 NOAA/PMEL TMAP FERRET v5.53 (beta5) IRIX 6.5 - 09/10/03 19-Sep-03 12:04

yes? !uh oh -- there's no way I have 10000MB!

Andrew

And in Ferret v6.00:

% ferret -memsize 1024 Unable to allocate the requested 1024.000000 Mwords of memory. % ferret -memsize 2024 Unable to allocate the requested 2024.000000 Mwords of memory. % ferret -memsize 3024 NOAA/PMEL TMAP FERRET v6
Linux(g77) 2.4.21-32 - 08/23/06 28-Aug-06 12:33

yes?

Migrated-From: http://dunkel.pmel.noaa.gov/trac/ferret/ticket/1438

[karlmsmith]

Comment by @AnsleyManke on 29 Aug 2006 15:50 UTC (Ferret v6.0)

The lack of error message occurs on 32-bit linux but not on 64-bit or solaris. On both 64-bit and solaris it reports the wrong attempted memsize, but does give an error message

on stout (32 bit)

ferret -memsize 1000 Unable to allocate the requested 1000.000000 Mwords of memory. ferret -memsize 10000

yes? sh mem Current size of FERRET memory cache: -2147 MegaWords (1 word = 4 bytes) yes? use etopo20 yes? shade rose Segmentation fault

likewise, within Ferret, a ridiculous request SET MEM/SIZ= results in no error or warning, and we can't use ordinary amounts of memory.

yes? set mem/size=200000 Cached data cleared from memory yes? show mem Current size of FERRET memory cache: -1863 MegaWords (1 word = 4 bytes) yes? set mem/siz=20000 Cached data cleared from memory yes? use etopo20 yes? shade rose STOP FREE_MEMORY statement executed

On porter (64-bit) things seem more rational ferret -memsize 100000 Unable to allocate the requested -2147.483648 Mwords of memory.

ferret yes? set mem/siz=200000 Cached data cleared from memory Unable to allocate -1863.462912 Mwords of memory. Restoring previous memory of 6.400000 Mwords. yes? sh mem Current size of FERRET memory cache: 6.4 MegaWords (1 word = 4 bytes)

[karlmsmith]

Comment by @AnsleyManke on 5 Dec 2006 19:43 UTC On some of the operating systems, the attempt to express the number of words requested resulted in an overflow, so the request looked like a negative number. Trap for this in xeq_set.F and report the failure to allocate the requested amount of memory.

[karlmsmith]

Comment by @AnsleyManke on 7 Oct 2010 01:01 UTC The above fix is in xeq_set.F, but in some cases requesting even larger amounts of memory, there still seems to be an integer overflow that is not trapped.

This works as expected

yes? set mem/siz=9e+4
 *** NOTE: internal overflow expressing 9e4 Mwords as words. Restoring previous memory size.
 Cached data cleared from memory
yes? show mem
 Current size of FERRET memory cache: 25.6 MegaWords  (1 word = 4 bytes)

But, try a larger value, same actual result, but no NOTE:

yes? set mem/siz=9e+12
 Cached data cleared from memory
yes? sh mem
 Current size of FERRET memory cache: 25.6 MegaWords  (1 word = 4 bytes)

[karlmsmith]

Comment by @AndrewWittenberg on 8 Sep 2011 21:20 UTC Bumping this bug up in priority, since we're starting to see problems for which the many Gigabytes of memory available on some of our systems would aid analysis of high-res data. What is the maximum memory usable within Ferret, via SET MEMORY?

On my workstation with 6 Gb (1520 Mwords) of memory:

        NOAA/PMEL TMAP
        FERRET v6.71  
        Linux 2.6.18-274.el5PAE 32-bit - 08/05/11
         8-Sep-11 17:08     

yes? set mem/size=500
 Cached data cleared from memory
yes? set mem/size=600
 Cached data cleared from memory
Unable to allocate 600.0 Mwords of memory.
Restoring previous memory of 500.0 Mwords.
yes? set mem/size=1100
 Cached data cleared from memory

On our analysis node with 96 Gb (24000 Mwords) of memory:

        NOAA/PMEL TMAP
        FERRET v6.401  
        Linux(g77) 2.6.9-89.ELsmp - 12/09/09
         8-Sep-11 16:42     
yes? set mem/size=2147
 Cached data cleared from memory
yes? set mem/size=2148
 *** NOTE: internal overflow expressing =2148 Mwords as words. Restoring previous memory size.
 Cached data cleared from memory
yes? set mem/size=10737
 Cached data cleared from memory

Lots of confusion there.

[karlmsmith]

Comment by @AnsleyManke on 12 Dec 2011 23:22 UTC This was fixed in version 6.7; the value of the setting is now saved differently. Previous to v6.7, the number entered by the user was converted immediately to the number of words, and that calculation could trigger an integer overflow. That condition was apparently not correctly diagnosed, as seen in the example using Ferret v6.4 where the smaller size of 2148 gives us the error but 10737 does not.

In v6.7 and higher, the setting that the user makes is stored in Mwords form, and only at the point where memory is re-allocated, the size in words is computed as a long-integer in C.