OOM with largeish grids

[dschwoerer]

If running with a not-very-small grid, BOUT++ fails, even if a sufficient number of nodes is used. @bendudson suggested that the grid is read in total, and only then sliced. This is happening on a 3.1 GB grid file, on a 72 core machine with 256 GB of ram (3.55 GB/thread) A 2.4 GB file still works.

    Option ZMAX = 1 (default)
    EQUILIBRIUM IS SINGLE NULL (SND) 
Connection between top of Y processor 8 and bottom of 0 in range 0 <= x < 725
=> This processor sending in down
WARNING adding connection: poloidal index -1 out of range
    MYPE_IN_CORE = true
    DXS = 20, DIN = 360. DOUT = -1
    UXS = 0, UIN = -1. UOUT = 45
    XIN = -1, XOUT = 1
    Twist-shift: DI 
    Option twistshift = 0 (default)

< here the OOM happens >

Possible boundary regions are: core, sol, Boundary regions in this processor: core, 
Constructing default regions
    Boundary region inner X
    Option mesh:extrapolate_x = 0 (default)
    Option mesh:extrapolate_y = false (/ptmp/dave/hermes-2/7-timeing.c171//BOUT.inp)
    Option dx = Tensor<BoutReal> (v17/W7X-conf0-724x72x192.emc3.inner:f.vessel:f.island:f.mfi:15e-1.offt:4.fci.nc)
    Option dy = Tensor<BoutReal> (v17/W7X-conf0-724x72x192.emc3.inner:f.vessel:f.island:f.mfi:15e-1.offt:4.fci.nc)
    Option ZMIN = 0 (default)
    Option ZMAX = 1 (default)
    Option dz = Tensor<BoutReal> (v17/W7X-conf0-724x72x192.emc3.inner:f.vessel:f.island:f.mfi:15e-1.offt:4.fci.nc)
    Option mesh:paralleltransform:type = fci (/ptmp/dave/hermes-2/7-timeing.c171//BOUT.inp)
    Option mesh:paralleltransform:z_periodic = 1 (default)
    Option parallel_transform =  (default)

(output is from the smaller grid, to have an idea what is going on)

That looks a lot like the whole grid is read into memory, including the full domain as well as all variables.