Closed aidanheerdegen closed 6 years ago
@aidanheerdegen Those values of i, j and ij are ludicrous and are way out of bounds. I think something else is going on here.
I recompiled with debug on (so -O0
) and bt full
using gdb
and the core dump gave this:
at ice_therm_vertical.f90:1556
phi_i_mushy = 0.84999999999999998
qbot0 = -9.2559631349317831e+61
qbotp = -9.2559631349317831e+61
qbotm = -9.2559631349317831e+61
hstot = 0
zqsnew = -9.2559631349317831e+61
wk1 = -797388.44339828182
hqtot = -4024092.2779668011
qsub = -9.2559631349317831e+61
qbot = -279501620.48217207
ts = -9.2559631349317831e+61
ti = -9.2559631349317831e+61
tmlts = -0.17222207969971798
dhs = 4.5467548571190606e-05
dhi = -0
k = 5
ij = 199
j = 248
i = 9
qmlt = (( 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...) ...)
qm = (( 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...) ...)
dzs = <error reading variable dzs (Cannot access memory at address 0x7ffc964dcd70)>
dzi = (( 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...) ...)
zs2 = <error reading variable zs2 (Cannot access memory at address 0x7ffc964d2000)>
zs1 = <error reading variable zs1 (Cannot access memory at address 0x7ffc964ce630)>
zi2 = (( 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...) ...)
zi1 = (( 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...) ...)
In particular, that these can all have dodgy values:
qbot0 = -9.2559631349317831e+61
qbotp = -9.2559631349317831e+61
qbotm = -9.2559631349317831e+61
hstot = 0
zqsnew = -9.2559631349317831e+61
wk1 = -797388.44339828182
hqtot = -4024092.2779668011
qsub = -9.2559631349317831e+61
qbot = -279501620.48217207
ts = -9.2559631349317831e+61
ti = -9.2559631349317831e+61
tmlts = -0.17222207969971798
dhs = 4.5467548571190606e-05
dhi = -0
Is problematic.
@russfiedler the unoptimised version gives more sensible numbers for the indices (i,j,ij).
Looks like a grid problem. (i,j)=(9,248) should be land (88.5E,65.2N). About 500km inland in fact...
Well spotted @russfiedler, thanks. You're right, so it shouldn't be surprising that there are garbage values there, but why is it even trying to calculate anything there at all? I use the same model inputs (only changing the vertical grid and topog (partial cells)), and it works fine. It uses the same kmt.nc file which is where it seems to be getting it's ocean mask from. I'm a little confused.
Hang on, I might be wrong on the location there. I think i,j might just refer to a local i,j rather than global. What tile are we on?
Not sure. I tried to wring that info out of the core dump with no success.
I have to run it through a debugger, but ran out of time today. Thanks for the feedback @russfiedler
Forgot to ping @nicjhan
Thanks to @nicjhan I have discovered it was a land/sea mask mismatch. I didn't use the most recent topography with the changes to the Bering Strait that Nic made.
@aidanheerdegen Was the location at about (i,j)=113,248), (x,y)=(-167,65.6)? I worked out 5 possible coordinates from the crash info above. Just checking if I was right or was I barking up the wrong tree?
Turns out the crash I thought was MATM (https://github.com/OceansAus/matm/issues/4) is in cice.
This is not a CICE issue, but I thought it important to document in case someone else has the same problem.
To recap, this is the ACCESS-OM-1deg JRA55 RYF config, but with a new (KDS50) vertical level scheme. I have interpolated the initial conditions but as far as I know nothing else depends on the ocean vertical grid.
The crash is a divide by zero, the initial traceback has no information:
even though I recompiled cice with
-g
. If I load the core dump with gdb, I get this info:The line number is probably not reliable (still
-02
flag on), so I think the crash is here:https://github.com/OceansAus/cice5/blob/49b36d4bfb97328e818d428d0b8438144dbd69a1/source/ice_therm_vertical.F90#L1554
as
qbotp = 0
.I'm guessing there is some ice initial conditions issues, but I don't know why change the ocean vertical grid would impact the ice. Any ideas?