Value out of range when using the KDS75_5mdz grid

[willaguiar]

The new simulation with the reduced vertical grid (KDZ75_5mdz) runs, however it crashes by getting values out of range for temperature (timestep =300) and salinity (timestep 200). e.g.,:

94845:FATAL from PE  4339:  Error: salinity out of range with value     1.000880841523E+02 at (i,j,k) = (2650,2673,  4),  (lon,lat,dpt) = (  -54.0848\
,   88.3861,   21.3670 m)

This was happening before in the control case, and was fixed by reducing the timestep from 600 to 300. However, for the reduced vertical grid case here, the error only goes away when I go to a timestep of 150 (which makes the simulation very slow). Maybe there is a workaround for this issue without reducing the tilmestep so much?

[willaguiar]

Perhaps I can separate the first trimester into 3 separate runs, and progressively increase the timestep from 150 to 300 from Jan to March?

[adele-morrison]

Have you looked at the location where it's crashing? Looks like it's in the Arctic somewhere? Does the bathymetry there look ok? I would have thought that just by deepening cells we shouldn't have made any points non-advective, but worth a check to have a close look at the crash location.

Another thought, how is the river insertion / mixing scheme set? Does it insert river runoff over a fixed depth range or over a fixed number of grid cells? If it's a fixed depth that will be problematic, because we now have a lot less cells than before. Worth a check of the input parameters and Kiss et al. 2020. Though in this crash it seems to be getting too salty, so I guess forming sea ice and not being able to move that salt around sufficiently.

On Thu, 13 Apr 2023 at 20:30, Wilton Aguiar @.***> wrote:

Perhaps I can separate the first trimester into 3 separate runs, and progressively increase the timestep from 150 to 300 from Jan to March?

— Reply to this email directly, view it on GitHub https://github.com/willaguiar/DSW-collaborative-project/issues/15#issuecomment-1506729146, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACA44U7JNRSP6QGMPMC53TTXA7ISPANCNFSM6AAAAAAW44DIGU . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[PaulSpence]

wondering what time of year is the crash (winter growth or summer melt), the max depth and how thick is the sea ice?also good ideas to check for non-advective cell

Re: river insertion depth.
Looks like a depth range in meters from https://github.com/mom-ocean/MOM5/blob/master/src/mom5/ocean_param/sources/ocean_rivermix.F90

! Thickness of the column over which to insert tracers from ! rivers. Default river_insertion_thickness=0.0 (all in top). !

[adele-morrison]

In input.nml: river_insertion_thickness = 40.0

Should still be ok, we still have a bunch on cells in the top 40m right?

[willaguiar]

I just check out n=4 in the KDS75_5mdz, which is z=21.425 m. Slightly different than the depth where the error occurs. Does that mean that this could be a problem with partial cells? @adele157

[dkhutch]

If you mean that the bottom of the k=4 cell is 21.425 m, and the bathymetry in question is 21.367 m depth, then (I think) that means you have something like 3.9 grid cells, which should be fine. If you mean that the bathymetry in 21.425 m, and the bottom of the k=4 cell is 21.367 m, then you would have something like 4.1 grid cells, which may be a problem.

[willaguiar]

Some updates... The salt_restore_tscale was changed to 45.4 days to maintain the piston velocity of 0.11m/day for the control and new run. Also partial cells were fixed using topogtools. The partial cells corrections changes Arctic topography as in the fig below [e,f].

However, no changes happen at the location of the salinity error (red circle). In all cases, the Zmax(-54.08,88.38)=2840.4 m. Running the experiment with the new Bathymetry (i.e., after all the changes described above) still gives the same salinity error. So I still don't know what generates the error... any suggestions?

[PaulSpence]

I'm a little surprised it's crashing in the open ocean. What is the sea ice thickness there? Maybe start outputting high frequency data just prior to the error and watch what is going on (ncview movies). How long does the simulation run before this error is reached?

[willaguiar]

I'm a little surprised it's crashing in the open ocean. What is the sea ice thickness there? Maybe start outputting high frequency data just prior to the error and watch what is going on (ncview movies). How long does the simulation run before this error is reached?

@PaulSpence The error appears right after CICE starts ... last lines on om2.out are:

 &MOM_OASIS3_INTERFACE_NML
 NUM_FIELDS_IN   =          15,
 NUM_FIELDS_OUT  =           7,
 FIELDS_IN       = u_flux  v_flux  lprec   fprec   salt_flxmh_flux sw_flux q_flux  t_flux  lw_flux runof   p       aice    wfimelt wfiform ,
 FIELDS_OUT      = t_surf  s_surf  u_surf  v_surf  dssldx  dssldy  frazil  ,
 SEND_BEFORE_OCEAN_UPDATE        = F,
 SEND_AFTER_OCEAN_UPDATE = T
 /
 CICE_COMMIT_HASH=d3e8bdfadd90bab211412da3b616869aed912501
 cicexx: LIBACCESSOM2_COMMIT_HASH=1bb890411da1484215f76c53772c55d38fbd5a05

I am able to run the model with timestep of 150 tho. Maybe I should run with this timestep for ~10 days and check the sea ice.

[PaulSpence]

okay, so the model is crashing essentially on start up, right? I think that means your model config is messed up. It's not some ocean process going crazy (e.g. too much sea ice growth). Perhaps start by listing all the input files you changed and what you changed in them. Do diffs with good files that you know work.

On Mon, Apr 17, 2023 at 2:46 PM Wilton Aguiar @.***> wrote:

I'm a little surprised it's crashing in the open ocean. What is the sea ice thickness there? Maybe start outputting high frequency data just prior to the error and watch what is going on (ncview movies). How long does the simulation run before this error is reached?

@PaulSpence https://github.com/PaulSpence The error appears right after CICE starts ... last lines on om2.out are:

&MOM_OASIS3_INTERFACE_NML NUM_FIELDS_IN = 15, NUM_FIELDS_OUT = 7, FIELDS_IN = u_flux v_flux lprec fprec salt_flxmh_flux sw_flux q_flux t_flux lw_flux runof p aice wfimelt wfiform , FIELDS_OUT = t_surf s_surf u_surf v_surf dssldx dssldy frazil , SEND_BEFORE_OCEAN_UPDATE = F, SEND_AFTER_OCEAN_UPDATE = T / CICE_COMMIT_HASH=d3e8bdfadd90bab211412da3b616869aed912501 cicexx: LIBACCESSOM2_COMMIT_HASH=1bb890411da1484215f76c53772c55d38fbd5a05

I am able to run the model with timestep of 150 tho. Maybe I should run with this timestep for ~10 days and check the sea ice.

— Reply to this email directly, view it on GitHub https://github.com/willaguiar/DSW-collaborative-project/issues/15#issuecomment-1510689632, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABSWJXC3DWVAB7TV5UMHKTLXBTDJBANCNFSM6AAAAAAW44DIGU . You are receiving this because you were mentioned.Message ID: @.***>

-- I’m sending this message at a time that suits me. I don’t expect you to reply outside of your own work hours.

Paul Spence, Assoc. Prof. ARC Future Fellow Institute for Marine and Antarctic Studies https://www.imas.utas.edu.au/ University of Tasmania, Hobart, Australia https://paulspence.github.io/

[willaguiar]

Okay, it was a stupid mistake. I used diff in all inputs and saw the only difference was that I set min_thickness=5 m for the low resolution run while in the high vertical resolution was min_thickness=1 m. I went double check if the 5m value was correct. Turns out that although I set 5.0 m as the min dz when creating the KDS75_5mdz grid, min dz became 5.3m after the hyperbolic function fitting in the vgrid creation tool. I changed to min_thickness=5.3 m and now the simulation runs fine. (I didn't know it was that sensitive to the decimal degree)

thanks @PaulSpence

PS: this fixes the salinity error. but the temperature error with tilmestep 300 still occurs, but in the Southern Ocean this time (checking now what it could be):

FATAL from PE    35:  Error: temperature out of range with value    -3.139712524018E+01 at (i,j,k) = (1225\
, 116,  8),  (lon,lat,dpt) = ( -157.5500,  -76.2511,   56.9817 m)

edit: Temperature error disappears with timestep 240. I guess we can run the control and reduced resolution simulations with an initial timestep of 240, and increase afterwards..... opinions?