CICE C-grid runs using 0.25deg

[anton-seaice]

Following recent discussions , I ran the 025deg ryf with

Diabatic_first=True Dt_therm = 5400s (1.5 hours)

and otherwise the default configuration (the om2 grid / bathymetry / initial conditions / cice b-grid)

It ran until the 18th March in the second year

And crashed with too many truncations :

FATAL from PE   379: write_energy : Ocean velocity has been truncated too many times.

Image              PC                Routine            Line        Source             
access-om3-MOM6-C  0000000002E3BB14  mpp_mod_mp_mpp_er          72  mpp_util_mpi.inc
access-om3-MOM6-C  0000000001E4F82F  mom_error_handler         154  MOM_error_handler.F90
access-om3-MOM6-C  0000000001D4F9DF  mom_sum_output_mp         904  MOM_sum_output.F90
access-om3-MOM6-C  0000000001C0E444  mom_mp_step_mom_         1016  MOM.F90
access-om3-MOM6-C  0000000001BAE725  mom_ocean_model_n         624  mom_ocean_model_nuopc.F90
access-om3-MOM6-C  0000000001B780C2  mom_cap_mod_mp_mo        1883  mom_cap.F90
access-om3-MOM6-C  0000000000AE1C88  Unknown               Unknown  Unknown
access-om3-MOM6-C  0000000000AE1BEF  Unknown               Unknown  Unknown
access-om3-MOM6-C  0000000000AE07A2  Unknown               Unknown  Unknown
access-om3-MOM6-C  0000000000C47E82  Unknown               Unknown  Unknown
access-om3-MOM6-C  0000000003B67E20  Unknown               Unknown  Unknown
access-om3-MOM6-C  00000000009F0FB2  Unknown               Unknown  Unknown
access-om3-MOM6-C  00000000009F4B0F  Unknown               Unknown  Unknown
access-om3-MOM6-C  0000000000B4B39A  Unknown               Unknown  Unknown
access-om3-MOM6-C  0000000000B37EE5  Unknown               Unknown  Unknown
access-om3-MOM6-C  00000000009F23EA  Unknown               Unknown  Unknown
access-om3-MOM6-C  000000000050CACA  Unknown               Unknown  Unknown
access-om3-MOM6-C  0000000000759361  Unknown               Unknown  Unknown
access-om3-MOM6-C  00000000009666A7  Unknown               Unknown  Unknown
...

Is this consistent with your work @minghangli-uni - should I increase the number of allowable truncations and see if they stabilise?

Config is here: https://github.com/ACCESS-NRI/access-om3-configs/compare/34e2136...51fbffa1d152d7773093c9d26924fd0d08fa5880 Output is in:

/g/data/tm70/as2285/payu/dev-025deg_jra55do_ryf-bgrid/work

[anton-seaice]

I reported the shortage of info in the stack trace here: https://github.com/ACCESS-NRI/ACCESS-OM3/issues/13

[anton-seaice]

I will set the DT_THERM back to 1350s and try again :)

[anton-seaice]

With DT_THERM set to the default, the Bgrid configuration is still running (almost 2 years).

With the Cgrid (https://github.com/ACCESS-NRI/access-om3-configs/compare/34e2136cab181fd38068d9e35bbc85f3f9de9db9...anton-seaice:MOM6-CICE6:dev-025deg_jra55do_ryf-cgrid?expand=1), I get the errors with eta dropped below bathyT

The errors are :

WARNING from PE 1036: btstep: eta has dropped below bathyT: -1.1825361824810603E+01 vs. -1.1805749893188477E+01 at -1.9538E+02 5.8943E+01 340 793

and

WARNING from PE 1036: Extreme surface sfc_state detected: i= 332 j= 788 lon=-197.125 lat= 58.423 x=-197.125 y= 58.423 D= 6.5867E+01 SSH=-6.2522E+02 SST=-1.7967E+00 SSS= 3.3376E+01 U-= 4.8522E+00 U+= 4.8522E+00 V-= 4.8793E+00 V+=-4.8252E+00

It doesn't say which extreme sfc_state detected is causing the issue, but presumably SSH.

There is a bit of a "shelf?" in the crash location (a nameless bay, western bering sea):

But nothing else obvious going on - I looked at daily mean of SSH, and sea-ice concentration.

I can't see how turning on C-grid in CICE has impacted this ? Should I turn on time-step output for SSH and re-run (or is eta saved some other way ? )

[aekiss]

SSH=-6.2522E+02 is pretty extreme! This indicates numerical instability. Have you tried a shorter barotropic or baroclinic timestep?

[aekiss]

From the messages it looks like the error might originate in the barotropic solver, so you could try reducing the magnitude of DTBT, and/or try setting DTBT_RESET_PERIOD=0.0.

[anton-seaice]

I ran with DTBT of 0.5 and found CICE crashed with negative area.

I raised https://github.com/CICE-Consortium/CICE/issues/992

Config is https://github.com/anton-seaice/MOM6-CICE6/commit/e450508875457c3e0345c0f03d810269437765c7

Analysis is https://github.com/anton-seaice/sandbox/blob/main/025deg_cgrid_neg_ice_area.ipynb

[minghangli-uni]

This might be unrelated, but does the crash occur with the 1-degree config? My understanding is that the logic behind the C-grid in CICE6 is the same for both 1 deg and 0.25 deg configs. If the 1 deg config runs without issues, it might suggest that the problem is still related to the bathymetry.

[anton-seaice]

Its possible - but there have been other reports / questions about excessive ice velocities and the new scheme being possibly unstable, so I thought it was worth raising. And the patterning in sea ice concentration looks more like an issue with boundary conditions at the edge of the land mask or similar ?

The 1 degree config did run, but I am not convinced there wasn't a problem with the velocities at the ice edge. e.g. see the last plot in https://github.com/COSIMA/access-om3/issues/39#issuecomment-2311719993 where the c grid velocities at the ice edge are higher than b grid

I think the idea of running with the short barotropic timestep was to improve ocean stability ? I'm a bit out of my depth with that though. An issue with bathymetry typically shows in the baroclinic timestep doesn't it ?

[minghangli-uni]

The 1 degree config did run, but I am not convinced there wasn't a problem with the velocities at the ice edge. e.g. see the last plot in https://github.com/COSIMA/access-om3/issues/39#issuecomment-2311719993 where the c grid velocities at the ice edge are higher than b grid

I think the greater velocities at the ice edge is sensible, because Cgrid configurations allow for a more accurate representation of velocity gradients, which can lead to hgiher velocities at the ice edge, especially when resolving finer details compared to Bgrid.

A better way to debug this issue could be reducing both barotropic and baroclinic timesteps further or increasing the coupling frequency to ensure more data exchange between mom and cice.