search

NOAA-GFDL / SIS2

NOAA-GFDL's Sea Ice Simulator version 2
Other
16 stars 43 forks source link

MOM6_SIS2_cgrid and MOM6_SIS2_OM4_025 tests crash #10

Closed nikizadehgfdl closed 9 years ago

nikizadehgfdl commented 9 years ago

dev/master commit 59133a0e623dee414c562758b00cdf2ee7aaeb47

MOM6_SIS2_cgrid and MOM6_SIS2_OM4_025 test cases crash at day 2.5 with

FATAL from PE 0: write_ice_statistics: Sea ice velocity has been truncated too many times.

This is new and these tests were passing with the tag dev/master/2014.11.17 (commit 555931cd15821ab830ec83fcc6de8c1add89855a)

Here's the stdouts on gaea:

/lustre/f1/Niki.Zadeh/testing_20141126_mom6_20141126_0/MOM6_SIS2_cgrid/ncrc2.intel-prod-openmp/stdout/run/MOM6_SIS2_cgrid_1x0m20d_32pe.o6792957

/lustre/f1/Niki.Zadeh/testing_20141126_mom6_20141126_0/MOM6_SIS2_OM4_025_mask_table.96.32x18/ncrc2.intel-prod-openmp/stdout/run/MOM6_SIS2_OM4_025_mask_table.96.32x18_1x0m10d_480pe.o6792972

Hallberg-NOAA commented 9 years ago

This is a known diagnostic issue that was introduced by checking for sea-ice CFL limits, and can be disabled by setting CFL_TRUNCATE in SIS_input to be negative, however it is indicative of potentially problematic behavior that can be avoided with the new parameter options subsequent to (commit 555931c). Specifically, this issue has been corrected for the SIS2_cgrid and AM2_SIS2_MOM6i_1deg test cases by changing settings in SIS_input file with commit 63821826 (both run successfully for 30 days). Similar changes to the algorithm will soon be checked in for the OM4_025 test case.

Hallberg-NOAA commented 9 years ago

I just checked in changes for the OM4_025 test case and successfully ran it for 10 days. I think that this issue will not reoccur, and if it does it can be avoided by taking a smaller value of DT_ICE_DYNAMICS.

nikizadehgfdl commented 9 years ago

This issue is there for test case SIS2_bergs_cgrid in commit f6085017994f573dc86bf2234583675fc44df838 This test case has DT_ICE_DYNAMICS=7200. Reducing it to 3600 fixes the problem. Should that be the default like the other test case SIS2_cgrid?

To reproduce increase days to 3 in coupler_nml to get:

FATAL from PE     0: write_ice_statistics: Sea ice velocity has been truncated too many times.

Rank 0 [Mon Jan  5 23:13:58 2015] [c4-1c0s3n1] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
SIS Day    2.250     27: Area 1.178644779341E+13 4.390281524067E+10, Mass  1.74586577392E+15 3.71638870314E+12
SIS Day       2.250 Sea Ice Truncations 1
 Total Ice Mass:   1.7495821626280622E+15, Change:  1.15080E+14 Error:  4.34300E-03 ( 2.5E-18)
 Total Ice Salt:   8.7139282084262142E+09, Change:  5.69394E+08 Error:  8.79845E-08 ( 1.0E-17)
 Total Ice Heat:  -5.9953932906225612E+20, Change: -4.06808E+19 Error:  1.70256E+04 (-2.8E-17)
nikizadehgfdl commented 9 years ago

SIS2_bergs_cgrid keeps crashing at day 2. Please set DT_ICE_DYNAMICS = 7200 in example SIS2_bergs_cgrid to avoid this.

adcroft commented 9 years ago

Was able to reproduce this. I'll change the time-step for all the C-grid 1-degree ice_ocean_SIS2 tests.