High horizontal wind speeds with ne30 (72 levels) -- Failed post-condition property check

[ndkeen]

This was noted in https://github.com/E3SM-Project/scream/issues/1909, but making a more specific issue just for the high wind speed error.

This uses Nov 9th scream repo.

 982: terminate called after throwing an instance of 'std::logic_error'
 982:   what():  /global/cfs/cdirs/e3sm/ndk/repos/se41-nov9/components/scream/src/share/atm_process/atmosphere_process.cpp:256: FAIL:
 982: false
 982: Error! Failed post-condition property check (cannot be repaired).
 982:   - Atmosphere process name: Dynamics
 982:   - Property check name: horiz_winds within interval [-400, 400]
 982:   - Atmosphere process MPI Rank: 982
 982:   - Message: Check failed.
 982:   - check name: horiz_winds within interval [-400, 400]
 982:   - field id: horiz_winds[Physics PG2] <double:COL,CMP,LEV>(8,2,72) [m s^-1]
 982:   - minimum:
 982:     - value: -115.634
 982:     - entry: (19504,1,0)
 982:     - lat/lon: (82.8914, 18.3642)
 982:   - maximum:
 982:     - value: 401.924
 982:     - entry: (19500,0,0)
 982:     - lat/lon: (83.2092, 353.688)

/pscratch/sd/n/ndk/e3sm_scratch/pm-cpu/se41-nov9/f30cpu.F2010-SCREAMv1.ne30pg2_ne30pg2.se41-nov9.gnu.3y.n022a128x1c8.pk8.so480n8.wr.FIo3

I think it fails at 1.98 years (after 726 days)

Normally, I've been avoiding this error by changing 2 files like so:

components/scream/src/dynamics/homme/atmosphere_dynamics.cpp:455:  add_postcondition_check<Interval>(get_field_out("horiz_winds",pgn),m_phys_grid,-500.0, 500.0,false);

components/scream/src/physics/shoc/atmosphere_macrophysics.cpp:372:  add_postcondition_check<Interval>(get_field_out("horiz_winds"),m_grid,-500.0,500.0,false);

I will try again with 128 vertical levels

[ndkeen]

Running with 128 vertical levels seems OK. It completed 3 years. (24.4 sypd)

/pscratch/sd/n/ndk/e3sm_scratch/pm-cpu/se41-nov9/f30cpu.F2010-SCREAMv1.ne30pg2_ne30pg2.se41-nov9.gnu.3y.L128.n022a128x1c8.pk8.so480n8.wr.s4.MCS

Should we close this issue since it works with 128 vertical levels?

@crterai suggested we leave it open for now as a problem with L72. We might look at it again when working on eamxx low-res.

[ndkeen]

For the L128 case, I restarted it and ran another 3 years. So we have 6 years total.