f.cam6_3_100.FWscHIST.ne30_L58.001

[cecilehannay]

Description: @adamrher , @JulioTBacmeister and @PeterHjortLauritzen would like a new baseline created with cam6_3_100 (FWscHIST, L58) in preparation for evaluating new science code (incl. condensates in pressure and having energy fixer use an energy consistent with the dynamical core).

This baseline will also include the bug fix from @tto061 and @bstephens82's latest clubb tuning as described in #223.

It also includes the TEM diags as requested by @dan800

user_nl_cam

mfilt           = 0,5,20,40,120,240,365,73,365
nhtfrq         =                  0, -24,  -6,  -3,  -6,   1, -24,-24
fexcl1 = ' '  

empty_htapes = .true.

fincl1 = 'ACTNI', 'ACTNL', 'ACTREI', 'ACTREL', 'AODDUST', 'AODVIS', 'BURDENBC', 'BURDENDUST', 'BURDENPOM', 'BURDENSEASALT', 
'BURDENSO4', 'BURDENSOA', 'CAPE', 'CCN3', 'CDNUMC', 'CH4', 'CLDHGH', 'CLDICE', 'CLDLIQ', 'CLDLOW', 'CLDMED', 'CLDTOT', 'CLOUD', 'CMFMC_DP', 
'CT_H2O', 'DCQ', 'DQCORE', 'DTCOND', 'DTCORE', 'DTV', 'EVAPPREC', 'EVAPSNOW', 'FCTI', 'FCTL', 'FICE', 'FLDS', 'FLNS', 'FLNSC', 'FLNT', 'FLNTC', 'FLUT', 
'FREQZM', 'FSDS', 'FSDSC', 'FSNS', 'FSNSC', 'FSNT', 'FSNTC', 'FSNTOA', 'ICEFRAC', 'LANDFRAC', 'LHFLX', 'LWCF', 'MPDICE', 'MPDLIQ', 'MPDQ', 'MPDT', 
'OCNFRAC', 'OMEGA', 'OMEGA500', 'PBLH', 'PHIS', 'PINT', 'PMID', 'PRECC', 'PRECL', 'PRECSC', 'PRECSL', 'PRECT', 'PS', 'PSL', 'PTEQ', 'PTTEND', 'Q', 
'QFLX', 'QRL', 'QRS', 'QTGW', 'RCMTEND_CLUBB', 'RELHUM', 'RVMTEND_CLUBB', 'SHFLX', 'SOLIN', 'SST', 'STEND_CLUBB', 'SWCF', 
'T', 'TAUX', 'TAUY', 'TFIX', 'TGCLDIWP', 'TGCLDLWP', 'TMQ', 'TREFHT', 'TS', 'TTGW', 'U', 'U10', 'UBOT', 'UTGWORO', 'UTGW_TOTAL', 
'V', 'VBOT', 'VTGWORO', 'VTGW_TOTAL', 'WPRTP_CLUBB', 'WPTHLP_CLUBB', 'Z3', 'ZMDQ', 'ZMDT', 'N2O',
'WV_phBF:A','WL_phBF:A','WI_phBF:A','SE_phBF:A','KE_phBF:A',
'WV_phBP:A','WL_phBP:A','WI_phBP:A','SE_phBP:A','KE_phBP:A',
'WV_phAP:A','WL_phAP:A','WI_phAP:A','SE_phAP:A','KE_phAP:A',
'WV_phAM:A','WL_phAM:A','WI_phAM:A','SE_phAM:A','KE_phAM:A'

fincl2 = 'OMEGA', 'PMID', 'PS', 'Q', 'QRL', 'QRS', 'T', 'TROP_P', 'TROP_T', 'U', 'V', 'Z3'

fincl4 =  'PRECC','PRECL'

history_aerosol = .true.

ncdata = '/glade/p/cesm/amwg_dev/juliob/FWsc_ne30pg3_58L_GRID_48_taperstart10km_lowtop_BL10_v3_beta1p75_Top_43km.nc'

interpolate_output = .true.,.true.
interpolate_nlat = 192,192
interpolate_nlon = 288,288

se_rsplit            = 3
se_nsplit           = 2
se_hypervis_subcycle =  4

bnd_topo  = '/glade/p/cgd/amp/juliob/bndtopo/latest/ne30pg3_gmted2010_modis_bedmachine_nc3000_Laplace0100_20230105.nc'

use_gw_convect_dp = .false.
use_gw_front = .false.

ubc_specifier = 'Q:H2O->UBC_FILE'
ubc_file_path = '/glade/p/cesmdata/inputdata/atm/cam/chem/ubc/f.e21.FWHISTBgcCrop.f09_f09_mg17.CMIP6-AMIP-WACCM.ensAvg123.cam.h0zm.UBC.195001-201412_c220322.nc'
ubc_file_input_type = 'SERIAL'

co2_flag = .false.

fincl5 = 'VTHzaphys','WTHzaphys','UVzaphys','UWzaphys'
phys_grid_ctem_nfreq=-6
phys_grid_ctem_zm_nbas=120
phys_grid_ctem_za_nlat=90

clubb_l_predict_upwp_vpwp=.true. 
clubb_l_mono_flux_lim_um   = .true.  
clubb_l_mono_flux_lim_vm   = .true.  
clubb_c_uu_shr = 0.1
clubb_c7=0.1

SourceMods Thomas's bug fix

/glade/p/cesmdata/cseg/runs/cesm2_0/f.cam6_3_100.FWscHIST.ne30_L58.001/SourceMods/src.cam/mono_flux_limiter.F90

Case directory: Locally (if still available):

/glade/p/cesmdata/cseg/runs/cesm2_0/f.cam6_3_100.FWscHIST.ne30_L58.001

On github: https://github.com/NCAR/amwg_dev/tree/f.cam6_3_100.FWscHIST.ne30_L58.001

Sandbox: Locally (if still available):

/glade/work/hannay/cesm_tags/f.cam6_3_100

On github: https://github.com/ESCOMP/CAM/tree/cam6_3_100

hash: 82baf1a

Diagnostics: AMWG diags (if available) https://webext.cgd.ucar.edu/FWscHIST/f.cam6_3_100.FWscHIST.ne30_L58.001/atm/

Contacts: @adamrher , @JulioTBacmeister, @PeterHjortLauritzen, @tto061, @bstephens82, @cecilehannay, @cacraigucar, @jtruesdal

[cecilehannay]

@islasimpson is suggesting to add Uzm, Wzm, Vzm, THzm to the TEM diagnostics. I am modifying the namelist to do that.

[cecilehannay]

@fvitt and @islasimpson: I tried to output the TEM diags, but it seems that the variables don't exist.

 FLDLST: VTHzaphys in fincl(1, 5) not found
 FLDLST: WTHzaphys in fincl(2, 5) not found
 FLDLST: UVzaphys in fincl(3, 5) not found
 FLDLST: UWzaphys in fincl(4, 5) not found
 FLDLST: Uzm in fincl(5, 5) only available with FV dycore
 FLDLST: Wzm in fincl(6, 5) only available with FV dycore
 FLDLST: Vzm in fincl(7, 5) only available with FV dycore
 FLDLST: THzm in fincl(8, 5) only available with FV dycore

[islasimpson]

I think you need "do_tem_diag=True" for the first four. It looks like not all of these are currently included in the TEM diagnostics code. It seems like it would be a lot easier if they were though. Should they be added?

[fvitt]

@cecilehannay The zm history fiields are available only when the FV dycore is used. You are using the SE dycore.

The calls to phys_grid_ctem are not in 'src/physics/cam_dev/physpkg.F90', only in src/physics/cam/physpkg.F90. This is a bug. Because you are using cam_dev physics the zaphys history fields were not found.

[adamrher]

uh oh! @fvitt can you open a git issue please?

[cecilehannay]

I am skipping the tem_diag for now.

[cecilehannay]

This run is crashing for the error:

2107: ERROR: In UpdateState_TopLayerFluxes, h2osoi_ice has gone significantly negativ
2107: e
2107: Bulk/tracer name = bulk
2107: c, lev_top(c) =           61           0
2107: h2osoi_ice_top_orig =   1.990291810086836E-006
2107: h2osoi_ice          =  -3.550762114890027E-018
2107: frac_sno_eff        =    1.00000000000000
2107: qflx_soliddew_to_top_layer*dtime =   0.000000000000000E+000
2107: qflx_solidevap_from_top_layer*dtime =   1.990291810090387E-006
2107:iam = 2107: local  column   index = 61
2107:iam = 2107: global column   index = 97626
2107:iam = 2107: global landunit index = 44433
2107:iam = 2107: global gridcell index = 15068
2107:iam = 2107: gridcell longitude    =  239.8560077
2107:iam = 2107: gridcell latitude     =   50.4828476
2107:iam = 2107: column   type         = 75
2107:iam = 2107: landunit type         = 9
2107: ENDRUN:
2107: ERROR:
2107: In UpdateState_TopLayerFluxes, h2osoi_ice has gone significantly negative

[adamrher]

Hmm ... this run got 4 years in before dumping out. @wwieder @olyson is there any way we can determine whether we need to adjust the h2osoi_ice threshold for negative values, or whether the state really is becoming increasingly unrealistic (and therefore an actual blow-up)?

[wwieder]

I don't know how one would generate negative soil ice, I may defer to @olyson on this one.

[olyson]

Based on previous conversations about this error that crops up very occasionally and the size of the error (e-18) I doubt the state is becoming unrealistic, e.g.,

https://bb.cgd.ucar.edu/cesm/threads/error-in-updatestate_toplayerfluxes-h2osoi_ice-has-gone-significantly-negative.7634/

I'll discuss this with @billsacks later to see if we can do something more robust (I think he is out on PTO), but in the meantime try changing custom_rel_epsilon to 1.e-11_r8 in this block of code in /components/clm/src/biogeophys/SnowHydrologyMod.F90 (make sure it's the h2osoi_ice block):

call truncate_small_values_one_lev( &
     num_f = num_snowc, &
     filter_f = filter_snowc, &
     lb = bounds%begc, &
     ub = bounds%endc, &
     lev_lb = -nlevsno+1, &
     lev = lev_top(bounds%begc:bounds%endc), &
     data_baseline = h2osoi_ice_top_orig(bounds%begc:bounds%endc), &
     data = h2osoi_ice(bounds%begc:bounds%endc, :), &
     custom_rel_epsilon = 1.e-12_r8)

[cecilehannay]

@olyson: Thanks for your fast reply. I changed the custom_rel_epsilon to 1.e-11_r8 in the h2osoi_ice block. Hopefully, this will allow us to keep going until we have a more robust solution.

[dan800]

uh oh! @fvitt can you open a git issue please?

@fvitt I am not understanding this issue - you had previously provided output from and SE run that I was able to calculate TEM diags from. I thought the code mods were merged into the version Cecile is running so we can do a similar TEM analysis on all runs going forward. As mentioned elsewhere, all we need to begin with is TEM terms to be calculated every 6 hours and averaged over a month - I'll likely be trying to get seasonal TEM diags as a standard part of the new climate diagnostics package.

[fvitt]

uh oh! @fvitt can you open a git issue please?

@fvitt I am not understanding this issue - you had previously provided output from and SE run that I was able to calculate TEM diags from. I thought the code mods were merged into the version Cecile is running so we can do a similar TEM analysis on all runs going forward. As mentioned elsewhere, all we need to begin with is TEM terms to be calculated every 6 hours and averaged over a month - I'll likely be trying to get seasonal TEM diags as a standard part of the new climate diagnostics package.

The tests I did for you were using cam6 physics. Cecile is using cam_dev physics which is not calling the phys_grid_ctem routines. As stated in another thread, the fix for this here https://github.com/ESCOMP/CAM/pull/770

[islasimpson]

@fvitt just to clarify, I think the other pieces that are needed are Uzm, Vzm, THzm and Wzm (or whatever we want to call them now). I see they are there in the code as the part that is subtracted to calculate the zonal deviations, before the fluxes are calculated (uzm, vzm, wzm, thzm). Thanks.