b.e30_beta02.BMT1850.ne30_t232.104

cecilehannay commented 3 months ago

Purpose:

Testing the 104 configuration with MT. The goal is to do a first coupled MT run

Description:

Similar to 104 but with MT

-add RRTMGP

./xmlchange --append CAM_CONFIG_OPTS="-rad rrtmgp"

-hybrid from 98 at year 201

./xmlchange RUN_REFCASE=b.e23_alpha17f.BLT1850.ne30_t232.098
./xmlchange RUN_REFDATE=0201-01-01
./xmlchange  RUN_TYPE=hybrid
./xmlchange  GET_REFCASE=true
./xmlchange  RUN_REFDIR=cesm2_init

-MOM vertical coordinate set to hycom1

./xmlchange MOM6_VERTICAL_GRID=hycom1

-user_nl_cam


mfilt           =       0,       5,     20,      40,      12,       120,      1,   1
nhtfrq              =       0,     -24,    -24,      -3,       0,       -2,      0,  -8760
ndens               =       2,       2,      2,       2,       2,       1,      2,   1
interpolate_output  =  .true.,  .true., .true., .false., .false., .true.,  .true.
interpolate_nlat    =     192,     192,    192,     192,     192,     192,   192
interpolate_nlon    =     288,     288,    288,     288,     288,     288,   288 

empty_htapes = .true.

fincl1 = 'ACTNI', 'ACTNL', 'ACTREI', 'ACTREL', 'AODDUST', 'AODVIS', 'AODVISdn','BURDENBC', 'BURDENDUST', 'BURDENPOM', 'BURDENSEASALT', 
'BURDENSO4', 'BURDENSOA', 'CAPE', 'CCN3', 'CDNUMC', 'CH4', 'CLDHGH', 'CLDICE', 'CLDLIQ', 'CLDLOW', 'CLDMED', 'CLDTOT', 'CLOUD', 'CMFMC_DP', 
'CT_H2O', 'DCQ', 'DQCORE', 'DTCOND', 'DTCORE', 'DTV', 'EVAPPREC', 'EVAPSNOW', 'FCTI', 'FCTL', 'FICE', 'FLDS', 'FLNS', 'FLNSC', 'FLNT', 'FLNTC', 'FLUT', 
'FREQZM', 'FSDS', 'FSDSC', 'FSNS', 'FSNSC', 'FSNT', 'FSNTC', 'FSNTOA', 'ICEFRAC', 'LANDFRAC', 'LHFLX', 'LWCF', 'MPDICE', 'MPDLIQ', 'MPDQ', 'MPDT', 
'OCNFRAC', 'OMEGA', 'OMEGA500', 'PBLH', 'PHIS', 'PINT', 'PMID', 'PRECC', 'PRECL', 'PRECSC', 'PRECSL', 'PRECT', 'PS', 'PSL', 'PTEQ', 'PTTEND', 'Q', 
'QFLX', 'QRL', 'QRS', 'QTGW', 'RCMTEND_CLUBB', 'RELHUM', 'RVMTEND_CLUBB', 'SHFLX', 'SOLIN', 'SST', 'STEND_CLUBB', 'SWCF', 
'T', 'TAUX', 'TAUY', 'TFIX', 'TGCLDIWP', 'TGCLDLWP', 'TMQ', 'TREFHT', 'TS', 'TTGW', 'U', 'U10', 'UBOT', 'UTGWORO', 'UTGW_TOTAL', 
'V', 'VBOT', 'VTGWORO', 'VTGW_TOTAL', 'WPRTP_CLUBB', 'WPTHLP_CLUBB', 'Z3', 'ZMDQ', 'ZMDT', 'N2O', 'CO2','CFC11','CFC12', 
'AODVISdn','CCN3', 'CDNUMC', 'H2O', 'NUMICE', 'NUMLIQ','OMEGA500'
 'AQSO4_H2O2','AQSO4_O3', 'bc_a1', 'bc_a4', 'dst_a1', 'dst_a2', 'dst_a3', 'ncl_a1',
'ncl_a1', 'ncl_a2', 'ncl_a3', 'pom_a1', 'pom_a4', 'so4_a1', 'so4_a2', 'so4_a3', 'soa_a2' ,
'soa_a1', 'num_a1', 'num_a2', 'num_a3', 'num_a4',
'bc_a1SFWET', 'bc_a4SFWET', 'dst_a1SFWET', 'dst_a2SFWET', 'dst_a3SFWET', 'ncl_a1SFWET',
'ncl_a2SFWET', 'ncl_a3SFWET', 'pom_a1SFWET', 'pom_a4SFWET', 'so4_a1SFWET', 'so4_a2SFWET', 'so4_a3SFWET', 'soa_a1SFWET',
'soa_a2SFWET', 'bc_c1SFWET', 'bc_c4SFWET', 'dst_c1SFWET', 'dst_c2SFWET', 'dst_c3SFWET', 'ncl_c1SFWET', 'ncl_c2SFWET',
'ncl_c3SFWET', 'pom_c1SFWET', 'pom_c4SFWET', 'so4_c1SFWET', 'so4_c2SFWET', 'so4_c3SFWET', 'soa_c1SFWET', 'soa_c2SFWET',
'bc_a1DDF', 'bc_a4DDF', 'dst_a1DDF', 'dst_a2DDF', 'dst_a3DDF', 'ncl_a1DDF', 'ncl_a2DDF', 'ncl_a3DDF',
'pom_a1DDF', 'pom_a4DDF', 'so4_a1DDF', 'so4_a2DDF', 'so4_a3DDF', 'soa_a1DDF', 'soa_a2DDF',
'so4_a1_CLXF', 'so4_a2_CLXF', 'SFbc_a4', 'SFpom_a4', 'SFso4_a1', 'SFso4_a2',
'so4_a1_sfgaex1', 'so4_a2_sfgaex1', 'so4_a3_sfgaex1', 'soa_a1_sfgaex1', 'soa_a2_sfgaex1',
'SFdst_a1','SFdst_a2', 'SFdst_a3', 'SFncl_a1', 'SFncl_a2', 'SFncl_a3',
'num_a2_sfnnuc1', 'SFSO2', 'OCN_FLUX_DMS', 'SAD_SULFC', 'SAD_TROP', 'SAD_AERO'

fincl3 = 'PRECT', 'PRECC', 'FLUT', 'U850', 'U200', 'V850', 'V200', 'OMEGA500', 'TS', 'SST', 'PSL'

fincl4 =  'PRECC','PRECL'

fincl5 = 'Uzm','Vzm','Wzm','THzm', 'VTHzm','WTHzm','UVzm','UWzm'
phys_grid_ctem_nfreq=-6
phys_grid_ctem_zm_nbas=120
phys_grid_ctem_za_nlat=90

 rad_climate        = 'A:Q:H2O', 'N:O2:O2',
         'A:CO2:CO2', 'N:ozone:O3',
         'A:N2O:N2O', 'A:CH4:CH4',
         'N:CFC11STAR:CFC11', 'A:CFC12:CFC12',
         'M:mam4_mode1:/glade/campaign/cesm/cesmdata/inputdata/atm/cam/physprops/mam4_mode1_rrtmg_aeronetdust_sig1.6_dgnh.48_c140304.nc', 'M:mam4_mode2:/glade/campaign/cesm/cesmdata/inputdata/atm/cam/physprops/mam4_mode2_rrtmg_aitkendust_c141106.nc',
         'M:mam4_mode3:/glade/campaign/cesm/cesmdata/inputdata/atm/cam/physprops/mam4_mode3_rrtmg_aeronetdust_c141106.nc', 'M:mam4_mode4:/glade/campaign/cesm/cesmdata/inputdata/atm/cam/physprops/mam4_mode4_rrtmg_c130628.nc',
         'N:VOLC_MMR1:/glade/campaign/cesm/cesmdata/inputdata/atm/cam/physprops/volc_camRRTMG_byradius_sigma1.6_mode1_c210211.nc', 'N:VOLC_MMR2:/glade/campaign/cesm/cesmdata/inputdata/atm/cam/physprops/volc_camRRTMG_byradius_sigma1.6_mode2_c210211.nc',
         'N:VOLC_MMR3:/glade/campaign/cesm/cesmdata/inputdata/atm/cam/physprops/volc_camRRTMG_byradius_sigma1.2_mode3_c210211.nc'

 seasalt_emis_scale             = 0.75D0

clubb_c8        =  4.95 

se_sponge_del4_lev = 4
se_sponge_del4_nu_fac = 3.375
se_sponge_del4_nu_div_fac = 3.375

ncdata = '/glade/campaign/cesm/cesmdata/inputdata/atm/cam/inic/se/c153_ne30pg3_FMTHIST_x02.cam.i.1990-01-01-00000_c240618.nc'

user_nl_cice

dumpfreq = 'x','x','x','x','x'
f_fresh = 'mxxxx'
f_fresh_ai = 'mxxxx'

user_nl_cpl
```
histaux_l2x1yrg = .true
```

-user_nl_mom

MIN_SALINITY = 1.e-06
INPUTDIR = "INPUT"
MAX_LAYER_THICKNESS_CONFIG = "FILE:dz_max_90th_quantile.nc,dz"
USE_CFC_CAP = True
MAXTRUNC = 100
EOF

Case directory:

Locally (if still available): /glade/campaign/cesm/cesmdata/cseg/runs/cesm2_0/b.e30_beta02.BMT1850.ne30_t232.104
On github: https://github.com/NCAR/amwg_dev/tree/b.e30_beta02.BMT1850.ne30_t232.104

Sandbox:
Locally (if still available): /glade/work/hannay/cesm_tags/cesm3_0_beta02
On github: https://github.com/ESCOMP/cesm/tree/cesm3_0_beta02

Diagnostics:
Diags (if available): https://webext.cgd.ucar.edu/BMT1850/b.e30_beta02.BMT1850.ne30_t232.104/

Output:
Raw output on derecho (if still available): /glade/derecho/scratch/hannay/archive/b.e30_beta02.BMT1850.ne30_t232.104/
CAM Climos: /glade/campaign/cgd/amp/amwg/climo/b.e30_beta02.BMT1850.ne30_t232.104

Contacts: @cecilehannay, @PeterHjortLauritzen, @adamrher, @brianpm, @JulioTBacmeister, @swrneale, @gustavo-marques, @fobryan3, @klindsay28, @dabail10, @wwieder, @slevis-lmwg, @olyson @tilmes, @dlawrenncar

Extra details:

cecilehannay commented 3 months ago

This run is crashing after 40 days after sending the warming and error:

WARNING from PE   404: Extreme surface sfc_state detected: i= 462 j= 480 lon=  72.616 lat=  70.815 x=  20.667 y=  87.741 D= \
1.3316E+01 SSH= 9.3476E+00 SST= 2.2799E-01 SSS=-7.1163E-30 U-= 9.5591E-04 U+= 3.7041E- FATAL from PE   404: There were a total of         1 locations detected with extreme surface values!

The crash is in the subroutine:

dec2279.hsn.de.hpc.ucar.edu 5725: cesm.exe           00000000038338F9  mpp_mod_mp_mpp_er          72  mpp_util_mpi.inc
dec2279.hsn.de.hpc.ucar.edu 5725: cesm.exe           00000000029BEED8  mom_error_handler         191  MOM_error_handler.F90
dec2279.hsn.de.hpc.ucar.edu 5725: cesm.exe           0000000002719575  mom_mp_extract_su        3982  MOM.F90
dec2279.hsn.de.hpc.ucar.edu 5725: cesm.exe           0000000002702FD5  mom_mp_step_mom_         1033  MOM.F90
dec2279.hsn.de.hpc.ucar.edu 5725: cesm.exe           00000000026D6DF9  mom_ocean_model_n         630  mom_ocean_model_nuopc.F90
dec2279.hsn.de.hpc.ucar.edu 5725: cesm.exe           000000000269D25E  mom_cap_mod_mp_mo        1759  mom_cap.F90

dabail10 commented 3 months ago

Looks like salinity is going negative. I thought we had a minimum salinity?

cecilehannay commented 3 months ago

Thanks, dabail10. Good catch: I had a typo in user_nl_mom. I resubmitted.

adamrher commented 3 months ago

This run looks pretty darn close to 104, with all changes primarily in the UTLS region and not really impacting the TOA radiative balance. I say extend this run.

jedwards4b commented 3 months ago

Here is an improved PE layout for this case: cp /glade/derecho/scratch/jedwards/PFS.ne30pg3_t232.BMT1850.derecho_intel.allactive-defaultio.20240815_082316_ek8nr4/env_mach_pes.xml.5678404.desched1.240815-084052 /glade/campaign/cesm/cesmdata/cseg/runs/cesm2_0/b.e30_beta02.BMT1850.ne30_t232.104/env_mach_pes.xml

Should improve performance and lower cost a little bit.

Model Cost: 10042.01 pe-hrs/simulated_year Model Throughput: 14.07 simulated_years/day

NCAR / amwg_dev

b.e30_beta02.BMT1850.ne30_t232.104 #574