NCAR / LMWG_dev

Repository to track LMWG development simulations
3 stars 0 forks source link

ctsm5.3 f19 spinup and historical for PPE #70

Open linniahawkins opened 1 month ago

linniahawkins commented 1 month ago

Description: For the CLM6-PPE, we need a 2-deg f19 full grid cold start spinup.

We will start from these initial conditions and run our own spin-up protocol for each ensemble member.

For reference, this is the restart we used in the CLM5-PPE, but it's been moved and I don't know where it came from: '/glade/p/cgd/tss/people/oleson/CLM5_restarts/clm51_PPEn02ctsm51d021_2deg_GSWP3V1_leafbiomassesai_PPE3_1850pAD.clm2.r.2041-01-01-00000.nc'

NOTE: CLM6-PPE is actually a ctsm5.3.0 PPE. It's okay if this spinup is done with a 5.2.02* tag, as long as it is close enough.

Timeframe: We would like to be able to start testing with the new restart by ~Oct.1 so we can kick off a full PPE by Oct. 15


Case directory: Locally: /glade/work/oleson/ctsm5.3.0/cime/scripts/ctsm530_f19_PPE_AD /glade/work/oleson/ctsm5.3.0/cime/scripts/ctsm530_f19_PPE_SASU /glade/work/oleson/ctsm5.3.0/cime/scripts/ctsm530_f19_PPE_pSASU /glade/work/oleson/ctsm5.3.0/cime/scripts/ctsm530_f19_PPE_hist


Sandbox: Locally: /glade/work/oleson/ctsm5.3.0


Case modifications

AD: user_nl_clm

! The 78pft version comes out of the box with this setup, so point explicitly to a 16pft version
fsurdat = '/glade/derecho/scratch/slevis/temp_work/new_rawdata/tools/mksurfdata_esmf/surfdata_1.9x2.5_hist_1850_16pfts_c240926.nc'

hist_fincl1 = 'TOTECOSYSC', 'TOTECOSYSN', 'TOTSOMC', 'TOTSOMN', 'TOTVEGC', 'TOTVEGN', 'TLAI', 'GPP', 'CPOOL', 'NPP', 'TWS','H2OSNO'

! turn off shifting cultivation
do_grossunrep = .false.

xml changes

./xmlchange MOSART_MODE=NULL
./xmlchange STOP_OPTION=nyears
./xmlchange STOP_N=40
./xmlchange REST_N=10
./xmlchange DOUT_S_SAVE_INTERIM_RESTART_FILES=TRUE
./xmlchange CLM_ACCELERATED_SPINUP=on
./xmlchange CLM_FORCE_COLDSTART=on
./xmlchange RESUBMIT=6

SASU, created from clone of AD case user_nl_clm

finidat = '/glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm530_f19_PPE_AD.clm2.r.0281-01-01-00000.nc'
reseed_dead_plants = .true.

! The 78pft version comes out of the box with this setup, so point explicitly to a 16pft version
fsurdat = '/glade/derecho/scratch/slevis/temp_work/new_rawdata/tools/mksurfdata_esmf/surfdata_1.9x2.5_hist_1850_16pfts_c240926.nc'

hist_fincl1 = 'TOTECOSYSC', 'TOTECOSYSN', 'TOTSOMC', 'TOTSOMN', 'TOTVEGC', 'TOTVEGN', 'TLAI', 'GPP', 'CPOOL', 'NPP', 'TWS','H2OSNO'

! turn off shifting cultivation
do_grossunrep = .false.

xml changes

./xmlchange RUN_TYPE=hybrid
./xmlchange CLM_ACCELERATED_SPINUP=sasu
./xmlchange CLM_FORCE_COLDSTART=off
./xmlchange RESUBMIT=6

pSASU, created from clone of pSASU case user_nl_clm

finidat = '/glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm530_f19_PPE_SASU.clm2.r.0281-01-01-00000.nc'
reseed_dead_plants = .true.

! The 78pft version comes out of the box with this setup, so point explicitly to a 16pft version
fsurdat = '/glade/derecho/scratch/slevis/temp_work/new_rawdata/tools/mksurfdata_esmf/surfdata_1.9x2.5_hist_1850_16pfts_c240926.nc'

! turn off shifting cultivation
do_grossunrep = .false.

xml changes

./xmlchange CLM_ACCELERATED_SPINUP=off
./xmlchange RESUBMIT=3

hist:

finidat = '/glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm530_f19_PPE_pSASU.clm2.r.0161-01-01-00000.nc'

! The 78pft version comes out of the box with this setup, so point explicitly to a 16pft version
fsurdat = '/glade/derecho/scratch/slevis/temp_work/new_rawdata/tools/mksurfdata_esmf/surfdata_1.9x2.5_hist_1850_16pfts_c240926.nc'
flanduse_timeseries = '/glade/derecho/scratch/slevis/temp_work/new_rawdata/tools/mksurfdata_esmf/landuse.timeseries_1.9x2.5_hist_1850-2023_16pfts_c240926.nc'

! turn off shifting cultivation
do_grossunrep = .false.

ndep_taxmode = 'extend'

hist_fincl1 += 'FSDSND','FSDSVD','FSDSNI','FSDSVI','RAIN_FROM_ATM','SNOW_FROM_ATM','FLDS_NOT_DOWNSCALED','ZBOT','Tair_from_atm','Thair_from_atm','QBOT_NOT_DOWNSCALED','PBOT_NOT_DOWNSCALED','Rho_from_atm','PCO2','UWIND','VWIND','BCPHIDRY','BCPHODRY','BCPHIWET','OCPHIDRY','OCPHODRY','OCPHIWET','DSTWET1','DSTDRY1','DSTWET2','DSTDRY2','DSTWET3','DSTDRY3','DSTWET4','DSTDRY4','ATM_TOPO'

use_init_interp = .true.

! Generated from ./create_newcase ..... --user-mods-dir output_crop_highfreq and edited for only h0,h1,h2,h3,h4

!----------------------------------------------------------------------------------
! Settings from output_base
!----------------------------------------------------------------------------------

! h0 stream (monthly average, gridcell-level)
! h0 stream ICE variables and other variables inactive by default (monthly average, gridcell-level)
! Emon, Lmon, LIMon
hist_mfilt(1) = 1
hist_dov2xy(1) = .true.
hist_nhtfrq(1) = 0
hist_type1d_pertape(1) = ' '
!hist_fexcl1 += 'PCT_GLC_MEC','PCT_NAT_PFT'
hist_fexcl1 += 'PCT_GLC_MEC'
hist_fincl1 += 'EFLX_LH_TOT_ICE', 'FIRE_ICE', 'FLDS_ICE', 'FSH_ICE', 'FSNO_ICE', 'FSR_ICE', 'QFLX_SOLIDEVAP_FROM_TOP_LAYER_ICE', 'QRUNOFF_ICE', 'QSNOFRZ_ICE', 'QSNOMELT_ICE', 'RAIN_ICE', 'SNOW_ICE', 'SNOWICE_ICE', 'SNOWLIQ_ICE', 'SNOTXMASS_ICE', 'TG_ICE', 'TOPO_COL_ICE', 'TSA_ICE', 'TSOI_ICE'

! h1 stream (monthly average, finest sub-grid)
! Emon, Lmon
hist_mfilt(2) = 1
hist_dov2xy(2) = .false.
hist_nhtfrq(2) = 0
hist_type1d_pertape(2) = ' '
hist_fincl2 += 'TLAI', 'TSA','TREFMNAV','TREFMXAV', 'BTRANMN', 'Vcmx25Z', 'FSH', 'VEGWP', 'FCTR', 'FCEV', 'FGEV', 'FIRE', 'FSR', 'FIRA', 'FSA', 'GSSUNLN', 'GSSHALN', 'TSKIN'

! h2 stream (monthly average, landunit-level)
! Emon
hist_mfilt(3) = 1
hist_dov2xy(3) = .false.
hist_nhtfrq(3) = 0
hist_type1d_pertape(3) = 'LAND'
hist_fincl3 += 'FSR', 'H2OSNO', 'Q2M', 'SNOWDP', 'TSA', 'TREFMNAV', 'TREFMXAV', 'TG', 'QRUNOFF', 'FSH', 'FIRE', 'FIRA', 'FGR', 'EFLX_LH_TOT', 'RH2M', 'TLAI', 'SOILWATER_10CM', 'TOTSOILLIQ', 'TOTSOILICE', 'U10', 'TSOI_10CM', 'QIRRIG_DRIP', 'URBAN_HEAT', 'WASTEHEAT', 'TSKIN'

! h3 stream (yearly average, gridcell-level)
! Eyr
hist_mfilt(4) = 1
hist_dov2xy(4) = .true.
hist_nhtfrq(4) = -8760
hist_type1d_pertape(4) = ' '
hist_fincl4 += 'PCT_GLC_MEC', 'QICE_FORC', 'TSRF_FORC', 'TOPO_FORC', 'PCT_NAT_PFT','PCT_LANDUNIT','FSNO_ICE'

! h4 stream (yearly average, landunit-level)
! Eyr landuse tiles
! (No fields in base, but other usermods add to this stream)
hist_mfilt(5) = 1
hist_dov2xy(5) = .false.
hist_nhtfrq(5) = -8760
hist_type1d_pertape(5) = 'LAND'

!----------------------------------------------------------------------------------
! Settings from output_bgc
!----------------------------------------------------------------------------------

! h0 stream (monthly average, gridcell-level)
hist_fexcl1 += 'SOM_ACT_C_vr', 'SOM_ACT_N_vr', 'SOM_SLO_C_vr', 'SOM_SLO_N_vr', 'SOM_PAS_C_vr', 'SOM_PAS_N_vr', 'SOILC_vr', 'SOILN_vr', 'CWD_C_vr', 'LIT_MET_C_vr', 'LIT_CEL_C_vr', 'LIT_LIG_C_vr', 'LIT_MET_N_vr', 'LIT_CEL_N_vr', 'LIT_LIG_C_vr', 'CWD_N_vr', 'SMIN_NO3_vr', 'CONC_O2_UNSAT', 'CONC_O2_SAT','SMIN_NH4_vr','SMINN_vr'
hist_fincl1 += 'LEAFC_TO_LITTER', 'FROOTC_TO_LITTER','LIT_MET_C_TO_SOM_ACT_C','LIT_MET_N_TO_SOM_ACT_N','LIT_CEL_C_TO_SOM_ACT_C', 'LIT_CEL_N_TO_SOM_ACT_N','LIT_LIG_C_TO_SOM_SLO_C','LIT_LIG_N_TO_SOM_SLO_N','DWT_WOOD_PRODUCTC_GAIN_PATCH'

! h1 stream (monthly average, finest sub-grid)
hist_fincl2 += 'GPP', 'NPP', 'AGNPP', 'TOTVEGC', 'NPP_NUPTAKE', 'AR', 'HR', 'HTOP'

! h2 stream (monthly average, landunit-level)
! TOT_WOODPRODC:I, and NEE are not available at the landunit level
hist_fincl3 += 'GPP', 'NPP', 'AR', 'HR', 'DWT_CONV_CFLUX_PATCH', 'WOOD_HARVESTC', 'DWT_WOOD_PRODUCTC_GAIN_PATCH', 'SLASH_HARVESTC', 'COL_FIRE_CLOSS', 'DWT_SLASH_CFLUX_PATCH', 'FROOTC:I', 'HTOP'

! h3 stream (yearly average, gridcell-level)
hist_fincl4 += 'SOILC_vr', 'SOILN_vr', 'CWD_C_vr', 'LIT_MET_C_vr', 'LIT_CEL_C_vr', 'LIT_LIG_C_vr', 'LIT_MET_N_vr', 'LIT_CEL_N_vr', 'LIT_LIG_N_vr','CWD_N_vr', 'TOTLITC:I', 'TOT_WOODPRODC:I', 'TOTSOMC:I','TOTVEGC:I'

! h4 stream (yearly average, landunit-level)
hist_fincl5 += 'TOTSOMC:I', 'TOTSOMC_1m:I', 'TOTECOSYSC:I', 'TOTVEGC:I', 'WOODC:I', 'TOTLITC:I', 'LIVECROOTC:I', 'DEADCROOTC:I', 'FROOTC:I'

!----------------------------------------------------------------------------------
! Settings from output_crop
!----------------------------------------------------------------------------------

! h0 stream (monthly average, gridcell-level)
hist_fexcl1 += 'PCT_CFT'

! h3 stream (yearly average, gridcell-level)
hist_fincl4 += 'PCT_CFT'

user_nl_datm_streams:

CLMCRUNCEPv7.Solar:datafiles=/glade/derecho/scratch/wwieder/TRENDY2024/inputs/three_stream/clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc, ...through 2023
CLMCRUNCEPv7.Solar:meshfile=/glade/derecho/scratch/slevis/temp_work/TRENDY2024/inputs/three_stream/mesh_cdf5.nc
CLMCRUNCEPv7.Solar:datavars= FSDS Faxa_swdn

CLMCRUNCEPv7.Precip:datafiles=/glade/derecho/scratch/wwieder/TRENDY2024/inputs/three_stream/clmforc.CRUJRAv2.5_0.5x0.5.Prec.1901.nc, ...through 2023
CLMCRUNCEPv7.Precip:meshfile=/glade/derecho/scratch/slevis/temp_work/TRENDY2024/inputs/three_stream/mesh_cdf5.nc

CLMCRUNCEPv7.TPQW:datafiles=/glade/derecho/scratch/wwieder/TRENDY2024/inputs/three_stream/clmforc.CRUJRAv2.5_0.5x0.5.TPQWL.1901.nc, ...through 2023
CLMCRUNCEPv7.TPQW:meshfile=/glade/derecho/scratch/slevis/temp_work/TRENDY2024/inputs/three_stream/mesh_cdf5.nc
CLMCRUNCEPv7.TPQW:datavars = QBOT Sa_shum,PSRF Sa_pbot,TBOT Sa_tbot,WIND Sa_wind,FLDS Faxa_lwdn

SourceMods:

/glade/work/oleson/ctsm5.3.0/cime/scripts/ctsm530_f19_PPE_AD/SourceMods/src.clm/CanopyFluxesMod.F90 (sa_leaf = sa_leaf + esai; see PR#2788)


Diagnostics: Diags (if available)

Standard: [https://webext.cgd.ucar.edu/I1850/$CASE/lnd/](https://webext.cgd.ucar.edu/I1850/$CASE/lnd/) ILAMB: https://webext.cgd.ucar.edu/I20TR/ctsm530_f19_PPE_hist/lnd/_build_ctsm530_f19_PPE_hist/


Output: Output (if still available): /glade/scratch/oleson/archive/ctsm530_f19_PPE_AD/ /glade/scratch/oleson/archive/ctsm530_f19_PPE_SASU/ /glade/scratch/oleson/archive/ctsm530_f19_PPE_pSASU/ /glade/scratch/oleson/archive/ctsm530_f19_PPE_hist/


Contacts: @linniahawkins @olyson


Extra details: AD: 280 yrs, 2.08% disequilibrium SASU: 280 yrs, 13.73% disequilibrium pSASU: 160 yrs, 2.15% disequilibrium hist: 1850-2023

wwieder commented 1 month ago

Thanks for creating this issue, Linnia. I'd suggest we use the same set up as in #69, but with f19. Maybe it makes sense for us to look at 69 first (at least in 1850) to make sure we don't need any quick fixes or additional adjustments?

Alternatively we can at least do this in the ctsm5.3 code base when it's tagged next week? What do you think @slevis-lmwg and @olyson?

slevis-lmwg commented 1 month ago

Alternatively we can at least do this in the ctsm5.3 code base when it's tagged next week? What do you think @slevis-lmwg and @olyson?

At the moment either seems fine, since #69 is using the latest ctsm5.3 tag (branch_tags/ctsm5.3.n04_ctsm5.2.028), though we should first have resolution regarding Keith's latest posts in issue 2567.

olyson commented 1 month ago

The f09 and ne30 spinups should be in their post-SASU phase by this afternoon, so we could probably start looking at results tomorrow afternoon. As I posted in https://github.com/ESCOMP/CTSM/issues/2567, the differences in CNVegStructMod.F90 code were bfb, and the changes to sa_leaf in CanopyFluxesMod.F90 resulted in small differences, at least in I2000 cases as I posted in https://github.com/ESCOMP/CTSM/issues/2777.

wwieder commented 1 month ago

Is it worth touching base on this at some point on Thursday? 930 or 100 times are better for me

olyson commented 1 month ago

Looks like we have a CTSM5.3 tag now, so it seems like we could start this. Could include the sa_leaf change as a SourceMod. One question I have is that our approach last time was to run without the crop model on since we weren't changing crop parameters. But we weren't tuning either. So I assume we'll run with the crop model on...? I can meet Thurs, but we have CTSM software meeting at 9:30?

wwieder commented 1 month ago

Good idea on the sa_leaf issue, @olyson. I'll let @linniahawkins or @djk2120 weigh in here if they want the crop model on for the 2 degree spinup?

olyson commented 1 month ago

A couple of other questions:

  1. No need to have isotopes on right?
  2. We have been setting do_grossunrep = .false. (turn off shifting cultivation). Do the same here?
wwieder commented 1 month ago

@slevis-lmwg let's make a 16PFT surface dataset at the f19 resolution in 1850 + a historical simulation.

slevis-lmwg commented 1 month ago

I submitted the job to generate new fsurdat/landuse files earlier this afternoon. (Issue https://github.com/ESCOMP/CTSM/issues/2791)

olyson commented 1 month ago

Crops off (use 16pft dataset).

slevis-lmwg commented 1 month ago

The files are ready (again, details in https://github.com/ESCOMP/CTSM/issues/2791)

olyson commented 1 month ago

I think this is ready to start (beginning with the AD). But it might be good for anyone who is interested to check the setup: /glade/work/oleson/ctsm5.3.0/cime/scripts/ctsm530_f19_PPE_AD @wwieder @slevis-lmwg @linniahawkins @djk2120

djk2120 commented 1 month ago

everything matches my expectations

djk2120 commented 1 month ago

is it still appropriate to include: reseed_dead_plants = .true.

edit: I can see that this is indeed the setting

slevis-lmwg commented 1 month ago

My understanding is: yes.

slevis-lmwg commented 1 month ago

Looks good to me.

olyson commented 1 month ago

reseed_dead_plants is set to .true. by default, but in a cold start (as this AD is) I don't think it actually does anything.

olyson commented 1 month ago

This spinup is complete. The restart file is:

/glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm530_f19_PPE_pSASU.clm2.r.0161-01-01-00000.nc

I'm running a branch now to look at survivability. Pending that looking ok, I plan to proceed with the historical.

olyson commented 1 month ago

ctsm530_f19_PPE_fullout_pSASU_pft_2dPDF_whittaker-0

ctsm530_f19_PPE_fullout_pSASU_pft_2dPDF_whittaker-1

linniahawkins commented 1 month ago

Thanks Everyone! I am testing this restart in our PPE workflow now. @olyson can you change the permissions on this file: /glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm530_f19_PPE_pSASU.clm2.r.0161-01-01-00000.nc

olyson commented 1 month ago

Ok, done.

djk2120 commented 2 weeks ago

@olyson @wwieder Would it be possible to continue this simulation through the full historical? It would be really helpful for assessing if our old sparsegrid still works, and for training a new sparsegrid if not.

wwieder commented 2 weeks ago

Seems reasonable. Can you kick off a historical run, Keith?

olyson commented 2 weeks ago

Yes, I'll start that up.

olyson commented 2 weeks ago

The historical is done. The output is here:

/glade/derecho/scratch/oleson/archive/ctsm530_f19_PPE_hist

I'll run some diagnostics when I'm back in the office.

linniahawkins commented 2 weeks ago

Great, thank you Keith!

olyson commented 1 week ago

Survivability and max TLAI from the historical:

ctsm530_f19_PPE_hist_pft_2dPDF_whittaker-0

ctsm530_f19_PPE_hist_pft_2dPDF_whittaker-1