NCAR / LMWG_dev

Repository to track LMWG development simulations
3 stars 0 forks source link

ctsm5.3 f19 spinup and historical for PPE #70

Open linniahawkins opened 2 months ago

linniahawkins commented 2 months ago

Description: For the CLM6-PPE, we need a 2-deg f19 full grid cold start spinup.

We will start from these initial conditions and run our own spin-up protocol for each ensemble member.

For reference, this is the restart we used in the CLM5-PPE, but it's been moved and I don't know where it came from: '/glade/p/cgd/tss/people/oleson/CLM5_restarts/clm51_PPEn02ctsm51d021_2deg_GSWP3V1_leafbiomassesai_PPE3_1850pAD.clm2.r.2041-01-01-00000.nc'

NOTE: CLM6-PPE is actually a ctsm5.3.0 PPE. It's okay if this spinup is done with a 5.2.02* tag, as long as it is close enough.

Timeframe: We would like to be able to start testing with the new restart by ~Oct.1 so we can kick off a full PPE by Oct. 15


Case directory: Locally: /glade/work/oleson/ctsm5.3.0/cime/scripts/ctsm530_f19_PPE_AD /glade/work/oleson/ctsm5.3.0/cime/scripts/ctsm530_f19_PPE_SASU /glade/work/oleson/ctsm5.3.0/cime/scripts/ctsm530_f19_PPE_pSASU /glade/work/oleson/ctsm5.3.0/cime/scripts/ctsm530_f19_PPE_hist


Sandbox: Locally: /glade/work/oleson/ctsm5.3.0


Case modifications

AD: user_nl_clm

! The 78pft version comes out of the box with this setup, so point explicitly to a 16pft version
fsurdat = '/glade/derecho/scratch/slevis/temp_work/new_rawdata/tools/mksurfdata_esmf/surfdata_1.9x2.5_hist_1850_16pfts_c240926.nc'

hist_fincl1 = 'TOTECOSYSC', 'TOTECOSYSN', 'TOTSOMC', 'TOTSOMN', 'TOTVEGC', 'TOTVEGN', 'TLAI', 'GPP', 'CPOOL', 'NPP', 'TWS','H2OSNO'

! turn off shifting cultivation
do_grossunrep = .false.

xml changes

./xmlchange MOSART_MODE=NULL
./xmlchange STOP_OPTION=nyears
./xmlchange STOP_N=40
./xmlchange REST_N=10
./xmlchange DOUT_S_SAVE_INTERIM_RESTART_FILES=TRUE
./xmlchange CLM_ACCELERATED_SPINUP=on
./xmlchange CLM_FORCE_COLDSTART=on
./xmlchange RESUBMIT=6

SASU, created from clone of AD case user_nl_clm

finidat = '/glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm530_f19_PPE_AD.clm2.r.0281-01-01-00000.nc'
reseed_dead_plants = .true.

! The 78pft version comes out of the box with this setup, so point explicitly to a 16pft version
fsurdat = '/glade/derecho/scratch/slevis/temp_work/new_rawdata/tools/mksurfdata_esmf/surfdata_1.9x2.5_hist_1850_16pfts_c240926.nc'

hist_fincl1 = 'TOTECOSYSC', 'TOTECOSYSN', 'TOTSOMC', 'TOTSOMN', 'TOTVEGC', 'TOTVEGN', 'TLAI', 'GPP', 'CPOOL', 'NPP', 'TWS','H2OSNO'

! turn off shifting cultivation
do_grossunrep = .false.

xml changes

./xmlchange RUN_TYPE=hybrid
./xmlchange CLM_ACCELERATED_SPINUP=sasu
./xmlchange CLM_FORCE_COLDSTART=off
./xmlchange RESUBMIT=6

pSASU, created from clone of pSASU case user_nl_clm

finidat = '/glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm530_f19_PPE_SASU.clm2.r.0281-01-01-00000.nc'
reseed_dead_plants = .true.

! The 78pft version comes out of the box with this setup, so point explicitly to a 16pft version
fsurdat = '/glade/derecho/scratch/slevis/temp_work/new_rawdata/tools/mksurfdata_esmf/surfdata_1.9x2.5_hist_1850_16pfts_c240926.nc'

! turn off shifting cultivation
do_grossunrep = .false.

xml changes

./xmlchange CLM_ACCELERATED_SPINUP=off
./xmlchange RESUBMIT=3

hist:

finidat = '/glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm530_f19_PPE_pSASU.clm2.r.0161-01-01-00000.nc'

! The 78pft version comes out of the box with this setup, so point explicitly to a 16pft version
fsurdat = '/glade/derecho/scratch/slevis/temp_work/new_rawdata/tools/mksurfdata_esmf/surfdata_1.9x2.5_hist_1850_16pfts_c240926.nc'
flanduse_timeseries = '/glade/derecho/scratch/slevis/temp_work/new_rawdata/tools/mksurfdata_esmf/landuse.timeseries_1.9x2.5_hist_1850-2023_16pfts_c240926.nc'

! turn off shifting cultivation
do_grossunrep = .false.

ndep_taxmode = 'extend'

hist_fincl1 += 'FSDSND','FSDSVD','FSDSNI','FSDSVI','RAIN_FROM_ATM','SNOW_FROM_ATM','FLDS_NOT_DOWNSCALED','ZBOT','Tair_from_atm','Thair_from_atm','QBOT_NOT_DOWNSCALED','PBOT_NOT_DOWNSCALED','Rho_from_atm','PCO2','UWIND','VWIND','BCPHIDRY','BCPHODRY','BCPHIWET','OCPHIDRY','OCPHODRY','OCPHIWET','DSTWET1','DSTDRY1','DSTWET2','DSTDRY2','DSTWET3','DSTDRY3','DSTWET4','DSTDRY4','ATM_TOPO'

use_init_interp = .true.

! Generated from ./create_newcase ..... --user-mods-dir output_crop_highfreq and edited for only h0,h1,h2,h3,h4

!----------------------------------------------------------------------------------
! Settings from output_base
!----------------------------------------------------------------------------------

! h0 stream (monthly average, gridcell-level)
! h0 stream ICE variables and other variables inactive by default (monthly average, gridcell-level)
! Emon, Lmon, LIMon
hist_mfilt(1) = 1
hist_dov2xy(1) = .true.
hist_nhtfrq(1) = 0
hist_type1d_pertape(1) = ' '
!hist_fexcl1 += 'PCT_GLC_MEC','PCT_NAT_PFT'
hist_fexcl1 += 'PCT_GLC_MEC'
hist_fincl1 += 'EFLX_LH_TOT_ICE', 'FIRE_ICE', 'FLDS_ICE', 'FSH_ICE', 'FSNO_ICE', 'FSR_ICE', 'QFLX_SOLIDEVAP_FROM_TOP_LAYER_ICE', 'QRUNOFF_ICE', 'QSNOFRZ_ICE', 'QSNOMELT_ICE', 'RAIN_ICE', 'SNOW_ICE', 'SNOWICE_ICE', 'SNOWLIQ_ICE', 'SNOTXMASS_ICE', 'TG_ICE', 'TOPO_COL_ICE', 'TSA_ICE', 'TSOI_ICE'

! h1 stream (monthly average, finest sub-grid)
! Emon, Lmon
hist_mfilt(2) = 1
hist_dov2xy(2) = .false.
hist_nhtfrq(2) = 0
hist_type1d_pertape(2) = ' '
hist_fincl2 += 'TLAI', 'TSA','TREFMNAV','TREFMXAV', 'BTRANMN', 'Vcmx25Z', 'FSH', 'VEGWP', 'FCTR', 'FCEV', 'FGEV', 'FIRE', 'FSR', 'FIRA', 'FSA', 'GSSUNLN', 'GSSHALN', 'TSKIN'

! h2 stream (monthly average, landunit-level)
! Emon
hist_mfilt(3) = 1
hist_dov2xy(3) = .false.
hist_nhtfrq(3) = 0
hist_type1d_pertape(3) = 'LAND'
hist_fincl3 += 'FSR', 'H2OSNO', 'Q2M', 'SNOWDP', 'TSA', 'TREFMNAV', 'TREFMXAV', 'TG', 'QRUNOFF', 'FSH', 'FIRE', 'FIRA', 'FGR', 'EFLX_LH_TOT', 'RH2M', 'TLAI', 'SOILWATER_10CM', 'TOTSOILLIQ', 'TOTSOILICE', 'U10', 'TSOI_10CM', 'QIRRIG_DRIP', 'URBAN_HEAT', 'WASTEHEAT', 'TSKIN'

! h3 stream (yearly average, gridcell-level)
! Eyr
hist_mfilt(4) = 1
hist_dov2xy(4) = .true.
hist_nhtfrq(4) = -8760
hist_type1d_pertape(4) = ' '
hist_fincl4 += 'PCT_GLC_MEC', 'QICE_FORC', 'TSRF_FORC', 'TOPO_FORC', 'PCT_NAT_PFT','PCT_LANDUNIT','FSNO_ICE'

! h4 stream (yearly average, landunit-level)
! Eyr landuse tiles
! (No fields in base, but other usermods add to this stream)
hist_mfilt(5) = 1
hist_dov2xy(5) = .false.
hist_nhtfrq(5) = -8760
hist_type1d_pertape(5) = 'LAND'

!----------------------------------------------------------------------------------
! Settings from output_bgc
!----------------------------------------------------------------------------------

! h0 stream (monthly average, gridcell-level)
hist_fexcl1 += 'SOM_ACT_C_vr', 'SOM_ACT_N_vr', 'SOM_SLO_C_vr', 'SOM_SLO_N_vr', 'SOM_PAS_C_vr', 'SOM_PAS_N_vr', 'SOILC_vr', 'SOILN_vr', 'CWD_C_vr', 'LIT_MET_C_vr', 'LIT_CEL_C_vr', 'LIT_LIG_C_vr', 'LIT_MET_N_vr', 'LIT_CEL_N_vr', 'LIT_LIG_C_vr', 'CWD_N_vr', 'SMIN_NO3_vr', 'CONC_O2_UNSAT', 'CONC_O2_SAT','SMIN_NH4_vr','SMINN_vr'
hist_fincl1 += 'LEAFC_TO_LITTER', 'FROOTC_TO_LITTER','LIT_MET_C_TO_SOM_ACT_C','LIT_MET_N_TO_SOM_ACT_N','LIT_CEL_C_TO_SOM_ACT_C', 'LIT_CEL_N_TO_SOM_ACT_N','LIT_LIG_C_TO_SOM_SLO_C','LIT_LIG_N_TO_SOM_SLO_N','DWT_WOOD_PRODUCTC_GAIN_PATCH'

! h1 stream (monthly average, finest sub-grid)
hist_fincl2 += 'GPP', 'NPP', 'AGNPP', 'TOTVEGC', 'NPP_NUPTAKE', 'AR', 'HR', 'HTOP'

! h2 stream (monthly average, landunit-level)
! TOT_WOODPRODC:I, and NEE are not available at the landunit level
hist_fincl3 += 'GPP', 'NPP', 'AR', 'HR', 'DWT_CONV_CFLUX_PATCH', 'WOOD_HARVESTC', 'DWT_WOOD_PRODUCTC_GAIN_PATCH', 'SLASH_HARVESTC', 'COL_FIRE_CLOSS', 'DWT_SLASH_CFLUX_PATCH', 'FROOTC:I', 'HTOP'

! h3 stream (yearly average, gridcell-level)
hist_fincl4 += 'SOILC_vr', 'SOILN_vr', 'CWD_C_vr', 'LIT_MET_C_vr', 'LIT_CEL_C_vr', 'LIT_LIG_C_vr', 'LIT_MET_N_vr', 'LIT_CEL_N_vr', 'LIT_LIG_N_vr','CWD_N_vr', 'TOTLITC:I', 'TOT_WOODPRODC:I', 'TOTSOMC:I','TOTVEGC:I'

! h4 stream (yearly average, landunit-level)
hist_fincl5 += 'TOTSOMC:I', 'TOTSOMC_1m:I', 'TOTECOSYSC:I', 'TOTVEGC:I', 'WOODC:I', 'TOTLITC:I', 'LIVECROOTC:I', 'DEADCROOTC:I', 'FROOTC:I'

!----------------------------------------------------------------------------------
! Settings from output_crop
!----------------------------------------------------------------------------------

! h0 stream (monthly average, gridcell-level)
hist_fexcl1 += 'PCT_CFT'

! h3 stream (yearly average, gridcell-level)
hist_fincl4 += 'PCT_CFT'

user_nl_datm_streams:

CLMCRUNCEPv7.Solar:datafiles=/glade/derecho/scratch/wwieder/TRENDY2024/inputs/three_stream/clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc, ...through 2023
CLMCRUNCEPv7.Solar:meshfile=/glade/derecho/scratch/slevis/temp_work/TRENDY2024/inputs/three_stream/mesh_cdf5.nc
CLMCRUNCEPv7.Solar:datavars= FSDS Faxa_swdn

CLMCRUNCEPv7.Precip:datafiles=/glade/derecho/scratch/wwieder/TRENDY2024/inputs/three_stream/clmforc.CRUJRAv2.5_0.5x0.5.Prec.1901.nc, ...through 2023
CLMCRUNCEPv7.Precip:meshfile=/glade/derecho/scratch/slevis/temp_work/TRENDY2024/inputs/three_stream/mesh_cdf5.nc

CLMCRUNCEPv7.TPQW:datafiles=/glade/derecho/scratch/wwieder/TRENDY2024/inputs/three_stream/clmforc.CRUJRAv2.5_0.5x0.5.TPQWL.1901.nc, ...through 2023
CLMCRUNCEPv7.TPQW:meshfile=/glade/derecho/scratch/slevis/temp_work/TRENDY2024/inputs/three_stream/mesh_cdf5.nc
CLMCRUNCEPv7.TPQW:datavars = QBOT Sa_shum,PSRF Sa_pbot,TBOT Sa_tbot,WIND Sa_wind,FLDS Faxa_lwdn

SourceMods:

/glade/work/oleson/ctsm5.3.0/cime/scripts/ctsm530_f19_PPE_AD/SourceMods/src.clm/CanopyFluxesMod.F90 (sa_leaf = sa_leaf + esai; see PR#2788)


Diagnostics: Diags (if available)

Standard: [https://webext.cgd.ucar.edu/I1850/$CASE/lnd/](https://webext.cgd.ucar.edu/I1850/$CASE/lnd/) ILAMB: https://webext.cgd.ucar.edu/I20TR/ctsm530_f19_PPE_hist/lnd/_build_ctsm530_f19_PPE_hist/


Output: Output (if still available): /glade/scratch/oleson/archive/ctsm530_f19_PPE_AD/ /glade/scratch/oleson/archive/ctsm530_f19_PPE_SASU/ /glade/scratch/oleson/archive/ctsm530_f19_PPE_pSASU/ /glade/scratch/oleson/archive/ctsm530_f19_PPE_hist/

Timeseries files: /glade/derecho/scratch/oleson/timeseries/ctsm530_f19_PPE_hist/lnd/proc/tseries


Contacts: @linniahawkins @olyson


Extra details: AD: 280 yrs, 2.08% disequilibrium SASU: 280 yrs, 13.73% disequilibrium pSASU: 160 yrs, 2.15% disequilibrium hist: 1850-2023

wwieder commented 2 months ago

Thanks for creating this issue, Linnia. I'd suggest we use the same set up as in #69, but with f19. Maybe it makes sense for us to look at 69 first (at least in 1850) to make sure we don't need any quick fixes or additional adjustments?

Alternatively we can at least do this in the ctsm5.3 code base when it's tagged next week? What do you think @slevis-lmwg and @olyson?

slevis-lmwg commented 2 months ago

Alternatively we can at least do this in the ctsm5.3 code base when it's tagged next week? What do you think @slevis-lmwg and @olyson?

At the moment either seems fine, since #69 is using the latest ctsm5.3 tag (branch_tags/ctsm5.3.n04_ctsm5.2.028), though we should first have resolution regarding Keith's latest posts in issue 2567.

olyson commented 1 month ago

The f09 and ne30 spinups should be in their post-SASU phase by this afternoon, so we could probably start looking at results tomorrow afternoon. As I posted in https://github.com/ESCOMP/CTSM/issues/2567, the differences in CNVegStructMod.F90 code were bfb, and the changes to sa_leaf in CanopyFluxesMod.F90 resulted in small differences, at least in I2000 cases as I posted in https://github.com/ESCOMP/CTSM/issues/2777.

wwieder commented 1 month ago

Is it worth touching base on this at some point on Thursday? 930 or 100 times are better for me

olyson commented 1 month ago

Looks like we have a CTSM5.3 tag now, so it seems like we could start this. Could include the sa_leaf change as a SourceMod. One question I have is that our approach last time was to run without the crop model on since we weren't changing crop parameters. But we weren't tuning either. So I assume we'll run with the crop model on...? I can meet Thurs, but we have CTSM software meeting at 9:30?

wwieder commented 1 month ago

Good idea on the sa_leaf issue, @olyson. I'll let @linniahawkins or @djk2120 weigh in here if they want the crop model on for the 2 degree spinup?

olyson commented 1 month ago

A couple of other questions:

  1. No need to have isotopes on right?
  2. We have been setting do_grossunrep = .false. (turn off shifting cultivation). Do the same here?
wwieder commented 1 month ago

@slevis-lmwg let's make a 16PFT surface dataset at the f19 resolution in 1850 + a historical simulation.

slevis-lmwg commented 1 month ago

I submitted the job to generate new fsurdat/landuse files earlier this afternoon. (Issue https://github.com/ESCOMP/CTSM/issues/2791)

olyson commented 1 month ago

Crops off (use 16pft dataset).

slevis-lmwg commented 1 month ago

The files are ready (again, details in https://github.com/ESCOMP/CTSM/issues/2791)

olyson commented 1 month ago

I think this is ready to start (beginning with the AD). But it might be good for anyone who is interested to check the setup: /glade/work/oleson/ctsm5.3.0/cime/scripts/ctsm530_f19_PPE_AD @wwieder @slevis-lmwg @linniahawkins @djk2120

djk2120 commented 1 month ago

everything matches my expectations

djk2120 commented 1 month ago

is it still appropriate to include: reseed_dead_plants = .true.

edit: I can see that this is indeed the setting

slevis-lmwg commented 1 month ago

My understanding is: yes.

slevis-lmwg commented 1 month ago

Looks good to me.

olyson commented 1 month ago

reseed_dead_plants is set to .true. by default, but in a cold start (as this AD is) I don't think it actually does anything.

olyson commented 1 month ago

This spinup is complete. The restart file is:

/glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm530_f19_PPE_pSASU.clm2.r.0161-01-01-00000.nc

I'm running a branch now to look at survivability. Pending that looking ok, I plan to proceed with the historical.

olyson commented 1 month ago

ctsm530_f19_PPE_fullout_pSASU_pft_2dPDF_whittaker-0

ctsm530_f19_PPE_fullout_pSASU_pft_2dPDF_whittaker-1

linniahawkins commented 1 month ago

Thanks Everyone! I am testing this restart in our PPE workflow now. @olyson can you change the permissions on this file: /glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm530_f19_PPE_pSASU.clm2.r.0161-01-01-00000.nc

olyson commented 1 month ago

Ok, done.

djk2120 commented 1 month ago

@olyson @wwieder Would it be possible to continue this simulation through the full historical? It would be really helpful for assessing if our old sparsegrid still works, and for training a new sparsegrid if not.

wwieder commented 1 month ago

Seems reasonable. Can you kick off a historical run, Keith?

olyson commented 1 month ago

Yes, I'll start that up.

olyson commented 4 weeks ago

The historical is done. The output is here:

/glade/derecho/scratch/oleson/archive/ctsm530_f19_PPE_hist

I'll run some diagnostics when I'm back in the office.

linniahawkins commented 4 weeks ago

Great, thank you Keith!

olyson commented 3 weeks ago

Survivability and max TLAI from the historical:

ctsm530_f19_PPE_hist_pft_2dPDF_whittaker-0

ctsm530_f19_PPE_hist_pft_2dPDF_whittaker-1

wwieder commented 1 week ago

@olyson when you are able, can you create an SSP2-4.5 continuation with this case through 2100? (I think that's the land use timeseries we have available for the f19 resolution with 5.3 datasets?). I'll do the same with #72

olyson commented 1 week ago

It looks like the one to use is:

/glade/campaign/cesm/cesmdata/cseg/inputdata/lnd/clm2/surfdata_esmf/ctsm5.3.0/landuse.timeseries_1.9x2.5_SSP2-4.5_1850-2100_78pfts_c240905.nc

or:

/glade/campaign/cesm/cesmdata/cseg/inputdata/lnd/clm2/surfdata_esmf/ctsm5.3.0/landuse.timeseries_ne0np4.ARCTIC.ne30x4_SSP2-4.5_1979-2026_78pfts_c240908.nc

The historical landuse file that was used for the PPE historical was:

/glade/derecho/scratch/slevis/temp_work/new_rawdata/tools/mksurfdata_esmf/landuse.timeseries_1.9x2.5_hist_1850-2023_16pfts_c240926.nc

@slevis-lmwg , do you have a recommendation for which file to use, thanks.

slevis-lmwg commented 1 week ago

Though you didn't list it in your question, I recommend this one: landuse.timeseries_1.9x2.5_SSP2-4.5_1850-2100_78pfts_c240908.nc

All the files dated 0905.nc are an earlier iteration. @wwieder @olyson @ekluzek how about I remove them now to prevent recurring confusion?

olyson commented 1 week ago

Sorry, copy/pasted the wrong one with the 240908 date. That makes sense.

wwieder commented 1 week ago

These are no crop runs is the 78 pft data ok, or should we use 'glade/derecho/scratch/slevis/temp_work/new_rawdata/tools/mksurfdata_esmf/landuse.timeseries_1.9x2.5_hist_1850-2023_16pfts_c240926.nc'?

slevis-lmwg commented 1 week ago

You could, though the 78pft file is fine, too. Linnia is using the 16pft file to run a little faster if I remember correctly.

olyson commented 1 week ago

That's a historical that only goes through 2023, right? I used it for the PPE historical. I was thinking about using:

fsurdat = '/glade/campaign/cesm/cesmdata/cseg/inputdata/lnd/clm2/surfdata_esmf/ctsm5.3.0/surfdata_1.9x2.5_hist_1850_78pfts_c240908.nc' flanduse_timeseries = '/glade/campaign/cesm/cesmdata/cseg/inputdata/lnd/clm2/surfdata_esmf/ctsm5.3.0/landuse.timeseries_1.9x2.5_SSP2-4.5_1850-2100_78pfts_c240908.nc'

olyson commented 1 week ago

@wwieder , I think we want the anomaly forcing settings below. Otherwise, since we are starting in 2024, the datm will get the last time slice of the data (Dec 2100) of the anomaly forcing to interpolate from, at the start of the simulation:

Anomaly.Forcing.Temperature:year_first=2023 Anomaly.Forcing.Temperature:year_last=2100 Anomaly.Forcing.Temperature:year_align=2023

wwieder commented 1 week ago

sounds good

olyson commented 3 days ago

The SSP245 anomaly simulation is complete. Do we need to run some diagnostics?

wwieder commented 2 days ago

let's not worry about it now. I did some spot checking of results of fields Tom may be interested in, but he's likely going to want to make nicer figures for a publication / presentation