Open wwieder opened 8 months ago
Since, this requires CTSM5.2 it actually should wait for #2372. If we need it sooner it could be added to CESM3_dev. But, I'd like to avoid that if we can. This was also on the CTSM5.2 board as a post-5.2 activity, so I'm linking this there.
Also this relates to #554, so linking it here.
@wwieder and I discussed this and thought that we could just clone the most recent deadveg branch simulation and switch from CRUJRA to GSWP3. In that recent simulation, tillage and residue removal was on. And I have been using CTSM5.2 surface datasets, i.e., for that run I used
/glade/campaign/cesm/cesmdata/cseg/inputdata/lnd/clm2/surfdata_esmf/ctsm5.2.0/surfdata_0.9x1.25_hist_78pfts_CMIP6_1850_c230517.nc
I know that @slevis-lmwg has recently re-generated all of the datasets, so I was planning to use that version which appears to be:
/glade/campaign/cesm/cesmdata/cseg/inputdata/lnd/clm2/surfdata_esmf/ctsm5.2.0/surfdata_0.9x1.25_hist_1850_78pfts_c240216.nc
Is there another reason to wait for #2372 ?
Nope that makes perfect sense @olyson. Better to not have to wait when you don't have to.
I initially set this up as an f09 (1deg) simulation, but I think it should actually be an ne30 since we'll use it as initial conditions for the ne30 F- and B-cases coming up. I don't think the finidat interpolation will handle going from 1deg to ne30. Is that a correct assumption @ekluzek and @billsacks ?
@olyson it should be fine to interpolate from f09 to ne30np4.pg3 grid. The interpolation is just nearest neighbor so it will work between any type of grid.
But, I agree that we should spinup the land with the workhorse ne30 grid rather than f09. We'll interpolate to f09 from it for the land only simulations.
I agree with @ekluzek .
Here is an ne30 spunup initial file for 1850 with configuration as noted above:
/glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm51_ctsm51d166deadveg_ne30pg3ne30pg3mg17_GSWP3V1_ABsnoCDE_blk_A5BCD_1850pAD.clm2.r.1361-01-01-00000.nc
Thanks @olyson should this be the default finidat file we point to for simulations with 'modern' tags? Here I'm especially thinking about the CESM alpha (or beta) 17 tag that's upcoming?
Yes, I think it will be suitable for the upcoming CESM tags.
@ekluzek and @slevis-lmwg, can we make it so this initial conditions file is used out of the box with the next round of coupled model experiments?
/glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm51_ctsm51d166deadveg_ne30pg3ne30pg3mg17_GSWP3V1_ABsnoCDE_blk_A5BCD_1850pAD.clm2.r.1361-01-01-00000.nc
looking at the project board, I wonder if this should come in with #2492, or actually #2501?
@wwieder it looks like that's GSWP3 forcing finidat file. Do you want that with I1850Clm60BgcCrop and also with B1850/F1850 or just the former because it's for GSWP3 forcing?
all of the above B, I, & F1850 until we get new cpl.hist files to try a spinup for the B case. We want to provide living arctic vegetation as much as possible.
It looks like the fsurdat file used for that IC file was with a preliminary version of ctsm5.2.0 datasets, and is different by at least roundoff level. That means that it will likely require use_init_interp set to TRUE.
Also @olyson note that in CSEG Meeting Notes Mike Levy tells us that they will be going to the t232 mask for MOM. So we should start using it for our simulations as well.
Ok, sounds like it is a showstopper, so I'll start another spinup with the latest code and the latest surface dataset (which out of the box seems to be /glade/campaign/cesm/cesmdata/cseg/inputdata/lnd/clm2/surfdata_esmf/ctsm5.2.0/surfdata_ne30np4.pg3_hist_1850_78pfts_c231026.nc) and the t232 mask.
@olyson I don't mean to say it's a showstopper. I mean to say we need to transition over to it. I think we need this in place by end of June for the science capability freeze. But, actually we should talk as a group about this in CTSM as well as CESM.
Ah, ok, sorry, I didn't read your comments carefully enough.
FWIW it also seems like @slevis-lmwg is creating some new initial condition files for matrix-CN work, although these also are unlikely to have the correct ocean mask.
New finidat file name will be:
lnd/clm2/initdata_esmf/ctsm5.2/clmi.I1850Clm60BgcCrop-ciso.1361-01-01.ne30pg3_mg17_c240317.nc
And used for standard 1850 control and hist compsets starting at 1850 with clm5_1 and clm6_0 physics.
The list of initial condition files is complex. I'm going to map out what we currently have, so we can start to modify the list:
Resolution | Tuning | BGC | Irrigate | Date |
---|---|---|---|---|
f09_g17 | clm4_5_GSW | Bgc | T | 18500101 |
f09_g17 | clm4_5_CRU | Bgc | F | 18500101 |
f09_g17 | clm4_5_CAM6 | BgcCrop | F | 18500101 |
f09_g17 | clm5_0_GSW | BgcCrop | F | 18500101 |
f09_g17 | clm5_0_GSW | Sp | T | 18500101 |
f09_g17 | clm5_0_CRU | BgcCrop | F | 18500101 |
f09_g17 | clm5_0_CAM6 | BgcCrop | F | 18500101 |
f19_g17 | clm4_5_GSW | BgcCrop | T | 20110101 |
f09_g17 | clm5_0_CAM6 | BgcCrop | T | 20000101 |
f19_g17 | clm5_0_CAM6 | Sp | T | 20030101 |
f09_g17 | clm5_0_CAM6 | Sp | T | 19790101 |
f19_g17 | clm5_0_CAM6 | Sp | T | 19790101 |
ne0np4.ARCTIC | clm5_0_CAM6 | Sp | T | 19790101 |
ne0np4.ARCTICGRIS | clm5_0_CAM6 | Sp | T | 19790101 |
ne0np4.CONUS | clm5_0_CAM6 | Sp | T | 20130101 |
ne120np4.pg3 | clm5_0_CAM6 | Sp | T | 20000101 |
ne30np4.pg3 | clm51* | BgcCrop | F | 18500101 |
ne30np4.pg3 | clm60* | BgcCrop | F | 18500101 |
@adamrher we will likely want your help in getting initial conditions for the VR and ne120 grids for CESM3. We aren't ready to do this now, but wanted to give a heads up to even see if you are able to do that when the time comes. For these grids in CESM3.0 we could use the current finidat files, but that strikes at me as bad to do for the release.
I can help when the time comes. I'm thinking that I could initialize a ne120 clm60 AMIP run with the clm50 ne120 finidat file we are currently using to initialize the ARCTIC and ARCTICGRIS cases, and run it out for a bit.
I am happy to help, as well.
HI @ekluzek, I'm setting up ne120 to create some land inic for another project, so I can start this work now. Based on your table there are 4 grids that need to be addressed (ARCTIC, ARCTICGRIS, CONUS, and ne120pg3). Three of them use the same finidat file -- the ne120pg3 finidat file I create a few years back. However, CONUS has a special 2013 findat file on the CONUS grid that I don't really know the backstory on. Ill have to do some digging to figure out how or whether to change this CONUS finidat file.
In the meantime, I'm going to generate a new ne120 finidat file to replace the current one used by those three grids. I am using cesm2_3_alpha17f
and currently generating landuse_timeseries for the ne120 run. However I just noticed that our workhorse compsets FLTHIST
and BLTHIST_v0c
have CLM51%BGC-CROP
in this latest tag. Should these be CLM52 or CLM60?
Thanks for that update @adamrher this sounds great. Using cesm2_3_beta17 will be great for this purpose. The one caution is that there will be tuning changes that come in that might mean we should wait until AFTER the tuning to do finidat files? Or should we do it now as well as later?
I think the CAM testing starts in 2013 for the CONUS grid, so it's probably best to have a finidat file for that date. We might need to brainstorm how to get it though...
Using CLM51%BGC-CROP is perfectly fine (it'll be identical to CLM60%BGC-CROP. So I'd just leave it like that. If it were CLM50 we'd need to change it, but not CLM51.
Also note I haven't filled out the full table, but think I should to get the best handle on the big picture. But, in terms of your contribution it's only the VS grids and ne120, so your part should be covered.
OK thanks for clarifying my confusion regarding CLM51 and CLM60 (they're the same). I do agree that we may want to hold off on generating new inic until we're closer to a final tuned up version. Let's revisit sometime after the workshop.
To re-do the 2013 inic, I think we can just run the CONUS grid with nudging for a couple months before the 2013 inic date. Someone at ACOM must have done this so I will reach out to them.
I'll add a comment here, but wanted to note that the initial conditions being created in LMWGDEV#65 that has:
Is also being forced by CRUJRAv2, which has no data over Antarctica (consistent with other TRENDY products). I worry that these these initial conditions on the ne30 grid will not be suitable for coupled model simulations. see /glade/campaign/cgd/tss/projects/TRENDY2024/inputs/three_stream
Here's my suggestion:
Discussed at CTSM SE meeting today.
Needed for coupled model runs that Cecile will start hopefully in the next few weeks. Longest-term fix is to have a version of CRUJRA we can support that includes Antarctica—just put GSWP3 data there.
Decision: Go ahead and do that; skip intermediate steps. @wwieder, @slevis-lmwg, and @adrifoster to work on this. Can be done on Izumi next week during Derecho/Casper downtime if the datasets needed get copied—Sam L. will do this. Even just 1901-1920 to have something.
Keith shared his script from 2014 where we ran the model at 0.5 (resolution of CRUNCEP data) forced by Qian data (at T42) and saved 3-hourly coupler history files and then blended that into CRUNCEP forcing data.
/glade/u/home/oleson/cruncep/blend_Qian_CRUNCEP.ncl
Discussed at CTSM SE meeting today.
Needed for coupled model runs that Cecile will start hopefully in the next few weeks. Longest-term fix is to have a version of CRUJRA we can support that includes Antarctica—just put GSWP3 data there.
Decision: Go ahead and do that; skip intermediate steps. @wwieder, @slevis-lmwg, and @adrifoster to work on this. Can be done on Izumi next week during Derecho/Casper downtime if the datasets needed get copied—Sam L. will do this. Even just 1901-1920 to have something.
Since Thursday's meeting I have tried copying files to a common space on izumi (/project/tss02); however, permissions continue to fail, so I'm resorting to copying the files that I think we will need here:
/project/tss/slevis/atm_forcing.datm7.GSWP3.0.5d.v1.c200929
/project/tss/slevis/TRENDY2024/inputs/three_stream
see also CRUJRA files that include Antarctica mentioned here
I realize I didn't copy the mesh file if we're able to test longer simulations on Izumi, so hopefully you have these, Sam?
Here's what the new fields look like, for all us visual people.
And their location on izumi: /project/tss/TRENDY2024/inputs/three_stream
From 2024/9/16 stand-up meeting @slevis-lmwg will try two I1850SP simulations with hourly output with: 1) the new datm data /project/tss/TRENDY2024/inputs/three_stream 2) the gswp3 data /project/tss/slevis/atm_forcing.datm7.GSWP3.0.5d.v1.c200929 3) why not a third case with the orig. TR24 data, for an extra sanity check...
I realize I didn't copy the mesh file if we're able to test longer simulations on Izumi, so hopefully you have these, Sam?
I think I have the mesh files that I need for (2) and (3) which means gswp3-only and trendy-only runs. I will need to generate a new mesh file for the blended product.
Update
ncks --rgr infer --rgr scrip=/project/tss/slevis/TRENDY2024_GSWP3_blended/scrip_all1.nc clmforc.CRUJRAv2.5_0.5x0.5.TPQWL.1901.nc /project/tss/slevis/TRENDY2024_GSWP3_blended/foo_all1.nc
/project/esmf/PROGS/esmf/8.2.0/mpiuni/gfortran/9.3.0/bin/binO/Linux.gfortran.64.mpiuni.default/ESMF_Scrip2Unstruct scrip_all1.nc mesh_all1.nc 0
I'm running the mesh_mask_modifier tool now.
fwiw @adrifoster suggested just using the mesh_maker tool in tools/site_and_regional to generate the mesh, Sam.
Thanks for the reminder @wwieder. I tried mesh_maker just now and got the same error that I saw with all my attempts. I'm starting to wonder whether the problem is with the datm data somehow.
No idea whether this would be a problem, but here's the difference between a CRUJRA file that works:
float lon(lon) ;
lon:units = "degrees_east" ;
lon:long_name = "longitude" ;
lon:mode = "time-invariant" ;
float lat(lat) ;
lat:units = "degrees_north" ;
lat:long_name = "latitude" ;
lat:mode = "time-invariant" ;
float LONGXY(lat, lon) ;
LONGXY:units = "degrees_east" ;
LONGXY:long_name = "longitude" ;
LONGXY:mode = "time-invariant" ;
float LATIXY(lat, lon) ;
LATIXY:units = "degrees_north" ;
LATIXY:long_name = "latitude" ;
LATIXY:mode = "time-invariant" ;
float FSDS(time, lat, lon) ;
FSDS:_FillValue = 1.e+36f ;
FSDS:units = "W/m**2" ;
FSDS:long_name = "incident shortwave radiation" ;
FSDS:mode = "time-dependent" ;
FSDS:missing_value = 1.e+36f ;
and one of the blended files:
float lon(lon) ;
lon:_FillValue = NaNf ;
float lat(lat) ;
lat:_FillValue = NaNf ;
float LONGXY(lat, lon) ;
LONGXY:_FillValue = NaNf ;
LONGXY:units = "degrees_east" ;
LONGXY:long_name = "longitude" ;
LONGXY:mode = "time-invariant" ;
float LATIXY(lat, lon) ;
LATIXY:_FillValue = NaNf ;
LATIXY:units = "degrees_north" ;
LATIXY:long_name = "latitude" ;
LATIXY:mode = "time-invariant" ;
float FSDS(lat, time, lon) ;
FSDS:_FillValue = NaNf ;
For now I will switch my attention back to ctsm5.3 tests as we discussed.
ah, that seems like an issue.
By the way... in case it matters... it probably doesn't at this point:
I accidentally deleted clmforc.GSWP3.c2011.0.5x0.5.Solr.1920-12.nc
in my izumi directory /project/tss/slevis/atm_forcing.datm7.GSWP3.0.5d.v1.c200929
List of nco commands that make a blended file more like a CRUJRA file. First I will test 1901 to see if the run will work.
Summary: Delete _FillValue from most variables. Update time attributes. Update lon attributes. Update lat attributes. Update FSDS attributes. Order time before lat dimension.
ncatted -O -a _FillValue,time,d,, clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc
ncatted -O -a _FillValue,lon,d,, clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc
ncatted -O -a _FillValue,lat,d,, clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc
ncatted -O -a _FillValue,LONGXY,d,, clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc
ncatted -O -a _FillValue,LATIXY,d,, clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc
ncatted -a units,time,m,c,'days since 1901-01-01 00:00:00' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc
ncatted -O -a long_name,time,c,sng,'observation time' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc
ncatted -O -a units,lon,c,sng,'degrees_east' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc
ncatted -O -a long_name,lon,c,sng,'longitude' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc
ncatted -O -a mode,lon,c,sng,'time-invariant' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc
ncatted -O -a units,lat,c,sng,'degrees_north' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc
ncatted -O -a long_name,lat,c,sng,'latitude' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc
ncatted -O -a mode,lat,c,sng,'time-invariant' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc
ncatted -a _FillValue,FSDS,m,float,1.e+36 clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc
ncatted -O -a units,FSDS,c,sng,'W/m**2' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc
ncatted -O -a long_name,FSDS,c,sng,'incident shortwave radiation' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc
ncatted -O -a mode,FSDS,c,sng,'time-dependent' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc
ncatted -O -a missing_value,FSDS,c,float,1.e+36 clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc
ncpdq -a time,lat clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901_new.nc
I made scripts that execute the above for Solr, Prec, and TPQWL automatically. If all this works, then I can modify the scripts to operate on all the years rather than just 1901.
Good news. The above updates to the 1901 files made a difference, and the new simulation is running now. I will wait until we look at results before I also update 1902-1920.
Meanwhile the hourly output is on izumi here (except the third simulation is in progress):
/scratch/cluster/slevis/archive/ctsm52028_f19_GSWP3
/scratch/cluster/slevis/archive/ctsm52028_f19_TR24blended
/scratch/cluster/slevis/archive/ctsm52028_f19_TR24bl_actual
The second simulation should NOT say "blended" in its name (sorry).
Thanks Sam, I'll correct this in the notebook when we make the full dataset
@olyson and I were troubleshooting (in the last hour) a new issue that I came across with the blended data, and we discovered that the mesh file has an additional longitude. In particular, both 360 and zero appear, instead of just 0, and the file has a larger node count than expected as result.
I have the following hypothesis at the moment: I generated the mesh file from one of the datm files before "fixing" them. One of the "fixes" was a reorder of the (time, lat, lon) dimensions. I will regenerate the mesh file with a "fixed" datm file to see if it resolves the issue.
Previous hypothesis: wrong.
However, good news: Seems I misspoke at this morning's meeting when I said that all 3 methods by which I generated mesh files resulted in identical files. From what I see now, the mesh_maker tool added an unnecessary longitude (360) in the mesh file that the other methods did not. It didn't crash the model but changed answers in the first column (0-degree meridian).
I just looked at a new simulation using one of my other mesh files and confirmed this.
@olyson you can now start the spin-up: The new mesh file has the same name that I told you earlier, but you can see that it doesn't have the extra longitude at 360.
As @slevis-lmwg and I discussed, I ran two cases with hourly output on Derecho to verify the mesh file. One with the original atm forcing and mesh file (/glade/derecho/scratch/oleson/temp_work/TRENDY2024/inputs/three_stream, copied from /project/tss/slevis/TRENDY2024/inputs/three_stream/) and the other with the blended atm forcing and new mesh file (/glade/derecho/scratch/slevis/temp_work/TRENDY2024/inputs/three_stream). The forcing was bfb except over Antarctica.
@slevis-lmwg , the two cases are here: /glade/work/oleson/ctsm5.3.n04_ctsm5.2.028/cime/scripts/ctsm53n04ctsm52028_f09_blendedhourly_AD /glade/work/oleson/ctsm5.3.n04_ctsm5.2.028/cime/scripts/ctsm53n04ctsm52028_f09_orighourly_AD The diferenced hourly history file is here: /glade/derecho/scratch/oleson/ANALYSIS/blendedhourly-orighourly.nc
So I've started the ne30 and f09 spinups with the blended atm forcing and mesh file.
@adamrher in the table above linked here...
https://github.com/ESCOMP/CTSM/issues/2403#issuecomment-2105503933
There are finidat files for f19 for 2011 and 2003 as well as 2013 for the CONUS grid. I want to simplify our initial conditions so we are just supporting 1850, 1979 and 2000. Since 2013 is specific for the CONUS grid it sounds like that is important to have for the CONUS grid alone. But, I wanted to make sure you see it as important to have.
CAM does start simulations for a variety of dates outside of 1850, 1979, 2000 (and 2013) including: 1950, 1969, 1974, 1980, 1995, 1997, 1999, 2004, 2005, 2006, 2010, and 2015. We can't sensibly support all of those, but if we have IC files within about 15 years of the start date that seems pretty reasonable. The only outlier would be 1950 which is only used for the FWHIST compsets and I assume WACCM simulations aren't going to be as sensitive to the initial conditions from the land model.
How does that sound? Is there anyone else you'll want to weigh in on this?
Candidate initial files from latest spinups and historicals:
ne30: /glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm53n04ctsm52028_ne30pg3t232_pSASU.clm2.r.0121-01-01-00000.nc /glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm53n04ctsm52028_ne30pg3t232_hist.clm2.r.1979-01-01-00000.nc /glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm53n04ctsm52028_ne30pg3t232_hist.clm2.r.2000-01-01-00000.nc f09: /glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm53n04ctsm52028_f09_pSASU.clm2.r.0161-01-01-00000.nc /glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm53n04ctsm52028_f09_hist.clm2.r.1979-01-01-00000.nc /glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm53n04ctsm52028_f09_hist.clm2.r.2000-01-01-00000.nc
@olyson thank you for these.
Question: Is there a similar list of actual f19 files that I could use. If not, I will point to the f09 files for f19 cases.
There is an 1850 f19 file from the PPE spinup (https://github.com/NCAR/LMWG_dev/issues/70) here: /glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm530_f19_PPE_pSASU.clm2.r.0161-01-01-00000.nc
My understanding based on the last post of this issue is that the PPE group is testing this file now. I haven't run a historical yet which would provide year 1979 and year 2000 initial files.
As we move to CTSM5.2 datasets and introduce CLM6_0 physics options, it seems like it would also be helpful to provide new initial conditions files. @dlawrenncar noted, this may be most important for coupled model simulations, so we can provide more realistic land states in F and B cases as we move forward with CESM3 development runs.
Given very high productivity biases in CRU-JRA forced runs, I'd suggest we provide 1850 initial conditions forced with GSWP3 using CTSM5.2 surface data. It may be most straight forward to do this after #2348 comes to main?
Looking through the LMWG-Dev issues, I don't think we have a GSWP3 forced run that's similar to this case, #54, do we @olyson? If not, we can create a new issue on LWMG-Dev for this.
We should make sure that isotopes, tillage, and residue removal are all on for these spin ups.
Definition of done: