add support for coarser spectral-element grid for CAM regression testing

[adamrher]

As we are transitioning to the se dynamical core in CAM, we need a couple coarse se grids with land support for regression testing. This is analogous to our tests using coarse finite-volume grids like f10_f10_mg37, which we will be phasing out. We propose three grids, two of which are already supported in CESM and just need CTSM support:

ne5pg3_ne5pg3_mg37
ne16pg3_ne16pg3_mg17

We plan to generate an entirely new grid that will also need land support:

ne3pg3_ne3pg3_mg37

Adding a new grid requires coordination between CIME, CAM and CTSM. This serves as the CTSM leg of that effort, whereas the CAM effort is here: https://github.com/ESCOMP/CAM/issues/726. @jedwards4b @fischer-ncar would you like me to open up an issue in CIME to add this new grid to ccs_config/modelgrid_aliases_nuopc.xml and ccs_config/component_grids_nuopc.xml?

@cacraigucar

Definition of done:

• [x] New resolutions added to ctsm5.1 main-dev
• [x] Add some tests for ne3pg3_ne3pg3_mg37 to the aux_clm test suite
• [x] New resolutions added to mksurfdata_esmf on the ctsm5.2 branch

[wwieder]

Thanks for this note @adamrher . What timeline are you looking for with this? @slevisconsulting @ekluzek and @billsacks are we close enough with CTSM5.2 tag to creating these surface datasets for CAM with our updated input data and workflows?

[adamrher]

What timeline are you looking for with this?

@cacraigucar would have to confirm, but I think we'd like this completed sometime before the end of January. I agree with adding support only with the new surface dataset workflow if that can be ready in time.

[cacraigucar]

I can say that once the grids are available, we will use them. From a regression testing standpoint, the ne3pg3_ne3pg3_mg37 will be the workhorse grid we will utilize the most and is from a testing standpoint, the most important to have made for us. If it is ready in January, we will use it. The others will be used in regression testing to a lesser extent to maintain their integrity.

[fischer-ncar]

@adamrher You'll need to add a https://github.com/ESMCI/ccs_config_cesm issue. You can assign it to me.

[ekluzek]

@wwieder in terms of CTSM5.2 I think a missing piece right now is the future scenario PFT data. We might have that in January though. We are also missing the 0850-1849 data, but that doesn't matter for the standard files. And it would also be OK to have these files created outside of the CTSM5.2 process.

But, we do need to add these as grids that we'll create in CTSM. @adamrher could you add an issue in CTSM for creating the surface datasets for these grids?

Also @adamrher and @cacraigucar what periods do you need for these grids? Just 2000? Or both 1850 and 2000 and historical? And will you need future scenarios?

[cacraigucar]

All I can speak of is for the testing scenarios. From a testing standpoint, the coarsest ne3pg3 will probably be utilized for a significant part of our testing. As such, I would think that we might need all time periods. Our current testing uses f10 for F2000climo and F1850. The FHIST go as coarse as 2 degree, but with the increased vertical resolution, we may opt for ne3pg3 for it as well.

[adamrher]

The FHIST go as coarse as 2 degree, but with the increased vertical resolution, we may opt for ne3pg3 for it as well.

agreed, we want flanduse_timeseries historical support (nothing before 1850 or past 2015), in addition to 2000 and 1850 surface datasets.

@adamrher could you add an issue in CTSM for creating the surface datasets for these grids?

Once ne3pg3 is installed in CAM and ccs_config, then that is all this issue would be asking for, correct? Should I rename this one to say that, or do you still want a separate issue for the surface datasets?

[ekluzek]

Oh, whoops sorry @adamrher I thought this was an inssue in ccs_config and not CTSM. You probably need an issue in ccs_config, CTSM, and CAM that is coordinated for this work. I'll remove the issue I made you add as a duplicate. Sorry about that!

[adamrher]

You should have access to all the ESMF mesh files needed to complete this work. The ne5pg3 and ne16pg3 files are in inputdata/share/ and ne3pg3 is currently residing here:

/glade/work/aherring/grids/uniform-res/ne3np4.pg3/grids/ne3pg3_ESMFmesh_c221214_cdf5.nc

until it gets pushed into the inputdata repo via: https://github.com/ESMCI/ccs_config_cesm/issues/76.

[adamrher]

Chris just completed the ccs_config PR for this work: https://github.com/ESMCI/ccs_config_cesm/pull/85

Just for my sanity: the next one up should be CAM, right? Or can it be CLM first? That is, is there a practical way for CLM to implement without CAM support for this grid, where CLM would just have access to the fsurdat / flanduse_timeseries files and the new ccs_config externals? Or do we need CAM support first?

[ekluzek]

It can and probably needs to be CLM first. We need a CTSM tag that includes surface datasets for ne3pg3_ne3pg3_mg37, and then in CAM you'll need to point to that CLM tag. The only way around that for CAM is for it to explicitly set the fsurdat file in the testmod in user_nl_clm. So we might as well get this in CLM first.

Since, we are on the cusp of new datasets with CTSM5.2, but not yet there, we probably need this to come to CTSM main first. Unfortunately that means for example creating mapping files for this resolution, even though we won't need them for CTSM5.2.

[adamrher]

OK, sounds good Erik. I was thinking we could get the aqua-planets running in a CAM tag first, and then CTSM would be able to actually run with CAM when installing the ne3pg3 surface datasets. But I like your idea of having CTSM install ne3pg3 support before CAM, better ;)

@cacraigucar pls see this comment from @ekluzek. Do you want to wait until CTSM5.2 is completed, or move forward with what Erik is proposing?

[cacraigucar]

@ekluzek - We would like to have the ability to run with ne3pg3 to start migrating our regression testing from FV to SE. When do you foresee CTSM5.2 being released.? In it is in the next few weeks, we can probably wait. If it is months out, we need these data files before then.

[ekluzek]

@cacraigucar and @adamrher I can give you an update on our latest estimate for CTSM5.2 tomorrow. I know of one open task for it, that is likely at least a few weeks out. So I'm kind of thinking we should go ahead with this pre-ctsm5.2. But, I'll make sure we talk about this tomorrow.

If CAM is going to do more testing with SE than FV, that also might color what we should do in our CTSM test lists.

[ekluzek]

@cacraigucar and @adamrher are one of you going to work on a PR for this?

[adamrher]

@ekluzek, I can start by generating the surface datasets, and then I can probably add them to namelist_defaults_ctsm. I think that's all that is required. Oh and I think I recall I have to add the new grid to a list somewhere. So yes, I think I can make the CTSM PR branch.

Why don't you let us know tmrw how to proceed? At that point could you tell me which tag containing the clm tools I should be using to generate the surface datasets.

Since FV will not be the workhorse in CESM3, replaced by SE, I think all of CESM should stop, or at least slow-down introducing new FV tests, instead favoring new tests with an SE grid. At some point, probably a long ways out, the FV dycore will be a thing of the past, and so we should adapt out testing accordingly.

[slevis-lmwg]

@ekluzek, I can start by generating the surface datasets, and then I can probably add them to namelist_defaults_ctsm. I think that's all that is required. Oh and I think I recall I have to add the new grid to a list somewhere. So yes, I think I can make the CTSM PR branch.

Why don't you let us know tmrw how to proceed? At that point could you tell me which tag containing the clm tools I should be using to generate the surface datasets.

Since FV will not be the workhorse in CESM3, replaced by SE, I think all of CESM should stop, or at least slow-down introducing new FV tests, instead favoring new tests with an SE grid. At some point, probably a long ways out, the FV dycore will be a thing of the past, and so we should adapt out testing accordingly.

@adamrher I generated the ne30 surface dataset yesterday using a local ctsm5.2 branch of mine that's waiting for further updates to the raw data, as we discussed in #1951. I am happy to generate surface datasets as needed. Pls let me know.

[adamrher]

@slevisconsulting thanks so much --- I'd love to take you up on that offer ;) Here are the mesh files and official names:

ne3np4.pg3, nx=486, ny=1

/glade/work/aherring/grids/uniform-res/ne3np4.pg3/grids/ne3pg3_ESMFmesh_c221214_cdf5.nc

ne5np4.pg3, nx=1350, ny=1

/glade/p/cesmdata/inputdata/share/meshes/ne5pg3_ESMFmesh_cdf5_c20210118.nc

ne16np4.pg3, ny=13824, ny=1

/glade/p/cesmdata/inputdata/share/meshes/ne16pg3_ESMFmesh_cdf5_c20211018.nc

Will need flanduse_timeseries (1850->whenever) as well for these.

[slevis-lmwg]

I have generated new files for ne16 and ne5. New files for ne3 should be ready later this afternoon.

surfdata_ne16np4.pg3_hist_78pfts_CMIP6_1850-2015_c230217.nc
landuse.timeseries_ne16np4.pg3_hist_78_CMIP6_1850-2015_c230217.nc
surfdata_ne5np4.pg3_hist_78pfts_CMIP6_1850-2015_c230217.nc
landuse.timeseries_ne5np4.pg3_hist_78_CMIP6_1850-2015_c230217.nc
surfdata_ne3np4.pg3_hist_78pfts_CMIP6_1850-2015_c230217.nc
landuse.timeseries_ne3np4.pg3_hist_78_CMIP6_1850-2015_c230217.nc

in /glade/work/slevis/git/mksurfdata_toolchain/tools/mksurfdata_esmf

[adamrher]

Naive question. Should this PR also bring in the new CCS_CONFIG externals that supports these grids? That way I could at least run an I compset to make sure the mods work.

[adamrher]

Since these files are from the 5.2 toolkit, and we're installing this into 5.1, should I perform these "hacks" from Sam to make them work: https://github.com/ESCOMP/CTSM/discussions/1951#discussioncomment-5013348?

It seems inefficient to do this twice, once for 5.1 and once for 5.2, but maybe that's what needs to happen?

[slevis-lmwg]

@adamrher I'm pretty sure the answer to your question is "yes, that's what needs to happen" and I will let others confirm.

[olyson]

@adamrher I'm pretty sure the answer to your question is "yes, that's what needs to happen" and I will let others confirm.

Regarding the hacks, what I have tried is to (1) Modify the code so that PFTDATA_MASK is not read in, as suggested by @slevisconsulting . See my change to surfrdMod.F90 where I've commented out the read of PFTDATA_MASK (my changes are marked with "!KO"):

/glade/work/oleson/ctsm5.1.dev118/cime/scripts/ctsm51_ctsm51d118_ne30pg3g17_GSWP3V1_hist/SourceMods/src.clm/surfrdMod.F90

(2) Modify the code to prescribe nlevurb as 10 (instead of 5) in clm_varpar.F90:

/glade/work/oleson/ctsm5.1.dev118/cime/scripts/ctsm51_ctsm51d118_ne30pg3g17_GSWP3V1_hist/SourceMods/src.clm/clm_varpar.F90

Those changes have enabled me to start running an IHIST with ctsm5.1.dev118 and the ne30np4.pg3 datasets @slevisconsulting created.

[adamrher]

Hi All,

I just remade the surface datasets using the clm tools from the cesm2.2.0 release tag. I made a PR branch that contains:

(1) update ccs_config externals to ccs_config_cesm0.0.57 (2) add surface datasets for new grids in namelist_defaults_ctsm.xml (+ add new grids to namelist_defintion_ctsm)

I think that more needs to be done to make the new ccs_config externals work. When I tried run an I compset in create_newcase with the new grid, I get:

ERROR: Command: '/usr/bin/xmllint --xinclude --noout --schema /glade/u/home/aherring/src/ctsm5.1.dev118.newgrids/cime/CIME/data/config/xml_schemas/config_batch.xsd /glade/u/home/aherring/src/ctsm5.1.dev118.newgrids/ccs_config/machines/config_batch.xml' failed with error '/glade/u/home/aherring/src/ctsm5.1.dev118.newgrids/ccs_config/machines/config_batch.xml:59: element argument: Schemas validity error : Element 'argument': This element is not expected. Expected is ( arg ).

Any ideas? My branch is here: https://github.com/adamrher/CTSM/tree/ctsm5.1.dev118.newgrids

[jedwards4b]

Adam - cesm2.2.0 is incompatible with the latest ccs_config, can you use cesm2.3?

[jedwards4b]

Or you can create a branch starting from the ccs_config that works in 2.2.0 and make the updates you need in the branch.

[adamrher]

I am not branching from 2.2.0. I'm branching from the head of CTSM development ctsm5.1.dev118. The externals on the head are ccs_config_cesm0.0.38, and when I switched them to ccs_config_cesm0.0.57, create_newcase stopped working.

[billsacks]

It sounds like we're probably due for an externals update tag in CTSM.... I'll talk to others about this.

[jedwards4b]

You can try updating cime to the latest, that should take care of the ccs_config issue.

[adamrher]

@jedwards4b that got me to the build stage. Now I get a cdeps build error:

Running cmake for CDEPS get_standard_cmake_args() got an unexpected keyword argument 'shared_lib'

[jedwards4b]

okay - that's another new incompatibility - try cime6.0.90

[adamrher]

Based on @billsacks email, he's arranging for a new ctsm tag with updated externals. So I'm going to pause this work until that tag is made ... once complete, I'll merge up and issue the PR for this issue.

[adamrher]

@billsacks just want to clarify that you are waiting to update the CTSM externals until alpha12d resolves the test failures introduced in alpha12c, due to updating the ccs_config, cime, cmeps, cdeps, share and pio externals?

[edit - which is fine. I'm just confirming the to-do list]

[billsacks]

Thanks for checking in. I'm working on the externals update now. We have a set of externals that resolve the relevant issues from CTSM's perspective, so I'm moving ahead with what we have. However, I have a few failures in the aux_clm test suite that I'm looking into, then hope to restart testing today or tomorrow.

[billsacks]

@adamrher - we ran into some snags with the externals update, but this is now done: ctsm5.1.dev120.

[cacraigucar]

@billsacks - Did you need to change your externals from what they are in alpha12c? (i.e. when we bring in ctsm5.1.dev120 into a CAM checkout, will updating our other externals to match CESM 2_3_alpha12c be sufficient?)

[ekluzek]

@cacraigucar @billsacks is out this week for Spring Break. So I'll try to answer.

They aren't exactly cesm2_3_alpha12c, as cime had to deviate from that and is using cime6.0.100, rather than cime6.0.94. cmeps is also cmeps0.14.17 rather than cmeps0.14.14. From my read through everything else matches cesm2_3_alpha12c. Looks like there isn't an alpha tag that matches those externals yet. Hopefully, beta12 will get updated to that version of cime.

[ekluzek]

Since, ne3pg3_ne3pg3_mg37 will be the workhorse resolution for CAM, we should add some CTSM tests for it to the aux_clm testsuite. I'll add that to this PR.

[ekluzek]

@adamrher the files you created were with mksurfdata_map, which is what we need here for now. But, we also need to add the capability of running these resolutions to the CTSM5.2 branch.

[adamrher]

Since, ne3pg3_ne3pg3_mg37 will be the workhorse resolution for CAM

ne30pg3_ne30pg3_mg17 will be the workhorse resolution, not ne3. So maybe make the test ne30? ne3 is a coarse (~10deg) grid intended to for quick tests of the spectral-element grid.

[adamrher]

@ekluzek, scroll up to this post and the three following posts, suggesting @slevisconsulting already made the surface datasets for CTSM5.2:

https://github.com/ESCOMP/CTSM/issues/1927#issuecomment-1435146506

[ekluzek]

@adamrher what I mean is that ne3pg3_ne3pg3_mg37 will be the workhorse resolution for testing in CAM as stated by @cacraigucar in for example this message above.

I can say that once the grids are available, we will use them. From a regression testing standpoint, the ne3pg3_ne3pg3_mg37 will be the workhorse grid we will utilize the most and is from a testing standpoint, the most important to have made for us. If it is ready in January, we will use it. The others will be used in regression testing to a lesser extent to maintain their integrity.

You are right that ne30 is the workhorse resolution for running the model, but ne3 will be for testing.

Glad @slevisconsulting already has the datasets ready for CTSM5.2.

[cacraigucar]

When will ctsm5.1.dev120 be brought into the CESM2_3_alpha12 series? Could it go into the next tag scheduled for testing?

[ekluzek]

ctsm5.1.dev120 is scheduled for cesm2_3_alpha12e which is the next tag for testing.

[ekluzek]

We haven't identified a tag for when this work will come in, but we have it in our plans.

[ekluzek]

This is done other than the last item, which is coming in with #2008. So I'm closing now.