kdraeder
commented
1 year ago Here's the SourceMods I developed for the CAM6 Reanalysis. cesm2_1_relsd_m5.6z_DART+CAM_SourceMods.tgz
Here are the changes to CIME I made for the CAM6 Reanalysis, which are candidates for merging into main. See the included file merge_list_2023-3-11 for details. DART_CIME_maint-5.6_mods.tgz
There were more changes needed (or useful), but they are too specific to DART and the Reanalysis to include. So we will still have mods that we need to install manually (unless there's a SourceMods mechanism for CIME).
hkershaw-brown
braczka
Use case
It would be very helpful to people who use CESM+DART if the released versions of CESM have been tested to ensure that they provide the functionality required by DART, such as the ability to build and run large, multi-instance jobs and interact with DART as an External System Processing component.
Is your feature request related to a problem?
Recently developed CESM components can be used and tested with DART only if the CESM infrastructure continues to enable it. Often in the past CESM or CIME development has neglected to test for this use, and the resulting release has been incompatible with DART (CESM issue #1807). This required people who are not experts in CESM code to inefficiently find and suboptimally fix the problems. This can take as long as it takes the CESM developers to release a new version, resulting in a seemingly endless cycle, during which DART cannot be used with recent CESMs.
Describe your preferred solution
(Further) Integrate testing of the functionality required by DART into the CESM testing suite. This is most important for major releases, which are most likely to be used with DART. But it would be useful whenever a new version of a component model is made available (if there's a DART interface to that component), so that DART can be used to evaluate it. We anticipate that the CESM testing would not involve running an assimilation, but just the features of CESM that enable that. After passing the CESM tests, then people who want to do the assimilation would do the full assimilation testing and model evaluation.
Describe any alternatives you have considered
See "Is this related to a problem"
The rest of the text is a discussion of strategies for implementing solutions.
Short list of issues
Depth of testing
It's possible that a different level of testing could be done, based on what parts of CESM had been upgraded (e.g. multi-instance scripting vs a wave-model upgrade, which is currently irrelevant to DART) or what level of release is being made (e.g. CESM2_3_1 vs CESM3). ? Does the testing currently vary, depending on these things?
Size of Ensemble
Several times in the past, tests using 2-3 members have passed, but tests with larger numbers have failed, or shown unacceptable performance degradation. My (Kevin's) intuition is that at least 10 members are required to see the latter. For example, at one point the number of calls to the serial task, python build-namelist was a function of the number of instances squared. That was not noticable for 2 members, but for 80 it was 6400 calls.
Components
DART currently has interfaces to atm (CAM-FV, CAM-SE, WACCM(-X)), CLM5, POP(2?), and CICE. The potential next interfaces are to a river or land ice model. Work is underway to do assimilation with multiple components, but that will (probably) be hidden within the ESP component and not require testing by CESM. As of 2018-6 any compset that included a land ice component other than "stub" couldn't be used, because all of those other land ice models cannot use the gregorian calendar. A compset defined (2018) specifically for CAM6 assimilations is FHIST_DARTC6 = HIST_CAM60_CLM50%SP_CICE%PRES_DOCN%DOM_SROF_SGLC_SWAV.
SourceMods
Each DART interface may currently require SourceMods in order to build a model that works with DART effectively. These have been necessary partly because of the lack of testing within CESM. It may be easy, or at least appropriate, to include some of them into CESM, while others may not be. See attached files for examples. There may be changes to CIME, which we implemented for DART, which are not in the (cam.src) SourceMods. For example, /glade/work/raeder/Models/cesm2_1_relsd_m5.6/cime/src/drivers/mct/cime_config/buildnml has a time saving upgrade that just changes the log file name in modelio namelist files, instead of regenerating them in every assimilation cycle. There's no SourceMods mechanism for CIME code, that I know of, so it needs to be substituted manually. ? Do other components have nonSourceMods changes?
Size of the model(s)
We have not run into cases in which the resolution of the model was a factor in the testing or functionality of the code. Of course, it's always possible to exceed resources using a high resolution model, but that's not in the testing scope. So testing a "large" ensemble may not require a large number of nodes, which can delay testing.