Change how dynamic model configuration is update

[xylar]

Frequently, steps that run the model need to update yaml files, namelist options and/or streams in the same way at setup and runtime based on config options and calculations that make use of them. A common example is the model time step, which may be based on both the mesh resolution and config options that determine the scaling between mesh resolution and time step. Run durations for short regression tests should typically be 2 or 3 time steps (thus, they also ideally need to be set dynamically both a setup and runtime).

This merge adds a new dynamic_model_config() method to ModelStep that is called at both setup and runtime. A step should override this method. Within it, a step can call the same add_model_config_options(), add_yaml_file(), etc. methods that can be called at construction or setup. The only difference is that they will be updated both as setup and runtime.

This merge also adds the capability to use yaml files that are jinja templates. These will be useful for ocean test cases that used this kind of templating in compass. It isn't convenient to specify replacements for streams data in a dictionary as it is with other model config options.

Additionally, this merge makes a modification to how MPI resources are constrained based on the number of cells in a mesh. This change from #29 was found to be more convenient and intuitive in #27 and I'm including it here as it seems somewhat conceptually related. Instead of a cell_count attribute in OceanModelStep, there is a compute_cell_count() method that gets called at both setup and runtime. At setup, it will not necessarily be possible to determine the exact cell count so, for example, in the cosine bell test case, a heuristic is used as setup. But at runtime, it is possible to read nCells from an input file so it can be known exactly and this is what is done in compute_cell_count() in the cosine bell's forward step.

The cosine bell test case has been updated with these changes. It doesn't take advantage of the jinja-templated yaml files but it does use dynamic_model_config() and compute_cell_count().

Checklist

• [x] Developer's Guide has been updated
• [x] API documentation in the Developer's Guide (api.md) has any new or modified class, method and/or functions listed
• [x] Documentation has been built locally and changes look as expected
• [x] Testing comment in the PR documents testing used to verify the changes

[xylar]

This can remain on hold until there is a need for it. I thought I would use it in #27 but eventually decided against it.

[xylar]

Testing

I tested this on the cosine bell test cases on Chrysalis with Intel and OpenMPI. The tests pass but I'm seeing machine-precision differences in tracer1 between the test results and a baseline from the current main (using the same MPAS-Ocean executable). This is odd and may need future investigation but since I saw something similar in #29, I'm fairly sure it's not something introduced here.

/lcrc/group/e3sm/ac.xylar/polaris_0.1/chrysalis/test_20230401/cosine_bell_dynamic_model_config

[xylar]

@cbegeman, this is ready for you to have a look. Feel free to review by inspection and also feel free to only look as deep into the code changes as you think you have time to. It's kind of hairy.

[xylar]

After a few more tweaks, this works in #27 as well.

[xylar]

@cbegeman, thanks so much for the review!