Initial design

[mcgibbon]

This issue should be replaced with sub-issues as we scope out particular features. But for the start, it's good to have a place for over-arching design discussion.

This package is for doing anything you'd need to do to a run directory. It should provide an interface between any new configuration style we want (yaml files, python keyword arguments) and run directories. It should be clear, documented, and narrow in scope (just working on run directories here, workflow tools can import and use this package). Because it is clear and narrow, it should be easy to make it well-tested.

Some features needed:

• create a new run directory
• prepare a run directory for resubmission
  • move restart files from RESTART to INPUT
  • move output files to a location where they will not be overwritten
• modify configuration in an existing run directory
• handle downloading and caching input data for linking/copying into run directories

How to specify run directory configuration is an open design question. A likely option is in dictionaries. You can pass values from a dictionary config to a function func using func(**config). yaml files can be loaded into a dictionary. Python code can easily manipulate a dictionary.

Namelist manipulation can be done with an existing package, like f90nml.

[mcgibbon]

@nbren12 @rheacangeo @ofuhrer Since this is for running the model and for use by the python version, @oliverwm1 and I have mostly been discussing the design, but if you have requirements for this please share them.

[nbren12]

This is a great idea for a package. I would be a fan of a pure yaml front end and a dict + xarray python backend. Also @frodre might be interested since he also has some experience with configuration management.

[nbren12]

I’ve had good luck with f90nml for the NGAQUA work. Here is the code I wrote to configure/run SAM: https://github.com/VulcanClimateModeling/uwnet/blob/master/src/sam/case.py

I originally had the ability to run these “cases” using the same object, but that got pretty unwieldy and violated the single responsibility principle.

[mcgibbon]

Apparently Click also has some good integration with yaml configuration. It lets you use a wizard for configuration flags and then save the configuration you made to a yaml file, for example. This would let us have both a yaml and a command-line frontend with pretty seamless integration. I see the command-line frontend being used mostly for testing where we create fairly standard cases that only change a few parameters (like say, run time).

[mcgibbon]

We're definitely past the initial design stage.