This PR adds a collection of Parsl wrappers that configure and execute the various applications needed for a simple cycled 3DVar JEDI QG workflow. The wrappers consist of "configure" and "run" tasks that execute separately on different resources. The "configure" tasks will run on either a "service" (when internet access is required) or a "serial" resource while the "run" tasks can execute on compute nodes in parallel. Wrappers are created for: installing JEDI, running a "truth" forecast, creating simulated observations from the "truth", running 3DVar using the simulated obs and a background from a previous cycle forecast, and running a forecast initialized from the analysis produced by 3DVar.
These are not the most efficient or best organized wrappers. Much more thought should be put into design that can be applicable for future applications and experiment types. However, effort was made to build configurable components that could be used to run cycled 3DVar experiments with varying configuration options.
An experiment program, workflow.py was created to illustrate how the wrappers can be used to build a workflow.
Automated tests for the various wrappers has not been added yet because it isn't clear what the best way to do it is. All such tests would depend on a JEDI install, which takes time to build, and would take even longer in CI. Testing will be added once a strategy is determined.
This PR adds a collection of Parsl wrappers that configure and execute the various applications needed for a simple cycled 3DVar JEDI QG workflow. The wrappers consist of "configure" and "run" tasks that execute separately on different resources. The "configure" tasks will run on either a "service" (when internet access is required) or a "serial" resource while the "run" tasks can execute on compute nodes in parallel. Wrappers are created for: installing JEDI, running a "truth" forecast, creating simulated observations from the "truth", running 3DVar using the simulated obs and a background from a previous cycle forecast, and running a forecast initialized from the analysis produced by 3DVar.
These are not the most efficient or best organized wrappers. Much more thought should be put into design that can be applicable for future applications and experiment types. However, effort was made to build configurable components that could be used to run cycled 3DVar experiments with varying configuration options.
An experiment program,
workflow.pywas created to illustrate how the wrappers can be used to build a workflow.Automated tests for the various wrappers has not been added yet because it isn't clear what the best way to do it is. All such tests would depend on a JEDI install, which takes time to build, and would take even longer in CI. Testing will be added once a strategy is determined.
Closes #84