E3SM-Project / polaris

Testing and analysis for OMEGA, MPAS-Ocean, MALI and MPAS-Seaice
BSD 3-Clause "New" or "Revised" License
6 stars 13 forks source link

Add dependencies to Step class #39

Closed xylar closed 1 year ago

xylar commented 1 year ago

A step can now have a list of other steps that it depends on. This is useful when file-base dependencies cannot be used (e.g. the file names cannot be known at setup time).

This merge adds a dictionary of dependencies to each step (empty by default) and a method, add_dependency() for adding another step as a dependency.

As fleshed out in the documentation, the dependency must run before the dependent step. The results of running the dependency are pickled at the end of the run (each dependency is identified by its is_dependency attribute being True), and then unpickled along with the dependent step before the dependent step runs. This allows the dependent step to access all attributes of the dependency, including any changes during the dependency's run.

We check to make sure all dependencies of all the steps being set up at a given time have also been set up. This prevents failures at runtime when a dependency belong s to a different test case than the dependent step, and the user has forgotten to set up the dependency's test case (or doesn't know that it, too, is required).

Checklist

xylar commented 1 year ago

Testing

I used this new capability to define a Viz step in the cosine_bell test case that is dependent on a VizMap step. The name of the mapping file that both steps need is not known at setup because it depends on lon-lat resolution and remapping method that come from config options.

I will make a separate PR to add the remapping step to the framework, and then update #34 to only add the global plotting capability. This will be much easier to follow than adding everything in one omnibus PR.

xylar commented 1 year ago

@cbegeman, I doubt you want to review this capability but I do want you to be aware of it. It is mostly necessary in anticipation of task parallelism.