Implement blended-area-weighted coarsening for restart files and diagnostics

This PR implements blended area-weighted coarse-graining for restart files in FV3GFS, introducing a single new "strategy" option: "blended_area_weighted". It implements the approach described by Chris in Section 7 of this document to smoothly transition between model level coarse graining near the surface to pressure level coarse-graining aloft.

This PR is split off from #336 to isolate the implementation of the blended-area-weighted strategy, I will push a commit that just implements the restart file coarsening from that PR, make sure the tests pass, and then push a commit that implements the strategy for diagnostics, which will require updates to the regression tests.

Timing results

While this approach technically involves coarsening twice (once with a model-level-area-weighted average and once with a pressure-level average), the model-level-area-weighted average has minimal cost, and so this approach has little overhead over outputting data with the pressure-level approach alone.

The table below summarizes the results of some timing experiments. In these experiments we have run 5-day C384 simulations with the three implemented coarse-graining approaches, outputting coarse diagnostics and restart files every three hours, similar to the outputs we made from the PIRE simulations, and also run a reference simulation with no intermediate restart files or diagnostics written.

Strategy	Total runtime (s)	Runtime above case with no outputs (s)
No outputs	1300	0
`model_level`	1411	111
`pressure_level`	1474	174
`blended_area_weighted`	1481	181

The results suggest that outputting data with the blended-area-weighted approach increases wall clock time by less than 1% over outputting the same data with the pressure-level approach.

In terms of other things we can conclude from these experiments:

Using model-level coarse-graining instead of the more complicated coarsening approaches results in a time savings of about 4%.
We've illustrated this in prior timing experiments as well, but there is a non-trivial I/O and communication cost overhead of outputting this many diagnostics with a 1x1 I/O layout alone, which likely makes up much of the cost from going from no outputs to model-level coarsened outputs in the table. This because we can infer from the delta of going from pressure-level to blended-area-weighted coarsening that the computational cost of model-level coarsening is minimal.

Validation

This notebook validates this fortran implementation for both restart files, dynamical core diagnostics, and physics diagnostics against the version we implemented earlier in Python in fv3net.

ai2cm / fv3gfs-fortran

Implement blended-area-weighted coarsening for restart files and diagnostics #370

Timing results

Validation