Add steps for ISOMIP+ base and culled meshes

[xylar]

This PR represents the first step in porting the ISOMIP+ experiments over from Compass (or, in the case of Ocean 3 and 4, adding new experiments).

I have opted for 3 resolutions here: 1, 2 and 4 km. We had thought long ago that 5 km might be a "typical" resolution for Antarctic cavities in MPAS-Ocean but obviously 12 km is our standard and that is too coarse for ISOMIP+. I also better want to demonstrate and test the flexibility with resolution.

For now:

• I only have a standard z-star version of each test case since the culled mesh will be the same in any case.
• I have both planar and spherical versions of all test cases.
• I am processing the geometry for all 5 experiments: ocean0-4
• I have retained various parameters to the constructor of IsomipPlusTest as placeholders as these features get ported.

There are shared steps at each resolution for creating the base mesh, putting the topography (from ocean1), making a mapping file from the ISOMIP+ data grid to the MPAS base mesh, remapping the topography onto the base mesh (to get the land boundary), culling the mesh, making a mapping file to the culled mesh, and remapping the topography onto the culled mesh. After creating the planar or spherical base mesh and adding the lat/lon coordinate (to planar) or the x/y coordinate (to spherical), the following steps are agnostic to whether the mesh is planar or spherical.

For the insception, wetting and drying test cases, there is a scale step that computes the scaled land-ice pressure and ice draft. This will allow the modified geometry for these 3 experiments to be treated in the same way as the prescribed geometry for the ocean3 and ocean4 experiments.

The next step will be to add a step for creating initial conditions.

I have not yet started adding documentation because the tasks are not complete enough yet in my view to justify it. But I could get started if folks would prefer that.

Checklist

• [ ] User's Guide has been updated
• [ ] Developer's Guide has been updated
• [ ] API documentation in the Developer's Guide (api.md) has any new or modified class, method and/or functions listed
• [ ] Documentation has been built locally and changes look as expected
• [x] Testing comment in the PR documents testing used to verify the changes
• [ ] New tests have been added to a test suite

[xylar]

Testing

I ran all 30 tasks on Chrysalis. I saw some of the slowdown that I had been seeing in my testing of other large suites an my job died before 1 hour. I had to delete a corrupted file at that point but after that it ran successfully through the remaining tests.

I am not inclined to investigate the slowdown right now because these large suites are not how we plan to run typically. Either we will have task parallelism (soon I hope) or we will run smaller suites.

Update: When I ran the 15 planar and the 15 spherical in their own suites, the former took 5 minutes and the latter 10, so I think it's a memory leak that we should keep an eye on.

[xylar]

@cbegeman, no rush at all but this is ready to review when you have the time and energy.

I plan to build on it with an init step in the coming week.

[xylar]

One more comment: I think I will organize some of the steps into a mesh module inside of isomip_plus to keep things tidy.

[xylar]

@cbegeman, based on things I'm discovering in making the init step(s), this should be put in draft mode and you should hold off on reviewing it for now.

[xylar]

@cbegeman, this is ready for review. That can be as involved or as cursory as you see fit. Once this is in, I believe the initial condition PR is also ready to be reviewed. I will continue on, making a forward and a viz step for configurations that will work without wetting-and-drying.

[cbegeman]

@xylar Did you maybe forget to add isomip_plus_topo.cfg? I think inception, wetting, and drying rely on it.

  * step: topo_scale

polaris calling: polaris.ocean.tasks.isomip_plus.topo.scale.TopoScale.constrain_resources()
  inherited from: polaris.step.Step.constrain_resources()
  in /gpfs/fs1/home/ac.cbegeman/polaris-repo/add-isomip-plus-base-and-culled-meshes/polaris/step.py

polaris calling: polaris.ocean.tasks.isomip_plus.topo.scale.TopoScale.runtime_setup()
  inherited from: polaris.step.Step.runtime_setup()
  in /gpfs/fs1/home/ac.cbegeman/polaris-repo/add-isomip-plus-base-and-culled-meshes/polaris/step.py

polaris calling: polaris.ocean.tasks.isomip_plus.topo.scale.TopoScale.run()
  in /gpfs/fs1/home/ac.cbegeman/polaris-repo/add-isomip-plus-base-and-culled-meshes/polaris/ocean/tasks/isomip_plus/topo/scale.py

          execution:        ^[[91mERROR^[[0m
Exception raised while running the steps of the task
Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.cbegeman/polaris-repo/add-isomip-plus-base-and-culled-meshes/polaris/run/serial.py", line 324, in _log_and_run_task
    baselines_passed = _run_task(task, available_resources)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs/fs1/home/ac.cbegeman/polaris-repo/add-isomip-plus-base-and-culled-meshes/polaris/run/serial.py", line 403, in _run_task
    _run_step(task, step, task.new_step_log_file,
  File "/gpfs/fs1/home/ac.cbegeman/polaris-repo/add-isomip-plus-base-and-culled-meshes/polaris/run/serial.py", line 501, in _run_step
    step.run()
  File "/gpfs/fs1/home/ac.cbegeman/polaris-repo/add-isomip-plus-base-and-culled-meshes/polaris/ocean/tasks/isomip_plus/topo/scale.py", line 61, in run
    dates: List[str] = config.getlist('isomip_plus_scaling',
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ac.cbegeman/miniconda3/envs/polaris_isomipmesh/lib/python3.11/site-packages/mpas_tools/config.py", line 200, in getlist
    values = self.get(section, option)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ac.cbegeman/miniconda3/envs/polaris_isomipmesh/lib/python3.11/site-packages/mpas_tools/config.py", line 115, in get
    return self.combined.get(section, option)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ac.cbegeman/miniconda3/envs/polaris_isomipmesh/lib/python3.11/configparser.py", line 797, in get
    d = self._unify_values(section, vars)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ac.cbegeman/miniconda3/envs/polaris_isomipmesh/lib/python3.11/configparser.py", line 1168, in _unify_values
    raise NoSectionError(section) from None
configparser.NoSectionError: No section: 'isomip_plus_scaling'

[cbegeman]

@xylar Did you maybe forget to add isomip_plus_topo.cfg? I think inception, wetting, and drying rely on it.

Or maybe it has to do with the fact that those sections are in ocean/planar/isomip_plus/4km/isomip_plus_topo.cfg but not in ocean/planar/isomip_plus/4km/z-star/inception/inception.cfg and you're trying to retrieve it from the wrong config file?

[xylar]

Did you maybe forget to add isomip_plus_topo.cfg?

I'll get to the bottom of this as soon as I can. I must have a commit in my branch for adding init that fixes this and it should be on this branch instead. Because my init branch works fine.

[xylar]

Did you maybe forget to add isomip_plus_topo.cfg?

I'll get to the bottom of this as soon as I can. I must have a commit in my branch for adding init that fixes this and it should be on this branch instead. Because my init branch works fine.

Hmm, I don't understand why my init branch is working because there's definitely something wrong with scale. The config parameter is getting passed in but nothing is being done with it. The fix should be easy enough.

[xylar]

Hmm, ignore my last comment. It's not that. More debugging to do...

[xylar]

@cbegeman, I believe I addressed your comments and fixed the issue with config files and config options (in #157). I need to retest with the last few commits but I don't expect any surprises since they are just minor clean-up. I'll post when my testing is done but I also think this is ready to re-review, perhaps after or along with #157.

[xylar]

Retesting

I ran all the ISOMIP+ test cases from this PR again on Chrysalis. Other than a mysterious hang that I wasn't able to reproduce, everything ran fine.

[xylar]

Rebased following the merge of #157 to hopefully make review a little simpler.

[xylar]

Thanks @cbegeman. I continue to think documentation will make more sense in a later PR so I'm going to merge this now. I really appreciate you giving this such a thorough review. I'm excited for the next steps!