search

MPAS-Dev / MPAS-Tools

MPAS Tools Repository
Other
39 stars 65 forks source link

Add python mask creator #399

Closed xylar closed 3 years ago

xylar commented 3 years ago

This merge adds 4 functions and corresponding command-line tools for making MPAS region, transect and flood-fill masks and for making lon/lat regions masks:

The command-line tools will be used by compass as part of generating meshes and by MPAS-Analysis for generating region and transect masks efficiently for large meshes.

closes #321

xylar commented 3 years ago

Testing

The new masking tools have been tested in both compass and MPAS-Analysis on several machines as detailed below.

compass

The branch https://github.com/xylar/compass/tree/add_python_masks makes use of the new command-line tools for both culling global ocean meshes in mesh test cases and generating regions masks in files_for_e3sm test cases. The results show show substantially better performance than has been seen with mpas_tools.mesh.conversion.mask (which uses MpasMaskCreator.x), particularly on larger meshes.

Machines and configurations tests:

MPAS-Analysis

The branch https://github.com/xylar/MPAS-Analysis/tree/use_mpas_tools_python_masks changes MPAS-Analysis to use the new command-line tools to generate regions masks if they aren't already cached for a given mesh. This approach has been tested on several machines:

xylar commented 3 years ago

@dengwirda, I wanted to make you aware of this work in case it is of use to you. This work replaces nearly every use of MpasMaskCreator.x or mpas_tools.mesh.conversion.mask() in my workflows. The only remaining use of mpas_tools.mesh.conversion.mask() is for creating transport transect masks because the python tool doesn't (yet) produce the edge sign.

I believe you have your own tools for mask creation. It would be great to converge on a set of interoperable tools for masking if possible. But this functionality is needed in compass and MPAS-Analysis urgently enough that I would like to go ahead with what I have for now.

dengwirda commented 3 years ago

@xylar, great that there's work happening re: the mask creator --- lots of opportunities to improve efficiency I think.

I'm happy to try to merge in various things from the ICoM workflow, I make use of a fast in-polygon routine + sparse-matrix treatments of various MPAS operators to improve algorithmic scaling. The ICoM workflows require keeping track of various additional constraints + non-ocean parts of the mesh, etc though, which may require some thought.

milenaveneziani commented 3 years ago

Hi @xylar: I've been using the MpasMaskcreator tool a ton recently, for creating ocean subbasins and new transects for the WC campaign. I would like to test these new tools: should I just call the new functions with a --help flag to see how to run them? Thanks for creating these.

xylar commented 3 years ago

@milenaveneziani, I always try to keep the documentation up-to-date: http://mpas-dev.github.io/MPAS-Tools/stable/mesh_conversion.html#mask-creation-with-python-multiprocessing Please let me know if the documentation needs work, though (or submit a PR of your own).

milenaveneziani commented 3 years ago

Fantastic. Thanks!

xylar commented 3 years ago

@dengwirda, inpoly looks really promising! I'll have to take a more careful look. It runs in serial, I assume? If it's fast, I suppose that doesn't matter.