Consider replacing remapping with ESMPy

[xylar]

It looks like ESMF's python environment might support everything we need to create mapping files and do remapping from within python without ESMF_RegridWeightGen and presumably supporting all grids and meshes we might like, unlike ncremap.

http://www.earthsystemmodeling.org/esmf_releases/last_built/esmpy_doc/html/examples.html#read-and-write-a-weight-file

[milenaveneziani]

I asked Alice Barthel (@alicebarthel) about this because I knew she has been using ESMpy recently, without being completely satisfied. Here is her response:

Yes, it is what I have been using. It is a python interface to the ESMF core routines, so I’d stick with what you guys have already. But it’d be worth considering if writing new code, as it is more user-friendly to stay in the python environment in my opinion.

Ps. The writing to file is a feature of the most recent ESMPy() release.

To reply to this: so, are you saying that we would still need ESM_RegridWeightGen because ESMpy relies on it?

[xylar]

It could, indeed, be that ESMPy uses ESM_RegridWeightGen under the hood. It seems like it still might be worth trying out since I've been having such a hard time getting ESM_RegridWeightGen to work with parallel processes (and now with the latest NetCDF, too). But maybe the same problems would emerge, just obscured by one more level of interface.

I don't think the rewrite would be very hard because we have a lot of this code contained in one or two places. I'm not so much worried about the logistics or the time it might take as I am that it wouldn't fix the issues.

[xylar]

@alicebarthel, if you have anymore advice here, I'd be keen to hear it.

[alicebarthel]

I reckon the rewrite would be straightforward, unless you use options ESMPy() is not ready for. My comment was more about whether you'd gain something from the rewrite.
I haven't used it for parallel processes, so can't speak to that.

I do know that Ben Koziol has been working on spatial decomposition to minimize memory load for large grids. Might be of interest. http://ocgis.readthedocs.io/en/latest/cli.html#chunked-regrid-weight-generation

[xylar]

Thanks, @alicebarthel. I also had a conversation with @jhkennedy and he says that ESMPy is just using ESMF command-line tools under the hood, which means that it obviously won't make the issues I've been having with trying to run ESMF_RegridWeightGen several times in parallel any better.

I'm a little skittish about adopting any tools that are works in progress but I'll definitely take a look at ocli chunked_rwg. My issue isn't about memory limitations but if that tool has the side effect of running without conflicts in several different python processes, that would be very helpful.