Closed xylar closed 5 years ago
I asked Alice Barthel (@alicebarthel) about this because I knew she has been using ESMpy recently, without being completely satisfied. Here is her response:
Yes, it is what I have been using. It is a python interface to the ESMF core routines, so I’d stick with what you guys have already. But it’d be worth considering if writing new code, as it is more user-friendly to stay in the python environment in my opinion.
Ps. The writing to file is a feature of the most recent ESMPy() release.
To reply to this: so, are you saying that we would still need ESM_RegridWeightGen
because ESMpy relies on it?
It could, indeed, be that ESMPy uses ESM_RegridWeightGen
under the hood. It seems like it still might be worth trying out since I've been having such a hard time getting ESM_RegridWeightGen
to work with parallel processes (and now with the latest NetCDF, too). But maybe the same problems would emerge, just obscured by one more level of interface.
I don't think the rewrite would be very hard because we have a lot of this code contained in one or two places. I'm not so much worried about the logistics or the time it might take as I am that it wouldn't fix the issues.
@alicebarthel, if you have anymore advice here, I'd be keen to hear it.
I reckon the rewrite would be straightforward, unless you use options ESMPy() is not ready for. My comment was more about whether you'd gain something from the rewrite.
I haven't used it for parallel processes, so can't speak to that.
I do know that Ben Koziol has been working on spatial decomposition to minimize memory load for large grids. Might be of interest. http://ocgis.readthedocs.io/en/latest/cli.html#chunked-regrid-weight-generation
Thanks, @alicebarthel. I also had a conversation with @jhkennedy and he says that ESMPy is just using ESMF command-line tools under the hood, which means that it obviously won't make the issues I've been having with trying to run ESMF_RegridWeightGen
several times in parallel any better.
I'm a little skittish about adopting any tools that are works in progress but I'll definitely take a look at ocli chunked_rwg
. My issue isn't about memory limitations but if that tool has the side effect of running without conflicts in several different python processes, that would be very helpful.
It looks like ESMF's python environment might support everything we need to create mapping files and do remapping from within python without
ESMF_RegridWeightGen
and presumably supporting all grids and meshes we might like, unlikencremap
.http://www.earthsystemmodeling.org/esmf_releases/last_built/esmpy_doc/html/examples.html#read-and-write-a-weight-file