Regridding functionalities (powered by xESMF)

[sol1105]

Pull Request Checklist:

• [x] This PR addresses an already opened issue (for bug fixes / features)

  • This PR closes #68, closes #180, closes #182, and closes #215
  • Follows up on #196, #141, #168. These will have to be addressed again at a later point

• [x] Tests for the changes have been added (for bug fixes / features)

• [x] Documentation has been added / updated (for bug fixes / features)

• [ ] HISTORY.rst has been updated (with summary of main changes) -> Will be added in a release preparing PR

• [x] I have added my relevant user information to AUTHORS.md

• What kind of change does this PR introduce?:

  • Extending #225 for all data_vars and coords of the xarray object (requires cf-xarray >= 0.7.5)
  • Adding jupyter notebook to showcase remapping functionalities
  • Adding regridding functionalities (powered by xESMF):
  • clisops.ops.regrid.regrid one line remapping function orchestrating below functions and classes
  • clisops.core.Grid class:
    • create xarray.Dataset holding description / coordinates of a regular lat-lon grid in CF compliant format:
    • from xarray.Dataset/DataArray (optionally of adaptive resolution, if source is not a regular lat-lon grid)
    • via grid_instructor (creating regional or global grid)
    • selecting a pre-defined grid (https://github.com/roocs/roocs-grids)
    • reformat grids (SCRIP, CF, xESMF formats)
    • detect extent, shape, format, type of the grid
    • detect collapsing or duplicated cells
    • create hash, compare grid objects
    • save to disk
    • exchange attributes and non-horizontal coordinates between datasets
    • calculate bounds (for regular lat-lon and curvilinear grids)
    • re-define data_vars and coords of xarray.Dataset for xESMF application
  • clisops.core.Weights:
    • orchestrate creation of weights with xESMF
    • holds xesmf.Regridder object
    • read from or store to local remapping weights cache (lock file mechanic for contemporary weights creation)
    • generate hash to identify similar weights
  • clisops.core.regrid:
    • application of remapping weights on xarray.Dataset/DataArray
    • optionally transfer attributes and non-horizontal coordinate variables from source to target dataset
    • set new attributes related to the remapping operation
  • Adding lock file mechanic (from source)

• Does this PR introduce a breaking change?:

  • adding regrid operator in ops
  • adding regrid function, Grid and Weights classes to core
  • the remapping makes use of xesmf, which is already a dependency
  • adding dependency for roocs-grids (might be removed with resolving #168)
  • potentially TBA

• Other information:

Future planned PR(s) specific for remapping:

• Support for manually provided masks
• Support for out-of-domain masking for nearest neighbour (likely better put to xesmf)
• Deal with the periodic attribute of xesmf (xesmf resets periodic to False for conservative remapping, but probably should not)
• Support datasets with shifted longitude frames (eg. ranging from (-60, 300) degrees_east) - base work already done in previous PR
• Support reformatting from/to further formats
• Support grid type detection for other formats
• Support reading / reformatting / using weight files from other tools like nco, cdo, ...
• Calculate nominal_resolution of the target grid, if not present
• Set up central / web based remapping weights cache and synchronizing cron job
• Find solution for vector variables / variables defined on cell edges
• Support vertical interpolation (eg. with xgcm)
• (Support other remapping backends than xesmf in the far future)

Future planned PR(s) generally for clisops:

• Unify the detect_coordinate functions
• Attribute a new tracking_id / PID (general requirement for clisops)
• Support datasets with missing missing_value / _FillValue attribute that feature missing values (add fix in dachar?)

[Zeitsperre]

Hey @sol1105, I'm wondering what the state of this PR is? Do you need reviewers?

[sol1105]

Hey @sol1105, I'm wondering what the state of this PR is? Do you need reviewers?

Hey @Zeitsperre, I'm sorry I wasn't able to continue to work on it in the past weeks, I try to get it into a reviewable state soon. Thanks for keeping it up to date with the master branch all this time :-)

[sol1105]

@Zeitsperre @cehbrecht @agstephens I did all the changes I wanted to make to the PR. It would be great, if you gave me some feedback. It is not urgent, however. In the background I will fix the tests that are no longer running through.

[coveralls]

Pull Request Test Coverage Report for Build 4629370587

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

• For more information on this, see Tracking coverage changes with pull request builds.
• To avoid this issue with future PRs, see these Recommended CI Configurations.
• For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

• 602 of 952 (63.24%) changed or added relevant lines in 8 files are covered.
• No unchanged relevant lines lost coverage.
• Overall first build on regrid-main at 70.677%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
clisops/ops/base_operation.py	23	30	76.67%
clisops/utils/common.py	14	22	63.64%
clisops/utils/output_utils.py	24	36	66.67%
clisops/utils/dataset_utils.py	131	163	80.37%
clisops/ops/regrid.py	20	57	35.09%
clisops/core/regrid.py	388	642	60.44%
<!--	Total:	602	952	63.24%	-->

Totals
Change from base Build 4619488998:	70.7%
Covered Lines:	1222
Relevant Lines:	1729

---

💛 - Coveralls

[Zeitsperre]

@sol1105

I wanted to know if you're still interested in finalizing the regridding implementation? I've adapted your PR to the newest package layout and will be seeing what I can do about roocs-grid (i.e. PyPI). Please let me know!

[sol1105]

@Zeitsperre @huard Thank you for all the help and feedback so far. I know there is still quite a lot of work to be done with regards to the remapping functionalities, but despite that I would like to merge this PR, so there can soon be a clisops release with regridding capabilities. From there I would keep on working on the regridding functionalities (incl. working on your above suggestions) in future and smaller PRs.

Would you be ok with this or would this conflict with your use case or plans with clisops? Is there anything crucial for me to do so a merge is possible? Problematic for your use case may be that two bugs (pydata/xarray#7794 and xarray-contrib/cf-xarray#442) prevent using the more recent xarray and cf-xarray releases. Another topic was that the roocs-grids repo (holding a set of pre-defined target grids) should be released as pypi and conda package before merging this PR.

[huard]

Hi Martin, works for me !

[sol1105]

@Zeitsperre @cehbrecht I removed python 3.8 from the CI checks because of the cf_xarray problem you adressed in roocs_utils using the following lines in the requirements.txt:

cf-xarray>=0.3.1,<=0.8.4; python_version == '3.8'
cf-xarray>=0.3.1; python_version >= '3.9'

I am not sure how such a setting can be added in the dependencies entry of the pyproject.toml, in case python 3.8 is still required. Would it simply be:

dependencies = [
  "bottleneck>=1.3.1",
  # cf-xarray is differently named on conda-forge
  "cf-xarray>=0.8.6;python_version>='3.9'",
  "cf-xarray>=0.7.5,<=0.8.0;python_version=='3.8'",
...

[Zeitsperre]

@sol1105

I am not sure how such a setting can be added in the dependencies entry of the pyproject.toml, in case python 3.8 is still required. Would it simply be:

dependencies = [
  "bottleneck>=1.3.1",
  # cf-xarray is differently named on conda-forge
  "cf-xarray>=0.8.6; python_version>='3.9'",
  "cf-xarray>=0.7.5,<=0.8.0; python_version=='3.8'",
...

That's exactly what you would need to do to maintain Python3.8 support. Everything is using pip under-the-hood. The only thing I'm not certain about is that I believe cf_xarray up until v0.8.4 is compatible with Python3.8 (at least, that's what I indicated in other places).

[cehbrecht]

@Zeitsperre I leave the last words to you :) We would like to merge this PR and make the new clisops release with the regrid operator. I have prepared daops and rook already for the new operator and it works.

[Zeitsperre]

Hey all,

Working on this now. I'm going to be adding a few changes to docstrings and updating some deprecated calls. I also might split the failing upstream build into its own workflow so that it isn't constantly failing. Will give everything a final overview after that's all done.

Should have something today.

[sol1105]

@Zeitsperre Thanks a lot for the review and your contribution. I addressed the outstanding issues you found

[Zeitsperre]

Well done. Feel free to merge whenever you're ready!