openclimatefix / ocf_datapipes

OCF's DataPipe based dataloader for training and inference
MIT License
13 stars 11 forks source link

Add support for ICON Global #221

Closed jacobbieker closed 1 year ago

jacobbieker commented 1 year ago

Pull Request

Description

This adds support for ICON Global selection, as the variables are in a icosahedral grid, and so is not regular, unlike ICON-EU, and other NWP providers.

This also adds support for collapsing the pressure levels of the input NWPs down to a single DataArray with a MultiIndex.

This PR leaves dealing with the unstructured grid of ICON Global to a later PR. Selecting spatial slices does work, but the grid would need to be re-gridded for many of the other parts of ocf_datapipes to function properly with it, as they expect regular grids.

DWD does have an official regridding Docker container, which would be of some help for that issue to do it properly: https://github.com/DeutscherWetterdienst/regrid

Fixes #222

How Has This Been Tested?

Unit Tests

Checklist:

codecov[bot] commented 1 year ago

Codecov Report

Merging #221 (e6ea4fd) into main (7a4db72) will increase coverage by 0.24%. Report is 1 commits behind head on main. The diff coverage is 93.22%.

@@            Coverage Diff             @@
##             main     #221      +/-   ##
==========================================
+ Coverage   81.22%   81.47%   +0.24%     
==========================================
  Files         131      132       +1     
  Lines        5641     5690      +49     
==========================================
+ Hits         4582     4636      +54     
+ Misses       1059     1054       -5     
Files Changed Coverage Δ
ocf_datapipes/select/select_spatial_slice.py 79.19% <85.18%> (+1.77%) :arrow_up:
ocf_datapipes/load/nwp/providers/icon.py 100.00% <100.00%> (+15.38%) :arrow_up:
ocf_datapipes/transform/xarray/__init__.py 100.00% <100.00%> (ø)
...y/convert_pressure_levels_to_separate_variables.py 100.00% <100.00%> (ø)

... and 3 files with indirect coverage changes

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more