pySTEPS / pysteps

Python framework for short-term ensemble prediction systems.

https://pysteps.github.io/

BSD 3-Clause "New" or "Revised" License

466 stars 168 forks source link

Bom import xarray #228

Closed cvelascof closed 3 years ago

cvelascof commented 3 years ago

This PR reads directly BoM netCDF files using xarray and returns a DataArray. Also it implements some tests for the new importer based on xarray.

cvelascof commented 3 years ago

This PR uses xarray to load the BoM netcdf files and adds the pySTEPS metadata as attributes to the DataArrays (Data and coordinates). New routines were created and named as the old ones plus the postfix "_xr". Old routines are still in the repository for legacy purposes. All tests related with the new routines are passing.

Test failing are related with io_import_mrms_grib routine. @dnerini . @RubenImhoff can you give it a look and try to fix them?

RubenImhoff commented 3 years ago

The failing MRMS tests seems like something @aperezhortal knows.

codecov[bot] commented 3 years ago

Codecov Report

:exclamation: No coverage uploaded for pull request base (pysteps-v2@3802f8d). Click here to learn what that means. The diff coverage is n/a.

@@              Coverage Diff              @@
##             pysteps-v2     #228   +/-   ##
=============================================
  Coverage              ?   73.41%           
=============================================
  Files                 ?      144           
  Lines                 ?    10892           
  Branches              ?        0           
=============================================
  Hits                  ?     7996           
  Misses                ?     2896           
  Partials              ?        0

Flag	Coverage Δ
unit_tests	`73.41% <0.00%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 3802f8d...0847846. Read the comment docs.

cvelascof commented 3 years ago

hi @ladc ... good point ... _import_bom_rf3_geodata_xr already have varname as parameter but it would be nice to connect it with kwargs passed to the parent function import_bom_rf3_xr. I will do that soon.

On the other hand, importers.py is in theory only for radar inputs ... so probably we may need a new nwp_importers.py to collect the specific routines to read and preprocess nwp datasets. For example, deal with lat/lon rather than x and y, or transform accumulated precipitation from runtime that it is typical for NWP variable to accumulated precipitation for each time step that it is what actually will be blended with radar data. What do you think?

cvelascof commented 3 years ago

it seems chunks are not working in python3.6 for some reason but they work great using python3.8... so at this point I will not pass chunks information to xarray. Something to think about later on.

cvelascof commented 3 years ago

@ladc this adds an importer for some of the sample nwp data in pysteps-data. We may need to add this nwp reader to other branches. This is just the foundation for additional NWP data readers.

ladc commented 3 years ago

@cvelascof Great! I don't know how realistic it is to make a generic NWP importer function that can read CF-compliant netcdf files but this looks like a good start.