search

pydata / xarray

N-D labeled arrays and datasets in Python
https://xarray.dev
Apache License 2.0
3.56k stars 1.07k forks source link

Custom Table when opening GRIB Files #3853

Open pgierz opened 4 years ago

pgierz commented 4 years ago

Hello,

I'd like to open some old-school Grib files from one of our climate models. I'm using the PyNIO backend for this; which works pretty well so far -- at least the data opens! However, I am getting errors:

warning:NclGRIB: Unrecognized parameter table (center 252, subcenter 1, table 128), defaulting to NCEP operational table: variable names and units may be incorrect

So, would it somehow be possible to provide a code table to be used when opening grb files? I have files next to my output where the codes are stored. An example is below.

I can imagine something like:

ds = xr.open_dataset("/path/to/file.grb", engine="pynio", code_table="/path/to/codetab")

Would this be difficult to implement? Cheers, Paul

130  47 st                                                                  0.00    1.00 temperature [K]
 138  47 svo                                                                 0.00    1.00 vorticity [1/s]
 152   1 lsp                                                                 0.00    1.00 log surface pressure []
 155  47 sd                                                                  0.00    1.00 divergence [1/s]
 133  47 q                                                                   0.00    1.00 specific humidity [kg/kg]
 153  47 xl                                                                  0.00    1.00 cloud water [kg/kg]
 154  47 xi                                                                  0.00    1.00 cloud ice [kg/kg]
  50   1 rsdscs                                                              0.00    1.00 surface downwelling shortwave radiation flux (clear-sky) [W/m**2]
  51   1 rsuscs                                                              0.00    1.00 surface upwelling shortwave radiation flux (clear-sky) [W/m**2]
  52   1 rsdscs_na                                                           0.00    1.00 instantaneous surface downwelling shortwave radiation flux (clear-sky) [W/m**2]
  53   1 rsuscs_na                                                           0.00    1.00 instantaneous surface upwelling shortwave radiation flux (clear-sky) [W/m**2]
  54   1 q2m                                                                 0.00    1.00 2m specific humidity []
  55   1 rh2m                                                                0.00    1.00 2m relative humidity []
  56   1 rsdsiac                                                             0.00    1.00 surface downwelling shortwave radiation over ice [W/m**2]
  57   1 rsdswac                                                             0.00    1.00 surface downwelling shortwave radiation over water [W/m**2]
  58   1 rsusiac                                                             0.00    1.00 surface upwelling shortwave radiation over ice [W/m**2]
  59   1 rsuswac                                                             0.00    1.00 surface upwelling shortwave radiation over water [W/m**2]
  60   1 rldsiac                                                             0.00    1.00 surface downwelling longwave radiation over ice [W/m**2]
  61   1 rldswac                                                             0.00    1.00 surface downwelling longwave radiation over water [W/m**2]
  62   1 rlusiac                                                             0.00    1.00 surface upwelling longwave radiation over ice [W/m**2]
  63   1 rluswac                                                             0.00    1.00 surface upwelling longwave radiation over water [W/m**2]
  64   1 sh_vdiff                                                            0.00    1.00 column heating due to vertical diffusion [W/m**2]
  65   1 ev_vdiff                                                            0.00    1.00 column moistening due to vertical diffusion [kg/m**2s]
  66   1 ch_concloud                                                         0.00    1.00 convective heating [W/m**s]
  67   1 cw_concloud                                                         0.00    1.00 convective moistening [kg/m**2s]
  68   1 fage                                                                0.00    1.00 aging factor of snow on ice []
  69   1 snifrac                                                             0.00    1.00 fraction of ice covered with snow []
  70   1 barefrac                                                            0.00    1.00 bare ice fraction []
  71   1 alsom                                                               0.00    1.00 albedo of melt ponds []
  72   1 alsobs                                                              0.00    1.00 albedo of bare ice and snow without ponds []
  73   1 sicepdw                                                             0.00    1.00 melt pond depth on sea-ice [m]
  74   1 sicepdi                                                             0.00    1.00 ice thickness on melt pond [m]
  75   1 tsicepdi                                                            0.00    1.00 ice temperature on frozen melt pond [K]
  76   1 sicepres                                                            0.00    1.00 residual heat flux [W/m**2]
  77   1 ameltdepth                                                          0.00    1.00 total melt pond depth [m]
  78   1 ameltfrac                                                           0.00    1.00 fract area of melt ponds on sea-ice []
  79   1 albedo_vis_dir                                                      0.00    1.00 surface albedo visible range direct []
  80   1 albedo_nir_dir                                                      0.00    1.00 surface albedo NIR range direct []
  81   1 albedo_vis_dif                                                      0.00    1.00 surface albedo visible range diffuse []
  82   1 albedo_nir_dif                                                      0.00    1.00 surface albedo NIR range diffuse []
  83   1 ocu                                                                 0.00    1.00 ocean eastward velocity [m/s]
  84   1 ocv                                                                 0.00    1.00 ocean northward velocity [m/s]
  85   1 tradl                                                               0.00    1.00 thermal radiation 200mb [W/m**2]
  86   1 sradl                                                               0.00    1.00 solar radiation 200mb [W/m**2]
  87   1 trafl                                                               0.00    1.00 thermal radiation 200mb (clear sky) [W/m**2]
  88   1 srafl                                                               0.00    1.00 solar radiation 200mb (clear sky) [W/m**2]
  89   1 amlcorac                                                            0.00    1.00 mixed layer flux correction [W/m**2]
  91   1 trfliac                                                             0.00    1.00 LW flux over ice [W/m**2]
  92   1 trflwac                                                             0.00    1.00 LW flux over water [W/m**2]
  93   1 trfllac                                                             0.00    1.00 LW flux over land [W/m**2]
  94   1 sofliac                                                             0.00    1.00 SW flux over ice [W/m**2]
  95   1 soflwac                                                             0.00    1.00 SW flux over water [W/m**2]
  96   1 sofllac                                                             0.00    1.00 SW flux over land [W/m**2]
  97   1 friac                                                               0.00    1.00 ice cover (fraction of grid box) []
 100   1 albedo_vis                                                          0.00    1.00 surface albedo visible range []
 101   1 albedo_nir                                                          0.00    1.00 surface albedo NIR range []
 102   1 tsi                                                                 0.00    1.00 surface temperature of ice [K]
 103   1 tsw                                                                 0.00    1.00 surface temperature of water [K]
 104   1 ustri                                                               0.00    1.00 zonal      wind stress over ice [Pa]
 105   1 vstri                                                               0.00    1.00 meridional wind stress over ice [Pa]
 106   1 ustrw                                                               0.00    1.00 zonal      wind stress over water [Pa]
 107   1 vstrw                                                               0.00    1.00 meridional wind stress over water [Pa]
 108   1 ustrl                                                               0.00    1.00 zonal      wind stress over land [Pa]
 109   1 vstrl                                                               0.00    1.00 meridional wind stress over land [Pa]
 110   1 ahfliac                                                             0.00    1.00 latent heat flux over ice [W/m**2]
 111   1 ahflwac                                                             0.00    1.00 latent heat flux over water [W/m**2]
 112   1 ahfllac                                                             0.00    1.00 latent heat flux over land [W/m**2]
 113   1 evapiac                                                             0.00    1.00 evaporation over ice [kg/m**2s]
 114   1 evapwac                                                             0.00    1.00 evaporation over water [kg/m**2s]
 115   1 evaplac                                                             0.00    1.00 evaporation over land [kg/m**2s]
 116   1 az0i                                                                0.00    1.00 roughness length over ice [m]
 117   1 az0w                                                                0.00    1.00 roughness length over water [m]
 118   1 az0l                                                                0.00    1.00 roughness length over land [m]
 119   1 ahfsiac                                                             0.00    1.00 sensible heat flux over ice [W/m**2]
 120   1 ahfswac                                                             0.00    1.00 sensible heat flux over water [W/m**2]
 121   1 ahfslac                                                             0.00    1.00 sensible heat flux over land [W/m**2]
 122   1 alsoi                                                               0.00    1.00 albedo of ice []
 123   1 alsow                                                               0.00    1.00 albedo of water []
 124   1 alsol                                                               0.00    1.00 albedo of land []
 125   1 ahfice                                                              0.00    1.00 conductive heat flux [W/m**2]
 126   1 qres                                                                0.00    1.00 residual heat flux for melting sea ice [W/m**2]
 129   1 geosp                                                               0.00    1.00 surface geopotential (orography) [m**2/s**2]
 134   1 aps                                                                 0.00    1.00 surface pressure [Pa]
 137   1 apmeb                                                               0.00    1.00 vertic integr tendenc of water [kg/m**2s]
 139   1 tslm1                                                               0.00    1.00 surface temperature of land [K]
 140   1 ws                                                                  0.00    1.00 soil wetness [m]
 141   1 sn                                                                  0.00    1.00 snow depth [m]
 142   1 aprl                                                                0.00    1.00 large scale precipitation [kg/m**2s]
 143   1 aprc                                                                0.00    1.00 convective  precipitation [kg/m**2s]
 144   1 aprs                                                                0.00    1.00 snow fall [kg/m**2s]
 145   1 vdis                                                                0.00    1.00 boundary layer dissipation [W/m**2]
 146   1 ahfs                                                                0.00    1.00 sensible heat flux [W/m**2]
 147   1 ahfl                                                                0.00    1.00 latent heat flux [W/m**2]
 150   1 xivi                                                                0.00    1.00 vertically integrated cloud ice [kg/m**2]
 157  47 relhum                                                              0.00    1.00 relative humidity []
 164   1 aclcov                                                              0.00    1.00 total cloud cover []
 165   1 u10                                                                 0.00    1.00 10m u-velocity [m/s]
 166   1 v10                                                                 0.00    1.00 10m v-velocity [m/s]
 167   1 temp2                                                               0.00    1.00 2m temperature [K]
 168   1 dew2                                                                0.00    1.00 2m dew point temperature [K]
 169   1 tsurf                                                               0.00    1.00 surface temperature [K]
 171   1 wind10                                                              0.00    1.00 10m windspeed [m/s]
 172   1 slm                                                                 0.00    1.00 land sea mask (1=land, 0=sea/lakes) []
 175   1 albedo                                                              0.00    1.00 surface albedo []
 176   1 srads                                                               0.00    1.00 net surface solar radiation [W/m**2]
 177   1 trads                                                               0.00    1.00 net surface thermal radiation [W/m**2]
 178   1 srad0                                                               0.00    1.00 net top solar radiation [W/m**2]
 179   1 trad0                                                               0.00    1.00 top thermal radiation (OLR) [W/m**2]
 180   1 ustr                                                                0.00    1.00 u-stress [Pa]
 181   1 vstr                                                                0.00    1.00 v-stress [Pa]
 182   1 evap                                                                0.00    1.00 evaporation [kg/m**2s]
 184   1 srad0d                                                              0.00    1.00 top incoming solar radiation [W/m**2]
 185   1 srafs                                                               0.00    1.00 net surface solar radiation (clear sky) [W/m**2]
 186   1 trafs                                                               0.00    1.00 net surface therm radiation (clear sky) [W/m**2]
 187   1 sraf0                                                               0.00    1.00 net top solar radiation     (clear sky) [W/m**2]
 188   1 traf0                                                               0.00    1.00 net top thermal radiation   (clear sky) [W/m**2]
 193   1 wl                                                                  0.00    1.00 skin reservoir content [m]
 194   1 slf                                                                 0.00    1.00 sea land fraction []
 197   1 vdisgw                                                              0.00    1.00 gravity wave dissipation [W/m**2]
 201   1 t2max                                                               0.00    1.00 maximum 2m temperature [K]
 202   1 t2min                                                               0.00    1.00 minimum 2m temperature [K]
 203   1 srad0u                                                              0.00    1.00 top solar radiation upward [W/m**2]
 204   1 sradsu                                                              0.00    1.00 surface solar radiation upward [W/m**2]
 205   1 tradsu                                                              0.00    1.00 surface thermal radiation upward [W/m**2]
 208   1 ahfcon                                                              0.00    1.00 conductive heat flux through ice [W/m**2]
 209   1 ahfres                                                              0.00    1.00 melting of ice [W/m**2]
 210   1 seaice                                                              0.00    1.00 ice cover (fraction of 1-SLM) []
 211   1 siced                                                               0.00    1.00 ice depth [m]
 213   1 gld                                                                 0.00    1.00 glacier depth [m]
 214   1 sni                                                                 0.00    1.00 water equivalent of snow on ice [m]
 216   1 wimax                                                               0.00    1.00 maximum 10m-wind speed [m/s]
 217   1 topmax                                                              0.00    1.00 max height of conv cloud tops [Pa]
 223  47 aclcac                                                              0.00    1.00 cloud cover []
 229   1 wsmx                                                                0.00    1.00 field capacity of soil [m]
 230   1 qvi                                                                 0.00    1.00 vertically integrated water vapor [kg/m**2]
 231   1 xlvi                                                                0.00    1.00 vertically integrated cloud water [kg/m**2]
 232   1 glac                                                                0.00    1.00 fraction of land covered by glaciers []
 233   1 snc                                                                 0.00    1.00 snow depth at the canopy [m]
 235   1 abso4                                                               0.00    1.00 antropogenic sulfur burden [kg/m**2]
 236  47 ao3                                                                 0.00    1.00 ipcc ozone [kg/kg]
 237   1 tropo                                                               0.00    1.00 WMO defined tropopause height [Pa]
 238   1 thvsig                                                              0.00    1.00 stddev virt pot temp at halflev klevm1 [K]
 239  47 tpot                                                                0.00    1.00 potential temperature [K]
dcherian commented 4 years ago

Is there a way to tell pynio to use a specific file?

If so, something like xr.open_dataset("/path/to/file.grb", engine="pynio", backend_kwargs=dict(code_table="/path/to/codetab")) should work

pgierz commented 4 years ago

Apparently this is possible by setting an environmental variable: http://www.pyngl.ucar.edu/NioFormats.shtml#GRIB1-user-defined-parameter-tables

So, if I have my codetable at /path/to/codetab; I can set NIO_GRIB_PTABLE_PATH=/path/to/codetab. I'm honestly not too happy with this solution. Would it be possible to add something like backend_env to open_dataset; where I can specify environmental variables that are active during opening, and then restore the environment afterwards?

aaronspring commented 4 years ago

I would love that feature. Also found that pyngl docu but didn’t get it working...

pgierz commented 4 years ago

I can have a look at implementing this. Might be a nice opportunity to actually contribute to some of the packages I use every day ;-)

aaronspring commented 4 years ago

I would use it with the echam6 and jsbach ‘.codes’ files in MPIESM.

pgierz commented 4 years ago

Me too! Then we have a good test case :-)

aaronspring commented 4 years ago

This was my workaround. https://github.com/aaronspring/pymistral/issues/19

aaronspring commented 2 years ago

It is possible to set ECCODES_DEFINITION_PATH in eccodes, see http://download.ecmwf.int/test-data/eccodes/html/namespaces.html. But the codetab you want to use here from MPIESM grb output is a CDO parameter table, see https://code.mpimet.mpg.de/projects/cdo/embedded/cdo.pdf#section.1.8, so in my understanding something CDO internal and likely not usable with eccodes/cfgrib.