search

ecmwf / cfgrib

A Python interface to map GRIB files to the NetCDF Common Data Model following the CF Convention using ecCodes
Apache License 2.0
391 stars 75 forks source link

KeyError: 'GRIB_name' - Problem reading NCEP NGAC data #16

Closed bbakernoaa closed 5 years ago

bbakernoaa commented 5 years ago

First off I think this project is an awesome idea and I am interested.

I'm trying this out on some of the NCEP data. Particularly the NGAC data.

http://www.ftp.ncep.noaa.gov/data/nccf/com/ngac/prod/

I am trying to read in some of the AOD variables and such but am getting an error

from cfgrib import xarray_store

xarray_store.open_dataset('ngac.t00z.a2df105.grib2',filter_by_keys={'typeOfLevel':'atmosphere'})
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-79-1033cece9edc> in <module>()
----> 1 test = xarray_store.open_dataset('ngac.t00z.a2df105.grib2',filter_by_keys={'typeOfLevel':'atmosphere'})

/naqfc/save/Barry.Baker/envs/monet/lib/python3.6/site-packages/cfgrib/xarray_store.py in open_dataset(path, flavour_name, filter_by_keys, errors, **kwargs)
    161         if k.startswith('encode_'):
    162             overrides[k] = kwargs.pop(k)
--> 163     store = GribDataStore.frompath(path, **overrides)
    164     return _open_dataset(store, **kwargs)
    165 

/naqfc/save/Barry.Baker/envs/monet/lib/python3.6/site-packages/cfgrib/xarray_store.py in frompath(cls, path, flavour_name, errors, **kwargs)
    102         config = flavour.pop('dataset', {}).copy()
    103         config.update(kwargs)
--> 104         return cls(ds=cfgrib.Dataset.frompath(path, errors=errors, **config), **flavour)
    105 
    106     def __attrs_post_init__(self):

/naqfc/save/Barry.Baker/envs/monet/lib/python3.6/site-packages/cfgrib/dataset.py in frompath(cls, path, mode, errors, **kwargs)
    379     def frompath(cls, path, mode='r', errors='ignore', **kwargs):
    380         stream = messages.Stream(path, mode, message_class=cfmessage.CfMessage, errors=errors)
--> 381         return cls(stream=stream, **kwargs)
    382 
    383     def __attrs_post_init__(self):

<attrs generated init c839a147c90eb3321ada82313ce86e3ade1b1758> in __init__(self, stream, encode_parameter, encode_time, encode_vertical, encode_geography, filter_by_keys)
      6     self.encode_geography = encode_geography
      7     self.filter_by_keys = filter_by_keys
----> 8     self.__attrs_post_init__()

/naqfc/save/Barry.Baker/envs/monet/lib/python3.6/site-packages/cfgrib/dataset.py in __attrs_post_init__(self)
    382 
    383     def __attrs_post_init__(self):
--> 384         dims, vars, attrs = build_dataset_components(**self.__dict__)
    385         self.dimensions = dims  # type: T.Dict[str, T.Optional[int]]
    386         self.variables = vars  # type: T.Dict[str, Variable]

/naqfc/save/Barry.Baker/envs/monet/lib/python3.6/site-packages/cfgrib/dataset.py in build_dataset_components(stream, encode_parameter, encode_time, encode_vertical, encode_geography, filter_by_keys)
    352         var_index = index.subindex(paramId=param_id)
    353         dims, data_var, coord_vars = build_data_var_components(
--> 354             var_index, encode_parameter, encode_time, encode_geography, encode_vertical,
    355         )
    356         if encode_parameter and var_name not in ('undef', 'unknown'):

/naqfc/save/Barry.Baker/envs/monet/lib/python3.6/site-packages/cfgrib/dataset.py in build_data_var_components(index, encode_parameter, encode_time, encode_geography, encode_vertical, log)
    268         if data_var_attrs.get('GRIB_cfName'):
    269             data_var_attrs['standard_name'] = data_var_attrs['GRIB_cfName']
--> 270         data_var_attrs['long_name'] = data_var_attrs['GRIB_name']
    271         data_var_attrs['units'] = data_var_attrs['GRIB_units']
    272 

KeyError: 'GRIB_name'

Now these variables are present in the NCEP grib tables but seems to be missing the GRIB_name attribute. From wgrib2

$ wgrib2 colmd.grb2 -set_ext_name 1 -netcdf junk_ext_name.nc
1:0:d=2016081200:COLMD.aerosol=Total_Aerosol.aerosol_size_<1e-05.:entire atmosphere:anl:
2:71357:d=2016081200:COLMD.aerosol=Total_Aerosol.aerosol_size_<2.5e-06.:entire atmosphere:anl:
3:160149:d=2016081200:COLMD.aerosol=Dust_Dry.aerosol_size_<2.5e-06.:entire atmosphere:anl:
4:162022:d=2016081200:COLMD.aerosol=Sea_Salt_Dry.aerosol_size_<2.5e-06.:entire atmosphere:anl:
5:226224:d=2016081200:COLMD.aerosol=Black_Carbon_Dry.aerosol_size_<2.36e-08.:entire atmosphere:anl:
6:275933:d=2016081200:COLMD.aerosol=Particulate_Organic_Matter_Dry.aerosol_size_<4.24e-08.:entire atmosphere:anl:
7:312737:d=2016081200:COLMD.aerosol=Sulphate_Dry.aerosol_size_<2.5e-06.:entire atmosphere:anl:
-bash-4.2$ wgrib2 ngac.t00z.a2df105.grib2 
1:0:d=2018091200:ASYSFK:entire atmosphere:105 hour fcst:aerosol=Total Aerosol:aerosol_size <2e-05:aerosol_wavelength >=3.38e-07,<=3.42e-07
2:88995:d=2018091200:SSALBK:entire atmosphere:105 hour fcst:aerosol=Total Aerosol:aerosol_size <2e-05:aerosol_wavelength >=3.38e-07,<=3.42e-07
3:173734:d=2018091200:AOTK:entire atmosphere:105 hour fcst:aerosol=Total Aerosol:aerosol_size <2e-05:aerosol_wavelength >=5.45e-07,<=5.65e-07
4:257495:d=2018091200:AOTK:entire atmosphere:105 hour fcst:aerosol=Dust Dry:aerosol_size <2e-05:aerosol_wavelength >=5.45e-07,<=5.65e-07
5:308070:d=2018091200:AOTK:entire atmosphere:105 hour fcst:aerosol=Sea Salt Dry:aerosol_size <2e-05:aerosol_wavelength >=5.45e-07,<=5.65e-07
6:381840:d=2018091200:AOTK:entire atmosphere:105 hour fcst:aerosol=Sulphate Dry:aerosol_size <7e-07:aerosol_wavelength >=5.45e-07,<=5.65e-07
7:434792:d=2018091200:AOTK:entire atmosphere:105 hour fcst:aerosol=Particulate Organic Matter Dry:aerosol_size <7e-07:aerosol_wavelength >=5.45e-07,<=5.65e-07
8:496510:d=2018091200:AOTK:entire atmosphere:105 hour fcst:aerosol=Black Carbon Dry:aerosol_size <7e-07:aerosol_wavelength >=5.45e-07,<=5.65e-07
9:551432:d=2018091200:AEMFLX:entire atmosphere:105 hour fcst:aerosol=Dust Dry:aerosol_size <2e-05:
10:559057:d=2018091200:SEDMFLX:entire atmosphere:105 hour fcst:aerosol=Dust Dry:aerosol_size <2e-05:
11:598927:d=2018091200:DDMFLX:entire atmosphere:105 hour fcst:aerosol=Dust Dry:aerosol_size <2e-05:
12:646148:d=2018091200:WLSMFLX:entire atmosphere:105 hour fcst:aerosol=Dust Dry:aerosol_size <2e-05:
13:684570:d=2018091200:WDCPMFLX:entire atmosphere:105 hour fcst:aerosol=Dust Dry:aerosol_size <2e-05:
14:718763:d=2018091200:AEMFLX:entire atmosphere:105 hour fcst:aerosol=Sea Salt Dry:aerosol_size <2e-05:
15:766737:d=2018091200:SEDMFLX:entire atmosphere:105 hour fcst:aerosol=Sea Salt Dry:aerosol_size <2e-05:
16:830919:d=2018091200:DDMFLX:entire atmosphere:105 hour fcst:aerosol=Sea Salt Dry:aerosol_size <2e-05:
17:897610:d=2018091200:WLSMFLX:entire atmosphere:105 hour fcst:aerosol=Sea Salt Dry:aerosol_size <2e-05:
18:952870:d=2018091200:WDCPMFLX:entire atmosphere:105 hour fcst:aerosol=Sea Salt Dry:aerosol_size <2e-05:
19:1000818:d=2018091200:AEMFLX:entire atmosphere:105 hour fcst:aerosol=Black Carbon Dry:aerosol_size <2.36e-08:
20:1020808:d=2018091200:SEDMFLX:entire atmosphere:105 hour fcst:aerosol=Black Carbon Dry:aerosol_size <7e-07:
21:1066505:d=2018091200:DDMFLX:entire atmosphere:105 hour fcst:aerosol=Black Carbon Dry:aerosol_size <7e-07:
22:1104627:d=2018091200:WLSMFLX:entire atmosphere:105 hour fcst:aerosol=Black Carbon Dry:aerosol_size <7e-07:
23:1140259:d=2018091200:WDCPMFLX:entire atmosphere:105 hour fcst:aerosol=Black Carbon Dry:aerosol_size <7e-07:
24:1168276:d=2018091200:AEMFLX:entire atmosphere:105 hour fcst:aerosol=Particulate Organic Matter Dry:aerosol_size <7e-07:
25:1186046:d=2018091200:SEDMFLX:entire atmosphere:105 hour fcst:aerosol=Particulate Organic Matter Dry:aerosol_size <7e-07:
26:1231200:d=2018091200:DDMFLX:entire atmosphere:105 hour fcst:aerosol=Particulate Organic Matter Dry:aerosol_size <7e-07:
27:1280346:d=2018091200:WLSMFLX:no_level:105 hour fcst:aerosol=Particulate Organic Matter Dry:aerosol_size <7e-07:
28:1327429:d=2018091200:WDCPMFLX:no_level:105 hour fcst:aerosol=Particulate Organic Matter Dry:aerosol_size <7e-07:
29:1359326:d=2018091200:MASSDEN:surface:105 hour fcst:aerosol=Dust Dry:aerosol_size <1e-05:
30:1391056:d=2018091200:MASSDEN:surface:105 hour fcst:aerosol=Dust Dry:aerosol_size <2.5e-06:
31:1427968:d=2018091200:COLMD:entire atmosphere:105 hour fcst:aerosol=Dust Dry:aerosol_size <1e-05:
32:1468168:d=2018091200:COLMD:entire atmosphere:105 hour fcst:aerosol=Dust Dry:aerosol_size <2.5e-06:
alexamici commented 5 years ago

Thanks for reporting the issue!

The bug itself is trivial to fix and I will make a minor release soon. In the meanwhile you can try to pass the encode_parameter=False option to xarray_store.open_dataset.

alexamici commented 5 years ago

@bbakernoaa please upgrade with pip install cfgrib>=0.8.4.5 and test if it solves your issue. Thanks!

bbakernoaa commented 5 years ago

@alexamici I am a little worried. It opens a single variable under atmosphere, a single variable under surface, and no variables under the 'unknown' typeOfLevel. I'm trying to figure out if this is an issue with how NCEP is writing the grib data or if there is something that can be done in cfgrib.

alexamici commented 5 years ago

@bbakernoaa this is a different issue, right? The KeyError: 'GRIB_name' is solved.

I'll have a look at it and open a new issue if it is needed. I'll close this one for now.

bbakernoaa commented 5 years ago

@alexamici Yes I suppose this is a separate issue. The KeyError: 'Grib_name' itself has been resolved

alexamici commented 5 years ago

@bbakernoaa I'd say ecCodes itself has no support for this kind of GRIB2 files, very few key values are recognised. All variables are mixed up in the single variable unknown and I think the resulting Dataset is unusable.

As cfgrib relies on ecCodes for GRIB decoding there is not much that I can do except filing a bug report to ecCodes.

Thanks again for reporting.

bbakernoaa commented 5 years ago

I’m talking with the guys in NCEP now. I’m convinced that this is an issue with how they are storing the data in the files. Seeing as our group is the only operational usage of this product no one has problably complained before or checked into the specifics. I’ll keep you posted on what I can find out.

Other variables that are not atmospheric composition types do not seem to have issue.

On Sat, Sep 15, 2018 at 11:20 AM Alessandro Amici notifications@github.com wrote:

@bbakernoaa https://github.com/bbakernoaa I'd say ecCodes itself has no support for this kind of GRIB2 files, very few key values are recognised. All variables are mixed up in the single variable unknown and I think the resulting Dataset is unusable.

As cfgrib relies on ecCodes for GRIB decoding there is not much that I can do except filing a bug report to ecCodes.

Thanks again for reporting.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ecmwf/cfgrib/issues/16#issuecomment-421586002, or mute the thread https://github.com/notifications/unsubscribe-auth/AVFKt5we_C7mo-qlpbnHUX3kS36HF50Aks5ubRrRgaJpZM4WoDHv .