Unable to read NCEP MRMS grib2 files

[dmfay]

(from https://github.com/ecmwf/cfgrib/issues/116 )

I'm on OSX, trying to work with NCEP MultiRadar MultiSensor grib2s such as in https://mrms.ncep.noaa.gov/data/2D/MergedReflectivityQCComposite/ , and always getting the following output/traceback reading files in:

$ cfgrib to_netcdf MRMS_MergedReflectivityQCComposite_00.50_20191209-210032.grib2
libpng warning: Application built with libpng-1.4.12 but running with 1.6.37
ECCODES ERROR   :  unable to get values as double array (Decoding invalid)
ECCODES ERROR   :  unable to create iterator
Traceback (most recent call last):
  File "/Users/dian/work/anaconda/envs/mypy3env/bin/cfgrib", line 10, in <module>
    sys.exit(cfgrib_cli())
  File "/Users/dian/work/anaconda/envs/mypy3env/lib/python3.8/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/Users/dian/work/anaconda/envs/mypy3env/lib/python3.8/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/dian/work/anaconda/envs/mypy3env/lib/python3.8/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/dian/work/anaconda/envs/mypy3env/lib/python3.8/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/dian/work/anaconda/envs/mypy3env/lib/python3.8/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/dian/work/anaconda/envs/mypy3env/lib/python3.8/site-packages/cfgrib/__main__.py", line 56, in to_netcdf
    ds = xr.open_mfdataset(inpaths, engine=engine)
  File "/Users/dian/work/anaconda/envs/mypy3env/lib/python3.8/site-packages/xarray/backends/api.py", line 900, in open_mfdataset
    datasets = [open_(p, **open_kwargs) for p in paths]
  File "/Users/dian/work/anaconda/envs/mypy3env/lib/python3.8/site-packages/xarray/backends/api.py", line 900, in <listcomp>
    datasets = [open_(p, **open_kwargs) for p in paths]
  File "/Users/dian/work/anaconda/envs/mypy3env/lib/python3.8/site-packages/xarray/backends/api.py", line 516, in open_dataset
    store = backends.CfGribDataStore(
  File "/Users/dian/work/anaconda/envs/mypy3env/lib/python3.8/site-packages/xarray/backends/cfgrib_.py", line 43, in __init__
    self.ds = cfgrib.open_file(filename, **backend_kwargs)
  File "/Users/dian/work/anaconda/envs/mypy3env/lib/python3.8/site-packages/cfgrib/dataset.py", line 628, in open_file
    return Dataset(*build_dataset_components(index, read_keys=read_keys, **kwargs))
  File "/Users/dian/work/anaconda/envs/mypy3env/lib/python3.8/site-packages/cfgrib/dataset.py", line 542, in build_dataset_components
    dims, data_var, coord_vars = build_variable_components(
  File "/Users/dian/work/anaconda/envs/mypy3env/lib/python3.8/site-packages/cfgrib/dataset.py", line 476, in build_variable_components
    geo_dims, geo_shape, geo_coord_vars = build_geography_coordinates(index, encode_cf, errors)
  File "/Users/dian/work/anaconda/envs/mypy3env/lib/python3.8/site-packages/cfgrib/dataset.py", line 357, in build_geography_coordinates
    latitudes = np.array(first['distinctLatitudes'])
  File "/Users/dian/work/anaconda/envs/mypy3env/lib/python3.8/site-packages/cfgrib/messages.py", line 178, in __getitem__
    return super(ComputedKeysMessage, self).__getitem__(item)
  File "/Users/dian/work/anaconda/envs/mypy3env/lib/python3.8/site-packages/cfgrib/messages.py", line 130, in __getitem__
    return self.message_get(item)
  File "/Users/dian/work/anaconda/envs/mypy3env/lib/python3.8/site-packages/cfgrib/messages.py", line 92, in message_get
    values = eccodes.codes_get_array(self.codes_id, item, key_type)
  File "/Users/dian/work/anaconda/envs/mypy3env/lib/python3.8/site-packages/cfgrib/bindings.py", line 365, in codes_get_array
    size = codes_get_size(handle, key)
  File "/Users/dian/work/anaconda/envs/mypy3env/lib/python3.8/site-packages/cfgrib/bindings.py", line 216, in codes_get_size
    _codes_get_size(handle, key.encode(ENC), size)
  File "/Users/dian/work/anaconda/envs/mypy3env/lib/python3.8/site-packages/cfgrib/bindings.py", line 165, in wrapper
    raise GribInternalError(code)
cfgrib.bindings.GribInternalError: ('Decoding invalid (-13).', -13)

Environment (conda list):

``` $ conda list attrs 19.3.0 py_0 conda-forge boost-cpp 1.70.0 h75728bb_2 conda-forge boto 2.49.0 py38_0 boto3 1.10.28 py_0 conda-forge botocore 1.13.35 py_0 conda-forge branca 0.3.1 py_0 conda-forge bzip2 1.0.8 h0b31af3_2 conda-forge ca-certificates 2019.11.28 hecc5488_0 conda-forge cairo 1.16.0 he1c11cd_1002 conda-forge certifi 2019.11.28 py38_0 conda-forge cffi 1.13.2 py38h33e799b_0 conda-forge cfgrib 0.9.7.3 py_0 conda-forge cfitsio 3.470 h84d2f63_2 conda-forge cftime 1.0.4.2 py38h3b54f70_0 conda-forge chardet 3.0.4 py38_1003 conda-forge click 7.0 py_0 conda-forge click-plugins 1.1.1 py_0 conda-forge cligj 0.5.0 py_0 conda-forge cryptography 2.8 py38hafa8578_1 conda-forge curl 7.65.3 h22ea746_0 conda-forge cycler 0.10.0 py_2 conda-forge descartes 1.1.0 py_4 conda-forge docutils 0.15.2 py38_0 conda-forge eccodes 2.14.1 ha74ff94_3 conda-forge expat 2.2.5 h4a8c4bd_1004 conda-forge fiona 1.8.13 py38he71f6a4_0 conda-forge folium 0.10.0 py_1 conda-forge fontconfig 2.13.1 h6b1039f_1001 conda-forge freetype 2.10.0 h24853df_1 conda-forge freexl 1.0.5 h1de35cc_1002 conda-forge g2clib 1.6.0 h4e57d6e_9 conda-forge gdal 3.0.2 py38h97c3584_4 conda-forge geopandas 0.6.2 py_0 conda-forge geos 3.7.2 h6de7cb9_2 conda-forge geotiff 1.5.1 ha113b03_7 conda-forge gettext 0.19.8.1 h46ab8bc_1002 conda-forge giflib 5.1.7 h01d97ff_1 conda-forge glib 2.58.3 py38h577aef8_1002 conda-forge hdf4 4.2.13 h84186c3_1003 conda-forge hdf5 1.10.5 nompi_h3e39495_1104 conda-forge hdfeos2 2.20 h207910d_1000 conda-forge hdfeos5 5.1.16 h60f75e3_5 conda-forge icu 64.2 h6de7cb9_1 conda-forge idna 2.8 py38_1000 conda-forge jasper 1.900.1 h636a363_1006 conda-forge jinja2 2.10.3 py_0 conda-forge jmespath 0.9.4 py_0 conda-forge jpeg 9c h1de35cc_1001 conda-forge json-c 0.13.1 h1de35cc_1001 conda-forge kealib 1.4.10 h6659575_1005 conda-forge kiwisolver 1.1.0 py38ha1b3eb9_0 conda-forge krb5 1.16.3 hcfa6398_1001 conda-forge libaec 1.0.4 h0a44026_0 conda-forge libblas 3.8.0 14_openblas conda-forge libcblas 3.8.0 14_openblas conda-forge libcurl 7.65.3 h16faf7d_0 conda-forge libcxx 9.0.0 h89e68fa_1 conda-forge libdap4 3.20.4 habf5908_0 conda-forge libedit 3.1.20170329 hcfe32e1_1001 conda-forge libffi 3.2.1 h6de7cb9_1006 conda-forge libgdal 3.0.2 h59ca9cb_4 conda-forge libgfortran 4.0.0 2 conda-forge libiconv 1.15 h01d97ff_1005 conda-forge libkml 1.3.0 hed7d534_1010 conda-forge liblapack 3.8.0 14_openblas conda-forge libnetcdf 4.7.1 nompi_hec86efb_102 conda-forge libopenblas 0.3.7 h3d69b6c_5 conda-forge libpng 1.6.37 h2573ce8_0 conda-forge libpq 11.5 h31a01ba_2 conda-forge libspatialindex 1.9.3 h4a8c4bd_1 conda-forge libspatialite 4.3.0a h7a659be_1032 conda-forge libssh2 1.8.2 hcdc9a53_2 conda-forge libtiff 4.1.0 ha78913b_1 conda-forge libuuid 2.32.1 h1de35cc_1000 conda-forge libwebp 1.0.2 h20df551_3 conda-forge libxml2 2.9.10 h53d96d6_0 conda-forge llvm-openmp 9.0.0 h40edb58_0 conda-forge lz4-c 1.8.3 h6de7cb9_1001 conda-forge markupsafe 1.1.1 py38h0b31af3_0 conda-forge matplotlib 3.1.2 py38_1 conda-forge matplotlib-base 3.1.2 py38h11da6c2_1 conda-forge munch 2.5.0 py_0 conda-forge ncurses 6.1 h0a44026_1002 conda-forge netcdf4 1.5.3 nompi_py38hd261794_101 conda-forge numpy 1.17.3 py38hde6bac1_0 conda-forge openjpeg 2.3.1 hcdae239_3 conda-forge openssl 1.1.1d h0b31af3_0 conda-forge pandas 0.25.3 py38h4f17bb1_0 conda-forge pcre 8.43 h4a8c4bd_0 conda-forge pip 19.3.1 py38_0 conda-forge pixman 0.38.0 h01d97ff_1003 conda-forge poppler 0.67.0 h16886b5_8 conda-forge poppler-data 0.4.9 1 conda-forge postgresql 11.5 h26bc10f_2 conda-forge proj 6.2.1 h773a61f_0 conda-forge pycparser 2.19 py38_1 conda-forge pynio 1.5.5 py38h24b5471_7 conda-forge pyopenssl 19.1.0 py38_0 conda-forge pyparsing 2.4.5 py_0 conda-forge pyproj 2.4.2.post1 py38h03a428a_0 conda-forge pysocks 1.7.1 py38_0 conda-forge python 3.8.0 hd366da7_5 conda-forge python-dateutil 2.8.0 py_0 conda-forge pytz 2019.3 py_0 conda-forge readline 8.0 hcfe32e1_0 conda-forge requests 2.22.0 py38_1 conda-forge rtree 0.9.3 py38h7b0cdae_0 conda-forge s3transfer 0.2.1 py38_0 conda-forge scipy 1.3.2 py38h82752d6_0 conda-forge setuptools 42.0.2 py38_0 conda-forge shapely 1.6.4 py38h5c88e11_1006 conda-forge six 1.13.0 py38_0 conda-forge sqlite 3.30.1 h93121df_0 conda-forge tbb 2018.0.5 h2d50403_0 conda-forge tiledb 1.7.0 hd5e958f_2 conda-forge tk 8.6.10 hbbe82c9_0 conda-forge tornado 6.0.3 py38h0b31af3_0 conda-forge tzcode 2019a h01d97ff_1002 conda-forge urllib3 1.25.7 py38_0 conda-forge vincent 0.4.4 py_1 conda-forge wheel 0.33.6 py38_0 conda-forge xarray 0.14.1 py_0 conda-forge xerces-c 3.2.2 hbda6038_1004 conda-forge xz 5.2.4 h1de35cc_1001 conda-forge zlib 1.2.11 h0b31af3_1006 conda-forge zstd 1.4.4 he7fca8b_1 conda-forge ```

Details about conda and system ( conda info ):

``` $ conda info active environment : mypy3env active env location : /Users/dian/work/anaconda/envs/mypy3env shell level : 2 user config file : /Users/dian/.condarc populated config files : /Users/dian/.condarc conda version : 4.7.12 conda-build version : 3.18.9 python version : 3.7.4.final.0 virtual packages : base environment : /Users/dian/work/anaconda (writable) channel URLs : https://repo.anaconda.com/pkgs/main/osx-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/osx-64 https://repo.anaconda.com/pkgs/r/noarch package cache : /Users/dian/work/anaconda/pkgs /Users/dian/.conda/pkgs envs directories : /Users/dian/work/anaconda/envs /Users/dian/.conda/envs platform : osx-64 user-agent : conda/4.7.12 requests/2.22.0 CPython/3.7.4 Darwin/18.2.0 OSX/10.14.3 UID:GID : 501:20 netrc file : None offline mode : False ```

[dtip]

How bizarre. During the OSX build we install libpng-1.6.37:

libpng:          1.6.37-h2573ce8_0          conda-forge

but cmake finds v1.4.12

-- Found PNG: $PREFIX/lib/libpng.dylib (found version "1.4.12") 

Compare this with the build on Linux, which installs and finds v1.6.37 correctly:

libpng:          1.6.37-hed695b0_0          conda-forge
...
-- Found PNG: $PREFIX/lib/libpng.so (found version "1.6.37") 

It's not obvious at all to me how this could happen. Possibly a problem in the libpng feedstock? Possible an ecCodes build issue. I'll ping someone at ECMWF to see if they can shed any light on it.

[kmuehlbauer]

@dtip

 PNG 1.4.12
--       includes : [/Library/Frameworks/Mono.framework/Headers;$PREFIX/include]
--       libs     : [$PREFIX/lib/libpng.dylib;$PREFIX/lib/libz.dylib]

There is some issue with this Mono-Framework. We have this in every MacOSX build on azure, right from the beginning (at least as far as azure keeps the logs). In the last travis builds this Mono-Framework is not there. So maybe we need to do some cleanup of the azure environment.

[kmuehlbauer]

Xref: https://github.com/conda-forge/libitk-feedstock/pull/37 for similar problem with JPEG.

[dtip]

Looks like they're having our exact problem with libpng on that thread you linked: https://github.com/conda-forge/libitk-feedstock/pull/37#issuecomment-532261626

[dtip]

Lets see if this works: https://github.com/conda-forge/eccodes-feedstock/pull/96

[dtip]

@dmfay I've pushed another build. Please try re-installing ecCodes from conda-forge in a couple of hours and see if it's fixed your problem.

[dmfay]

That does look like it helped! I'm getting a different traceback after a couple seconds' wait now:

$ cfgrib to_netcdf MRMS_MergedReflectivityQCComposite_00.50_20191209-210032.grib2
/Users/dian/work/anaconda/envs/mypy3env/lib/python3.8/site-packages/cfgrib/__main__.py:56: FutureWarning: In xarray version 0.15 the default behaviour of `open_mfdataset`
will change. To retain the existing behavior, pass
combine='nested'. To use future default behavior, pass
combine='by_coords'. See
http://xarray.pydata.org/en/stable/combining.html#combining-multi

  ds = xr.open_mfdataset(inpaths, engine=engine)
/Users/dian/work/anaconda/envs/mypy3env/lib/python3.8/site-packages/xarray/backends/api.py:926: FutureWarning: The datasets supplied have global dimension coordinates. You may want
to use the new `combine_by_coords` function (or the
`combine='by_coords'` option to `open_mfdataset`) to order the datasets
before concatenation. Alternatively, to continue concatenating based
on the order the datasets are supplied in future, please use the new
`combine_nested` function (or the `combine='nested'` option to
open_mfdataset).
  combined = auto_combine(
Exception ignored in: <function Pool.__del__ at 0x119206ee0>
Traceback (most recent call last):
  File "/Users/dian/work/anaconda/envs/mypy3env/lib/python3.8/multiprocessing/pool.py", line 268, in __del__
  File "/Users/dian/work/anaconda/envs/mypy3env/lib/python3.8/multiprocessing/queues.py", line 362, in put
AttributeError: 'NoneType' object has no attribute 'dumps'

Not the most helpful call stack I've ever seen, but I don't see ecCodes in there anywhere. Should I take this back to the cfgrib issue?

[dtip]

Yes please. I seems like the ecCodes side of things is fixed now. It looks like a problem in the cfgrib python code. But I could be wrong!

[dopplershift]

Looks like this was resolved.