search

SciTools / iris-grib

GRIB interface for Iris.
https://iris-grib.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
22 stars 43 forks source link

Reading file fails with latest iris version #520

Open schlunma opened 1 month ago

schlunma commented 1 month ago

Reading the file iris_sample_data/sample_data/polar_stereo.grib2 with the latest iris 3.10 produces the following error for me:

>>> import iris
>>> path = "polar_stereo.grib2"
>>> cubes = iris.load(path)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/manuel/micromamba/envs/esm/lib/python3.11/site-packages/iris/__init__.py", line 330, in load
    return _load_collection(uris, constraints, callback).merged().cubes()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/manuel/micromamba/envs/esm/lib/python3.11/site-packages/iris/__init__.py", line 298, in _load_collection
    result = _CubeFilterCollection.from_cubes(cubes, constraints)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/manuel/micromamba/envs/esm/lib/python3.11/site-packages/iris/cube.py", line 95, in from_cubes
    for cube in cubes:
  File "/home/manuel/micromamba/envs/esm/lib/python3.11/site-packages/iris/__init__.py", line 279, in _generate_cubes
    for cube in iris.io.load_files(part_names, callback, constraints):
  File "/home/manuel/micromamba/envs/esm/lib/python3.11/site-packages/iris/io/__init__.py", line 222, in load_files
    for cube in handling_format_spec.handler(fnames, callback):
  File "/home/manuel/micromamba/envs/esm/lib/python3.11/site-packages/iris/fileformats/rules.py", line 435, in load_cubes
    for cube, field in _load_pairs_from_fields_and_filenames(
  File "/home/manuel/micromamba/envs/esm/lib/python3.11/site-packages/iris/fileformats/rules.py", line 360, in _load_pairs_from_fields_and_filenames
    cube, factories, references = _make_cube(field, converter)
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/manuel/micromamba/envs/esm/lib/python3.11/site-packages/iris/fileformats/rules.py", line 297, in _make_cube
    cube_data = field.core_data()
                ^^^^^^^^^^^^^^^^^
  File "/home/manuel/micromamba/envs/esm/lib/python3.11/site-packages/iris_grib/message.py", line 119, in core_data
    return self.data
           ^^^^^^^^^
  File "/home/manuel/micromamba/envs/esm/lib/python3.11/site-packages/iris_grib/message.py", line 162, in data
    data = as_lazy_data(proxy)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/manuel/micromamba/envs/esm/lib/python3.11/site-packages/iris/_lazy_data.py", line 271, in as_lazy_data
    raise ValueError(
ValueError: For performance reasons, `meta` cannot be `None` if `data` is anything other than a Numpy or Dask array.

Here is my environment:

 $ mamba list iris                                                                                                                                                                                                                                                                                            
  Name              Version  Build         Channel
────────────────────────────────────────────────────────
  iris              3.10.0   pyha770c72_0  conda-forge
  iris-esmf-regrid  0.11.0   pyhd8ed1ab_0  conda-forge
  iris-grib         0.19.0   pyhd8ed1ab_0  conda-forge

With iris 3.9, we also get an error, but a slightly different one: https://app.circleci.com/pipelines/github/ESMValGroup/ESMValCore/11527/workflows/2dea67e4-e4c9-4f8e-87fc-bb960e0b7849/jobs/48457/parallel-runs/0/steps/0-105. I did not test this locally though, so it could also be a problem of the CI machine.

trexfeathers commented 1 month ago

Here's the extract from the ESMVT testing of 3.9:

    def __getitem__(self, key):
        if key not in self._cache:
            if key == 'numberOfSection':
                value = self._number
            else:
                if key not in self._keys:
                    key2 = KEY_ALIAS.get(key)
                    if key2 and key2 in self._keys:
                        key = key2
                    else:
                        emsg = f"{key} not defined in section {self._number}"
>                       raise KeyError(emsg)
E                       KeyError: 'indicatorOfUnitOfTimeRange not defined in section 4'
trexfeathers commented 2 weeks ago

Confirmed solved: https://github.com/ESMValGroup/ESMValCore/pull/2178#issuecomment-2329469816

schlunma commented 2 weeks ago

Sorry, the ESMValTool tests work indeed fine, but when I try to open the file mentioned above I get another error:

>>> import iris
>>> path = "polar_stereo.grib2"
>>> cubes = iris.load(path)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/manuel/micromamba/envs/xxx/lib/python3.12/site-packages/iris/__init__.py", line 330, in load
    return _load_collection(uris, constraints, callback).merged().cubes()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/manuel/micromamba/envs/xxx/lib/python3.12/site-packages/iris/__init__.py", line 298, in _load_collection
    result = _CubeFilterCollection.from_cubes(cubes, constraints)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/manuel/micromamba/envs/xxx/lib/python3.12/site-packages/iris/cube.py", line 95, in from_cubes
    for cube in cubes:
                ^^^^^
  File "/home/manuel/micromamba/envs/xxx/lib/python3.12/site-packages/iris/__init__.py", line 279, in _generate_cubes
    for cube in iris.io.load_files(part_names, callback, constraints):
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/manuel/micromamba/envs/xxx/lib/python3.12/site-packages/iris/io/__init__.py", line 212, in load_files
    handling_format_spec = FORMAT_AGENT.get_spec(os.path.basename(fn), fh)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/manuel/micromamba/envs/xxx/lib/python3.12/site-packages/iris/io/format_picker.py", line 150, in get_spec
    raise ValueError(msg)
ValueError: No format specification could be found for the given buffer. Perhaps a plugin is missing or has not been loaded. File element cache:
 {'UriProtocol()': 'file', 'LeadingLine()': "b'\\n'", 'MagicNumber(4, None)': '168430090', 'MagicNumber(8, None)': '723401728380779553', 'DataSourceObjectProtocol()': 'polar_stereo.grib2', 'FileExtension()': '.grib2', 'MagicNumber(100, None)': 'b\'\\n\\n\\n\\n\\n\\n<!DOCTYPE html>\\n<html\\n  lang="en"\\...'}

Loading the file with iris_grib.load_cubes(path) seems to work fine (there's no error).

This is my environment:

  Name       Version  Build         Channel
─────────────────────────────────────────────────
  iris       3.10.0   pyha770c72_1  conda-forge
  iris-grib  0.20.0   pyhd8ed1ab_1  conda-forge
pp-mo commented 2 weeks ago

'b\'\n\n\n\n\n\n<!DOCTYPE html>\n<html\n lang="en"\...'

Looks like something has downloaded an HTML file, not a (binary) GRIB file ??