search

pydata / xarray

N-D labeled arrays and datasets in Python
https://xarray.dev
Apache License 2.0
3.59k stars 1.08k forks source link

failing CI #5138

Closed keewis closed 3 years ago

keewis commented 3 years ago

It seems cfgrib (or one of its dependencies) is breaking the import of xarray on both upsteam-dev and some of the normal CI. Interestingly, the broken environments installed eccodes=2.19.1 while the ones which are passing have eccodes=2.21.0.

Is this a mamba / conda caching issue?

cc @andersy005 @alexamici

alexamici commented 3 years ago

Looking into ecmwf/cfgrib#226 that seems related.

keewis commented 3 years ago

it is interesting that this only happens for py37 macos, py37 ubuntu and py39 ubuntu but nothing else.

Looking into ecmwf/cfgrib#226 that seems related.

That does indeed seem related but seems to use py38 which does not fail for us.

For reference, this is the traceback:

traceback ```pytb Traceback (most recent call last): File "", line 1, in File "/home/runner/work/xarray/xarray/xarray/__init__.py", line 3, in from . import testing, tutorial, ufuncs File "/home/runner/work/xarray/xarray/xarray/tutorial.py", line 13, in from .backends.api import open_dataset as _open_dataset File "/home/runner/work/xarray/xarray/xarray/backends/__init__.py", line 6, in from .cfgrib_ import CfGribDataStore File "/home/runner/work/xarray/xarray/xarray/backends/cfgrib_.py", line 19, in import cfgrib File "/usr/share/miniconda/envs/xarray-tests/lib/python3.9/site-packages/cfgrib/__init__.py", line 19, in from .cfmessage import CfMessage File "/usr/share/miniconda/envs/xarray-tests/lib/python3.9/site-packages/cfgrib/cfmessage.py", line 29, in from . import messages File "/usr/share/miniconda/envs/xarray-tests/lib/python3.9/site-packages/cfgrib/messages.py", line 28, in import eccodes # type: ignore File "/usr/share/miniconda/envs/xarray-tests/lib/python3.9/site-packages/eccodes/__init__.py", line 15, in from .eccodes import * File "/usr/share/miniconda/envs/xarray-tests/lib/python3.9/site-packages/eccodes/eccodes.py", line 12, in from gribapi import __version__ File "/usr/share/miniconda/envs/xarray-tests/lib/python3.9/site-packages/gribapi/__init__.py", line 13, in from .gribapi import * # noqa File "/usr/share/miniconda/envs/xarray-tests/lib/python3.9/site-packages/gribapi/gribapi.py", line 32, in from .bindings import ENC, ffi, lib File "/usr/share/miniconda/envs/xarray-tests/lib/python3.9/site-packages/gribapi/bindings.py", line 29, in from ._bindings import ffi, lib ImportError: /usr/share/miniconda/envs/xarray-tests/lib/python3.9/site-packages/gribapi/_bindings.cpython-39-x86_64-linux-gnu.so: undefined symbol: codes_bufr_key_is_header ```
alexamici commented 3 years ago

Thanks @keewis! ATM the main suspect is a flaky the dependency between python-eccodes and eccodes, not cfgrib itself. So it may look like a random error in CI.

alexamici commented 3 years ago

As a general measure I would add ImportError to the exception caught on import so we avoid to crash xarray:

https://github.com/pydata/xarray/blob/186c2ff6542f090a8e97ffb755f866030be30be9/xarray/backends/cfgrib_.py#L18-L29

To work around the current problem we may need to add a eccodes>=2.21.0 to the CI :/

keewis commented 3 years ago

To work around the current problem we may need to add a eccodes>=2.21.0 to the CI

agreed. If eccodes<2.21.0 is truly incompatible with the new version of cfgrib, you could also release a new build on conda-forge.

alexamici commented 3 years ago

Technically this is an incompatibility between python-eccodes and eccodes, I'm trying to resolve it via a PR to the python-eccodes-feedstock: https://github.com/conda-forge/python-eccodes-feedstock/pull/71

alexamici commented 3 years ago

@keewis after merging #5139 the CI should not be failing anymore. The problem is now that cfgrib may not be consistently tested on all jobs and failures are silent.

I'll wait a bit for the fix in python-eccodes, otherwise I'll do a new release of the conda package of cfgrib.

keewis commented 3 years ago

great. If this is nothing we have control over, should we close this?

alexamici commented 3 years ago

Yes, I would close the issue.

keewis commented 3 years ago

thanks, @alexamici

keewis commented 3 years ago

@alexamici, I'm not sure why but mamba does not install the fixed build of python-eccodes on py39:

/home/runner/work/xarray/xarray/xarray/backends/cfgrib_.py:27: UserWarning: Failed to load cfgrib - most likely there is a problem accessing the ecCodes library. Try `import cfgrib` to get the full error message

Do you have any idea how to fix that?

alexamici commented 3 years ago

I don't know, I've never managed to get cfgrib and xarray to install on python 3.9. The conda solver fails.

It looks like there is a conflict between xarray and eccodes dependencies, but I didn't investigate.

keewis commented 3 years ago

both

mamba create -n py39-mamba python=3.9 mamba
mamba env update -n py39-mamba -f ci/requirements/environment.yml

and

conda create -n py39-conda python=3.9
conda env update -n py39-conda -f ci/requirements/environment.yml

work for me (on linux), but mamba installs the wrong version of python-eccodes (build number 0 instead of number 1). Of course, there's lots of other differences (mostly different build hashes).

Edit: not sure if I didn't do anything wrong, though