error with cfgrib + eccodes

[ocarino]

Hi, I tried to install cfgrib and eccodes on two different conda installations. Installation is always successful. Right now I do have:

% conda list | egrep "(eccodes|cfgrib)"
cfgrib                    0.9.9.0            pyhd8ed1ab_0    conda-forge
eccodes                   2.21.0               ha0e6eb6_0    conda-forge
python-eccodes            2021.03.0        py38hb5d20a5_1    conda-forge

Importing xarray works, but whenever I do an "import cfgrib" or an "ds = xarray.open_dataset('gribfile.grb', engine='cfgrib')", I get:

self.ds = cfgrib.open_file(filename, **backend_kwargs) AttributeError: partially initialized module 'cfgrib' has no attribute 'open_file' (most likely due to a circular import)

(full error output attached). This happens on both conda installations (one is miniforge3-4.9.2.3, the other one is anaconda3 conda version 4.10).

Any idea what can cause this issue? FYI: I just downgraded cfgrib to 0.9.8.5 and now it works.

Cheers, Olaf

[keewis]

We would need more information about your environments. Could you post the output of xr.show_versions()? Also, can you reproduce this with

xr.tutorial.open_dataset("era5-2mt-2019-03-uk.grib", engine="cfgrib")

[aurghs]

I can try to reproduce the error and fix it, but I need to know at least the xarray version.

[raybellwaves]

Not sure if https://github.com/conda-forge/python-eccodes-feedstock/pull/71 helps Came across this via https://github.com/ecmwf/cfgrib/issues/226

[ocarino]

conda-error-output.txt conda-xr-showversions.txt

Hi keewis and aurghs, as the working version with cfgrib 0.9.8.5 is already in production, it took some time to reproduce the previous environment from a backup. So here we are. Versions are:

% conda list | egrep "(cfgrib|xarray|eccodes)" cf_xarray 0.3.1 pyhd3deb0d_0 conda-forge cfgrib 0.9.9.0 pyhd8ed1ab_1 conda-forge eccodes 2.21.0 ha0e6eb6_0 conda-forge python-eccodes 2021.03.0 py38hb5d20a5_1 conda-forge xarray 0.16.2 pyhd8ed1ab_0 conda-forge

With "xr.tutorial.open_dataset("era5-2mt-2019-03-uk.grib", engine="cfgrib")" the result is the same, error message is:

AttributeError: partially initialized module 'cfgrib' has no attribute 'open_file' (most likely due to a circular import)

Full error output is attached. And I attached the output of "xr.show_versions()". I wonder why it shows "cfgrib: None"...

Cheers, Olaf

[ocarino]

And I have to correct myself: Though I still have an output of xr_open_dataset from cfgrib version 0.9.8.5 that shows that it worked (otherwise I would doubt on my senses), it does not work right now. xr.show_versions shows "cfgrib: None", too.

Also "import cfgrib" shows the same error. This sounds much like https://github.com/ecmwf/cfgrib/issues/226 , but obviously I already use eccodes 2.21.0 (which is the proposed fix there...)

[keewis]

I still can't reproduce this. For reference, here's my environment:

output of xr.show_versions()

``` INSTALLED VERSIONS ------------------ commit: None python: 3.8.8 | packaged by conda-forge | (default, Feb 20 2021, 16:22:27) [GCC 9.3.0] python-bits: 64 OS: Linux OS-release: 4.19.0-16-amd64 machine: x86_64 processor: byteorder: little libhdf5: None libnetcdf: None xarray: 0.16.2 pandas: 1.2.4 numpy: 1.20.2 scipy: None netCDF4: None pydap: None h5netcdf: None h5py: None Nio: None zarr: None cftime: None nc_time_axis: None PseudoNetCDF: None rasterio: None cfgrib: 0.9.9.0 iris: None bottleneck: None dask: None distributed: None matplotlib: None cartopy: None seaborn: None numbagg: None pint: None setuptools: 49.6.0.post20210108 pip: 21.0.1 conda: 4.10.1 pytest: None IPython: 7.22.0 sphinx: None ```

And I attached the output of "xr.show_versions()"

I can't seem to find that. Am I missing something?

xr.show_versions shows "cfgrib: None", too.

I guess that's because the import fails.

[ocarino]

I added the missing attachments to my previous comment, seems that my browser did not cope with the upload mechanism.

xr.show_versions shows "cfgrib: None", too.

I guess that's because the import fails.

Well, just to narrow down the error: I guess a working "import cfgrib" would be the precondition for it to work with xarray... (?)

[keewis]

strange. This seems like a environment issue. Could you try creating a environment with conda env create -f env.yml?

contents of env.yml

``` name: test channels: - conda-forge dependencies: - python=3.8.6 - xarray=0.16.2 - pandas=1.2.1 - numpy=1.19.2 - scipy=1.6.0 - netCDF4=1.5.5.1 - h5py=2.10.0 - zarr=2.6.1 - cftime=1.3.1 - nc-time-axis=1.2.0 - iris=2.4.0 - bottleneck=1.3.2 - dask=2021.01.0 - distributed=2021.01.0 - matplotlib=3.3.3 - cartopy=0.18.0 - seaborn=0.11.1 - setuptools=49.6.0 - pip=20.3.1 - conda=4.9.2 - pytest=6.2.1 - ipython=7.19.0 - sphinx=3.4.3 - libnetcdf=4.7.4 - cfgrib=0.9.9.0 - h5netcdf=0.8.1 ```

With that, python -c 'import xarray as xr; xr.tutorial.open_dataset("era5-2mt-2019-03-uk.grib", engine="cfgrib")' passes for me.

[ocarino]

Strange. I deployed a completely new conda (using the miniforge installer Miniforge3-4.9.2-3-Linux-x86_64.sh). With that I created an environment "test" with your yml file. When I run the xr.tutorial.open_dataset (exact copy-paste from your message), I get the same error as usual:

AttributeError: partially initialized module 'cfgrib' has no attribute 'open_file' (most likely due to a circular import

What the heck can be the problem that I face? I am bemused...

[keewis]

did you try mamba or conda from miniconda?

Edit: also, did you make sure to activate the new environment (check $PYTHONPATH and which python) and that you are not in a directory containing the code of xarray or cfgrib?

[ocarino]

Hi,

keewis wrote on 15.04.21 17:51:

did you try |mamba| or |conda| from |miniconda|?

right now I only did miniforge (though our elder python environments are anaconda or conda). And: Yes, the env was activated and no other xarray or cfgrib stuff around here. :-)

When I create a complete new miniforge3-4.9.2-3 on our system and only do a

conda install eccodes conda install cfgrib

the thing works. So there must be some relicts from earlier package installs or some other versions that cause the trouble. I got some hints from a different site that it might even be dependent on the order in which things are installed, not sure about that. Still I wonder that the error message seems to imply that there might be a circular import...

I'll do some further testing and will come back when I have something that is more concrete so that you can look at it.

Thanks, Olaf

-- Dipl. Inform. Olaf Gellert email @.*** Deutsches Klimarechenzentrum GmbH phone +49 (0)40 460094 214 Bundesstrasse 45a fax +49 (0)40 460094 270 D-20146 Hamburg, Germany www http://www.dkrz.de

Sitz der Gesellschaft: Hamburg Geschäftsführer: Prof. Dr. Thomas Ludwig Registergericht: Amtsgericht Hamburg, HRB 39784

[ocarino]

Ok, PEBKAC (problem exists between keyboard and chair). Your hints already pointed in the correct direction, I had a testscript in my homedir which was named cfgrib.py, a very unfortunate naming, I'd say... Issue can be closed, everything is working fine. And many thanks for your efforts. Olaf