Installation through conda, import xesmf fails

[rizziemma]

Hello, I have installed xesmf through conda (I include the other libraries I used) :

conda create -n xesmf_env
conda activate xesmf_env

conda install -c conda-forge xesmf
conda install -c conda-forge dask netCDF4
conda install -c conda-forge matplotlib cartopy jupyterlab

And when I execute import xesmf, I get :

ERROR:
>>> import xesmf
Traceback (most recent call last):
  File "C:\Users\adrie\miniconda3\envs\xesmf_env\lib\site-packages\esmpy\interface\loadESMF.py", line 26, in <module>
    esmfmk = os.environ["ESMFMKFILE"]
  File "C:\Users\adrie\miniconda3\envs\xesmf_env\lib\os.py", line 680, in __getitem__
    raise KeyError(key) from None
KeyError: 'ESMFMKFILE'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\adrie\miniconda3\envs\xesmf_env\lib\site-packages\xesmf\backend.py", line 22, in <module>
    import esmpy as ESMF
  File "C:\Users\adrie\miniconda3\envs\xesmf_env\lib\site-packages\esmpy\__init__.py", line 112, in <module>
    from esmpy.api.esmpymanager import *
  File "C:\Users\adrie\miniconda3\envs\xesmf_env\lib\site-packages\esmpy\api\esmpymanager.py", line 9, in <module>
    from esmpy.interface.cbindings import *
  File "C:\Users\adrie\miniconda3\envs\xesmf_env\lib\site-packages\esmpy\interface\cbindings.py", line 13, in <module>
    from esmpy.interface.loadESMF import _ESMF
  File "C:\Users\adrie\miniconda3\envs\xesmf_env\lib\site-packages\esmpy\interface\loadESMF.py", line 28, in <module>
    raise ImportError('The ESMFMKFILE environment variable is not available.')
ImportError: The ESMFMKFILE environment variable is not available.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\adrie\miniconda3\envs\xesmf_env\lib\site-packages\xesmf\__init__.py", line 4, in <module>
    from .frontend import Regridder, SpatialAverager
  File "C:\Users\adrie\miniconda3\envs\xesmf_env\lib\site-packages\xesmf\frontend.py", line 12, in <module>
    from .backend import Grid, LocStream, Mesh, add_corner, esmf_regrid_build, esmf_regrid_finalize
  File "C:\Users\adrie\miniconda3\envs\xesmf_env\lib\site-packages\xesmf\backend.py", line 24, in <module>
    import ESMF
ModuleNotFoundError: No module named 'ESMF'

ERROR IMPORTING ESMF:
>>> import ESMF
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'ESMF'

This used to work a few months ago, the error appeared when I needed to use the library again since last week.

I tried to revert to an older version (0.6.3) and to install with pip, same issue. 2 other coworkers have tried and faced the same issue on their computer.

Thanks for any help!

[aulemahal]

Have you tried deactivating and re-activating the environment after the installation ?

Simply adding:

conda deactivate
conda activate xesmf_env

If this work, then it's because you are encountering https://github.com/esmf-org/esmf/issues/117.

ESMF 8.4 had to major breaking changes:

1. The python package was renamed to esmpy
2. Importing the fortran wrapper depends on a file that is found through an environment variable which is set when activating the conda environment. Thus, one needs to "activate" before using the package.

Downgrading esmf to 8.3.1 (and not xesmf) will also fix the issue.

[rizziemma]

@aulemahal thanks that was it! We chose to downgrade esmf to 8.3.1 for now.

[zdefne-usgs]

I had the same error with @rizziemma when I tried to import xesmf. @aulemahal 's suggestion of downgrading worked for me. Somehow the esmf=8.3.1 did not exist in the conda-forge channel so I tried esmf=8.4.0 and it solved my problem.

[aulemahal]

@zdefne-usgs The issue is supposed to happen with esmf 8.4.0 and higher. However, in a normal conda installation, the simple fact of deactivating and reactivating the environment after installation makes esmf work, maybe that was what happened ?

[kthyng]

Thank goodness I found this! This wasn't easily googleable for some reason but I found this by checking the issues directly.

[zdefne-usgs]

@zdefne-usgs The issue is supposed to happen with esmf 8.4.0 and higher. However, in a normal conda installation, the simple fact of deactivating and reactivating the environment after installation makes esmf work, maybe that was what happened ?

I am not sure if I understand you correctly @aulemahal . esmf=8.3.1 is not available form conda-forge channel. Deactivating then activating did not work with 8.4.1, if that's what you mean. Downgrading to 8.4.0 has worked. Maybe @kthyng can comment with her solution.

>conda search -c conda-forge esmf
Loading channels: done
# Name                       Version           Build  Channel
esmf                           8.4.0 nompi_h693c31f_5  conda-forge
esmf                           8.4.0 nompi_hdb05531_3  conda-forge
esmf                           8.4.0 nompi_hdb05531_4  conda-forge
esmf                           8.4.1 nompi_h4247ae4_1  conda-forge
esmf                           8.4.1 nompi_h55e7d63_2  conda-forge
esmf                           8.4.1 nompi_h693c31f_0  conda-forge
esmf                           8.4.2 nompi_h55e7d63_0  conda-forge

[aulemahal]

Oh strange! conda search -c conda-forge esmf returns much more results on my machine, I see versions down to 7.1.0, and 8.3.1 included. I guess the search can be OS dependent?

If you say that downgrading to 8.4.0 worked, even though the bug this issue is about was introduced in 8.4.0, then I fear that you were having another bug! Good to know it is fixed, but I wonder what it could have been and if it was linked to something in xESMF...

[kthyng]

I just installed this all yesterday and here's what I have. It worked once I deactivated and reactivated my environment — something I would not have thought to do!

# packages in environment at /Users/kthyng/miniconda3/envs/ciofs:
#
# Name                    Version                   Build  Channel
esmf                      8.4.1           nompi_h2bb01ac_0    conda-forge
esmpy                     8.4.1              pyhc1e730c_0    conda-forge
xesmf                     0.7.1              pyhd8ed1ab_0    conda-forge

[zdefne-usgs]

hmm I wonder if the experience with activate/deactivate is different if installing from base environment or from within an environment? Beats me. But Kristen's comment confirms your solution Pascal. Thank you both!

[sbeale007]

hi! I am having this issue still even after downgrading to esmf 8.4.0 and deactivating and reactivating the environment. Is there something else I can try?

[aulemahal]

Hum, could you share the error message with traceback ? If reactivating the env doesn't work, then it might be another new problem...

[sbeale007]

I have been struggling a lot with downloading xesmf lately so it might be a different problem This is what I did conda create -n xesmf_env conda activate xesmf_env

conda install -c conda-forge xesmf conda install -c conda-forge dask netCDF4 conda install -c conda-forge matplotlib cartopy jupyterlab

conda install -c conda-forge esmf=8.4.0

Deactivated then activated again Then I get this error when importing

[aulemahal]

Strange, this is exactly the error I would expect for the activation problem... Could you check in the env if the ESMF and ESMpy version match ? Or simply share a conda env list with us.

[sbeale007]

this is my esmf and ESMpy! esmf 8.4.0 mpi_mpich_h3453638_105 conda-forge esmpy 8.4.0 mpi_mpich_py310h881952c_102 conda-forge

This is my first time really using conda environments so I'm worried I messed something up but I did follow the same steps as before. Thank you for the help!

[aulemahal]

You could try downgrading lower, to 8.3.1 ?

[sbeale007]

That seems to have worked thank you so much!