search

JiaweiZhuang / xESMF

Universal Regridder for Geospatial Data
http://xesmf.readthedocs.io/
MIT License
269 stars 49 forks source link

 MPI Errors when regridding multi-dimensional data #55

Closed tommylees112 closed 4 years ago

tommylees112 commented 4 years ago

The following just makes the example data to be used by the regridder

import pandas as pd
import numpy as np
import xarray as xr

def make_dataset():
    time = pd.date_range('2010-01-01','2018-12-31',freq='M')
    lat = np.linspace(-5.175003, -4.7250023, 10)
    lon = np.linspace(33.524994, 33.97499, 10)
    precip = np.random.normal(0, 1, size=(len(time), len(lat), len(lon)))

    ds = xr.Dataset(
        {'precip': (['time', 'lat', 'lon'], precip)},
        coords={
            'lon': lon,
            'lat': lat,
            'time': time,
        }
    )
    return ds

def make_fcast_dataset(date_str):
    initialisation_date = pd.date_range(start=date_str, periods=1, freq='M')
    number = [i for i in range(0, 10)]
    lat = np.linspace(-5.175003, -5.202, 5)
    lon = np.linspace(33.5, 42.25, 5)
    forecast_horizon = np.array(
        [ 2419200000000000,  2592000000000000,  2678400000000000,
          5097600000000000,  5270400000000000,  5356800000000000,
          7689600000000000,  7776000000000000,  7862400000000000,
          7948800000000000, 10368000000000000, 10454400000000000,
          10540800000000000, 10627200000000000, 12960000000000000,
          13046400000000000, 13219200000000000, 15638400000000000,
          15724800000000000, 15811200000000000, 15897600000000000,
          18316800000000000, 18489600000000000, 18576000000000000 ],
          dtype='timedelta64[ns]'
    )
    valid_time = initialisation_date[:, np.newaxis] + forecast_horizon
    precip = np.random.normal(
        0, 1, size=(len(number), len(initialisation_date), len(forecast_horizon), len(lat), len(lon))
    )

    ds = xr.Dataset(
        {'precip': (['number', 'initialisation_date', 'forecast_horizon', 'lat', 'lon'], precip)},
        coords={
            'lon': lon,
            'lat': lat,
            'initialisation_date': initialisation_date,
            'number': number,
            'forecast_horizon': forecast_horizon,
            'valid_time': (['initialisation_date', 'step'], valid_time)
        }
    )
    return ds

# make the example datasets
regrid = make_dataset().precip
ds = make_fcast_dataset(date_str='2000-01-01')

When using the regridder I get the following error.

import xesmf as xe

xe.Regridder(ds, regrid, 'nearest_s2d')

ipdb> xe.Regridder(d_, regrid.precip, 'nearest_s2d')
[mpiexec@linux1.ouce.ox.ac.uk] match_arg (utils/args/args.c:159): unrecognized argument pmi_args
[mpiexec@linux1.ouce.ox.ac.uk] HYDU_parse_array (utils/args/args.c:174): argument matching returned error
[mpiexec@linux1.ouce.ox.ac.uk] parse_args (ui/mpich/utils.c:1596): error parsing input array
[mpiexec@linux1.ouce.ox.ac.uk] HYD_uii_mpx_get_parameters (ui/mpich/utils.c:1648): unable to parse user arguments
[mpiexec@linux1.ouce.ox.ac.uk] main (ui/mpich/mpiexec.c:149): error parsing parameters

I am very inexperienced with MPI and so would really appreciate any pointers you might have!

Thanks very much

JiaweiZhuang commented 4 years ago

How did you install ESMPy? There's probably something wrong with the installation.

Try the following commands:

# better use a new conda environment
conda install -c conda-forge esmpy
pip install xesmf
pip install pytest
pytest -v --pyargs xesmf
# All 20 tests should pass. If not, post the failed ones.
tommylees112 commented 4 years ago

Downloading the packages

(crp) chri4118@linux3:/soge-home/projects/crop_yield/ml_drought$ conda uninstall esmpy --yes
Solving environment: done

==> WARNING: A newer version of conda exists. <==
  current version: 4.5.4
  latest version: 4.7.10

Please update conda by running

    $ conda update -n base conda

## Package Plan ##

  environment location: /soge-home/users/chri4118/.conda/envs/crp

  removed specs:
    - esmpy

The following packages will be REMOVED:

    esmpy: 7.1.0-py37h24bf2e0_3 conda-forge

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(crp) chri4118@linux3:/soge-home/projects/crop_yield/ml_drought$ conda install -c conda-forge esmpy
Solving environment: done

==> WARNING: A newer version of conda exists. <==
  current version: 4.5.4
  latest version: 4.7.10

Please update conda by running

    $ conda update -n base conda

## Package Plan ##

  environment location: /soge-home/users/chri4118/.conda/envs/crp

  added / updated specs:
    - esmpy

The following NEW packages will be INSTALLED:

    esmpy: 7.1.0-py37h24bf2e0_3 conda-forge

Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done

(crp) chri4118@linux3:/soge-home/projects/crop_yield/ml_drought$ pip uninstall xesmf
Uninstalling xesmf-0.1.1:
  Would remove:
    /lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/xesmf-0.1.1.dist-info/*
    /lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/xesmf/*
Proceed (y/n)? y
  Successfully uninstalled xesmf-0.1.1

(crp) chri4118@linux3:/soge-home/projects/crop_yield/ml_drought$ pip install xesmf
Collecting xesmf
Requirement already satisfied: xarray in /lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages (from xesmf) (0.12.2)
Requirement already satisfied: esmpy in /lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages (from xesmf) (7.1.0.dev0)
Requirement already satisfied: numpy in /lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages (from xesmf) (1.16.2)
Requirement already satisfied: scipy in /lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages (from xesmf) (1.3.0)
Requirement already satisfied: pandas>=0.19.2 in /lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages (from xarray->xesmf) (0.24.2)
Requirement already satisfied: python-dateutil>=2.5.0 in /lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages (from pandas>=0.19.2->xarray->xesmf) (2.8.0)
Requirement already satisfied: pytz>=2011k in /lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages (from pandas>=0.19.2->xarray->xesmf) (2019.1)
Requirement already satisfied: six>=1.5 in /lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages (from python-dateutil>=2.5.0->pandas>=0.19.2->xarray->xesmf) (1.12.0)
Installing collected packages: xesmf
Successfully installed xesmf-0.1.1

running the tests

(crp) chri4118@linux3:/soge-home/projects/crop_yield/ml_drought$ pytest -v --pyargs xesmf
================================================================ test session starts =================================================================
platform linux -- Python 3.7.3, pytest-4.5.0, py-1.8.0, pluggy-0.11.0 -- /soge-home/users/chri4118/.conda/envs/crp/bin/python
cachedir: .pytest_cache
rootdir: /soge-home/projects/crop_yield/ml_drought, inifile: pytest.ini
plugins: flake8-1.0.4
collected 10 items / 3 errors / 7 selected

======================================================================= ERRORS =======================================================================
_______________________________________________________ ERROR collecting tests/test_backend.py _______________________________________________________
ImportError while importing test module '/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/xesmf/tests/test_backend.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/ESMF/interface/loadESMF.py:118: in <module>
    mode=ct.RTLD_GLOBAL)
../../../users/chri4118/.conda/envs/crp/lib/python3.7/ctypes/__init__.py:356: in __init__
    self._handle = _dlopen(self._name, mode)
E   OSError: /soge-home/users/chri4118/.conda/envs/crp/lib/libmpicxx.so.12: undefined symbol: MPIR_Keyval_set_proxy

During handling of the above exception, another exception occurred:
/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/xesmf/__init__.py:3: in <module>
    from . frontend import Regridder
/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/xesmf/frontend.py:9: in <module>
    from . backend import (esmf_grid, add_corner,
/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/xesmf/backend.py:19: in <module>
    import ESMF
/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/ESMF/__init__.py:70: in <module>
    from ESMF.api.esmpymanager import *
/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/ESMF/api/esmpymanager.py:11: in <module>
    from ESMF.interface.cbindings import *
/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/ESMF/interface/cbindings.py:13: in <module>
    from ESMF.interface.loadESMF import _ESMF
/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/ESMF/interface/loadESMF.py:121: in <module>
    raise ImportError('The ESMF shared library did not load properly.')
E   ImportError: The ESMF shared library did not load properly.
------------------------------------------------------------------ Captured stdout -------------------------------------------------------------------
Traceback (most recent call last):
  File "/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/ESMF/interface/loadESMF.py", line 118, in <module>
    mode=ct.RTLD_GLOBAL)
  File "/soge-home/users/chri4118/.conda/envs/crp/lib/python3.7/ctypes/__init__.py", line 356, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /soge-home/users/chri4118/.conda/envs/crp/lib/libmpicxx.so.12: undefined symbol: MPIR_Keyval_set_proxy
______________________________________________________ ERROR collecting tests/test_frontend.py _______________________________________________________
ImportError while importing test module '/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/xesmf/tests/test_frontend.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/ESMF/interface/loadESMF.py:118: in <module>
    mode=ct.RTLD_GLOBAL)
../../../users/chri4118/.conda/envs/crp/lib/python3.7/ctypes/__init__.py:356: in __init__
    self._handle = _dlopen(self._name, mode)
E   OSError: /soge-home/users/chri4118/.conda/envs/crp/lib/libmpicxx.so.12: undefined symbol: MPIR_Keyval_set_proxy

During handling of the above exception, another exception occurred:
/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/xesmf/__init__.py:3: in <module>
    from . frontend import Regridder
/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/xesmf/frontend.py:9: in <module>
    from . backend import (esmf_grid, add_corner,
/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/xesmf/backend.py:19: in <module>
    import ESMF
/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/ESMF/__init__.py:70: in <module>
    from ESMF.api.esmpymanager import *
/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/ESMF/api/esmpymanager.py:11: in <module>
    from ESMF.interface.cbindings import *
/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/ESMF/interface/cbindings.py:13: in <module>
    from ESMF.interface.loadESMF import _ESMF
/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/ESMF/interface/loadESMF.py:121: in <module>
    raise ImportError('The ESMF shared library did not load properly.')
E   ImportError: The ESMF shared library did not load properly.
------------------------------------------------------------------ Captured stdout -------------------------------------------------------------------
Traceback (most recent call last):
  File "/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/ESMF/interface/loadESMF.py", line 118, in <module>
    mode=ct.RTLD_GLOBAL)
  File "/soge-home/users/chri4118/.conda/envs/crp/lib/python3.7/ctypes/__init__.py", line 356, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /soge-home/users/chri4118/.conda/envs/crp/lib/libmpicxx.so.12: undefined symbol: MPIR_Keyval_set_proxy
________________________________________________________ ERROR collecting tests/test_util.py _________________________________________________________
ImportError while importing test module '/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/xesmf/tests/test_util.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/ESMF/interface/loadESMF.py:118: in <module>
    mode=ct.RTLD_GLOBAL)
../../../users/chri4118/.conda/envs/crp/lib/python3.7/ctypes/__init__.py:356: in __init__
    self._handle = _dlopen(self._name, mode)
E   OSError: /soge-home/users/chri4118/.conda/envs/crp/lib/libmpicxx.so.12: undefined symbol: MPIR_Keyval_set_proxy

During handling of the above exception, another exception occurred:
/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/xesmf/__init__.py:3: in <module>
    from . frontend import Regridder
/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/xesmf/frontend.py:9: in <module>
    from . backend import (esmf_grid, add_corner,
/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/xesmf/backend.py:19: in <module>
    import ESMF
/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/ESMF/__init__.py:70: in <module>
    from ESMF.api.esmpymanager import *
/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/ESMF/api/esmpymanager.py:11: in <module>
    from ESMF.interface.cbindings import *
/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/ESMF/interface/cbindings.py:13: in <module>
    from ESMF.interface.loadESMF import _ESMF
/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/ESMF/interface/loadESMF.py:121: in <module>
    raise ImportError('The ESMF shared library did not load properly.')
E   ImportError: The ESMF shared library did not load properly.
------------------------------------------------------------------ Captured stdout -------------------------------------------------------------------
Traceback (most recent call last):
  File "/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/ESMF/interface/loadESMF.py", line 118, in <module>
    mode=ct.RTLD_GLOBAL)
  File "/soge-home/users/chri4118/.conda/envs/crp/lib/python3.7/ctypes/__init__.py", line 356, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /soge-home/users/chri4118/.conda/envs/crp/lib/libmpicxx.so.12: undefined symbol: MPIR_Keyval_set_proxy
================================================================== warnings summary ==================================================================
/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/yaml/constructor.py:126
  /lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/yaml/constructor.py:126: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    if not isinstance(key, collections.Hashable):

/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/zict/zip.py:3
  /lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/zict/zip.py:3: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import MutableMapping

/lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/zict/common.py:3
  /lustre/soge1/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/zict/common.py:3: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Mapping, MutableMapping

-- Docs: https://docs.pytest.org/en/latest/warnings.html
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 3 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================================== 3 warnings, 3 error in 15.33 seconds ========================================================
tommylees112 commented 4 years ago

Still getting the same errors even with the new download :(

JiaweiZhuang commented 4 years ago

OK it is definitely an ESMF/ESMPy installation error. The tests don't even run. This can happen on shared clusters with pre-loaded modules, because ESMF might be linking an incompatible NetCDF library during installation.

What does echo $LD_LIBRARY_PATH show? If it points to a system NetCDF installation, ESMF might be using it instead of the conda-installed one.

Try the following:

  1. Unload all modules (something like module purge), double check your .bashrc file, make sure that LD_LIBRARY_PATH is empty.
  2. With all system modules cleaned by step 1, make a clean miniconda installation, like: 
    wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
bash miniconda.sh -b -p miniconda
. miniconda/etc/profile.d/conda.sh
conda create -n test-esmpy
conda install -c conda-forge esmpy
pip install xesmf
# then try your own code or run pytest
tommylees112 commented 4 years ago 
> echo $LD_LIBRARY_PATH

/opt/intel/compilers_and_libraries_2019.2.187/linux/compiler/lib/intel64_lin:/opt/intel//compilers_and_libraries_2019.2.187/linux/mpi/intel64/libfabric/lib:/opt/intel//compilers_and_libraries_2019.2.187/linux/mpi/intel64/lib/release:/opt/intel//compilers_and_libraries_2019.2.187/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2019.2.187/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2019.2.187/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2019.2.187/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2019.2.187/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2019.2.187/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2019.2.187/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2019.2.187/linux/compiler/lib/intel64_lin:/opt/intel//compilers_and_libraries_2019.2.187/linux/mpi/intel64/libfabric/lib:/opt/intel//compilers_and_libraries_2019.2.187/linux/mpi/intel64/lib/release:/opt/intel//compilers_and_libraries_2019.2.187/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2019.2.187/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2019.2.187/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2019.2.187/linux/mkl/lib/intel64_lin:/opt/intel/compilers_and_libraries_2019.2.187/linux/tbb/lib/intel64/gcc4.7:/opt/intel/compilers_and_libraries_2019.2.187/linux/tbb/lib/intel64/gcc4.7:/opt/intel/debugger_2019/libipt/intel64/lib:/opt/intel/compilers_and_libraries_2019.2.187/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2019.2.187/linux/daal/../tbb/lib/intel64_lin/gcc4.4

So I want to empty all those paths? LD_LIBRARY_PATH=''

To be fair when i do this teh errors just become flake errors!


(esowc-drought) chri4118@linux20:/soge-home/projects/crop_yield/ml_drought$ pytest -v --pyargs xesmf
================================================================ test session starts =================================================================
platform linux -- Python 3.7.3, pytest-4.4.2, py-1.8.0, pluggy-0.11.0 -- /soge-home/users/chri4118/.conda/envs/esowc-drought/bin/python
cachedir: .pytest_cache
rootdir: /soge-home/projects/crop_yield/ml_drought, inifile: pytest.ini
plugins: flake8-1.0.4
collected 30 items

__init__.py::FLAKE8 FAILED                                                                                                                     [  3%]
backend.py::FLAKE8 FAILED                                                                                                                      [  6%]
data.py::FLAKE8 FAILED                                                                                                                         [ 10%]
frontend.py::FLAKE8 FAILED                                                                                                                     [ 13%]
smm.py::FLAKE8 FAILED                                                                                                                          [ 16%]
util.py::FLAKE8 FAILED                                                                                                                         [ 20%]
tests/__init__.py::FLAKE8 PASSED                                                                                                               [ 23%]
tests/test_backend.py::FLAKE8 FAILED                                                                                                           [ 26%]
tests/test_backend.py::test_flag PASSED                                                                                                        [ 30%]
tests/test_backend.py::test_warn_f_on_array PASSED                                                                                             [ 33%]
tests/test_backend.py::test_warn_f_on_grid PASSED                                                                                              [ 36%]
tests/test_backend.py::test_warn_lat_range PASSED                                                                                              [ 40%]
tests/test_backend.py::test_esmf_grid_with_corner PASSED                                                                                       [ 43%]
tests/test_backend.py::test_esmf_build_bilinear PASSED                                                                                         [ 46%]
tests/test_backend.py::test_regrid PASSED                                                                                                      [ 50%]
tests/test_backend.py::test_regrid_periodic_wrong PASSED                                                                                       [ 53%]
tests/test_backend.py::test_regrid_periodic_correct PASSED                                                                                     [ 56%]
tests/test_frontend.py::FLAKE8 FAILED                                                                                                          [ 60%]
tests/test_frontend.py::test_as_2d_mesh PASSED                                                                                                 [ 63%]
tests/test_frontend.py::test_build_regridder PASSED                                                                                            [ 66%]
tests/test_frontend.py::test_existing_weights PASSED                                                                                           [ 70%]
tests/test_frontend.py::test_conservative_without_bounds PASSED                                                                                [ 73%]
tests/test_frontend.py::test_build_regridder_from_dict PASSED                                                                                  [ 76%]
tests/test_frontend.py::test_regrid PASSED                                                                                                     [ 80%]
tests/test_frontend.py::test_regrid_periodic_wrong PASSED                                                                                      [ 83%]
tests/test_frontend.py::test_regrid_periodic_correct PASSED                                                                                    [ 86%]
tests/test_frontend.py::test_regrid_with_1d_grid PASSED                                                                                        [ 90%]
tests/test_util.py::FLAKE8 PASSED                                                                                                              [ 93%]
tests/test_util.py::test_grid_global PASSED                                                                                                    [ 96%]
tests/test_util.py::test_grid_global_bad_resolution PASSED                                                                                     [100%]

====================================================================== FAILURES ======================================================================
_______________________________________ FLAKE8-check(ignoring E701 // allows multiple statements on one line) ________________________________________
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/__init__.py:1:1: F401 '.util' imported but unused
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/__init__.py:2:1: F401 '.data' imported but unused
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/__init__.py:3:1: F401 '.frontend.Regridder' imported but unused

_______________________________________ FLAKE8-check(ignoring E701 // allows multiple statements on one line) ________________________________________
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/backend.py:145:54: E226 missing whitespace around arithmetic operator
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/backend.py:146:12: E126 continuation line over-indented for hanging indent
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/backend.py:148:12: E126 continuation line over-indented for hanging indent
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/backend.py:213:5: E722 do not use bare 'except'

_______________________________________ FLAKE8-check(ignoring E701 // allows multiple statements on one line) ________________________________________
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/data.py:26:27: W605 invalid escape sequence '\c'
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/data.py:26:43: W605 invalid escape sequence '\c'
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/data.py:26:50: W605 invalid escape sequence '\p'
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/data.py:40:14: E226 missing whitespace around arithmetic operator
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/data.py:40:20: E226 missing whitespace around arithmetic operator
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/data.py:41:14: E226 missing whitespace around arithmetic operator
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/data.py:41:20: E226 missing whitespace around arithmetic operator
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/data.py:43:38: E226 missing whitespace around arithmetic operator

_______________________________________ FLAKE8-check(ignoring E701 // allows multiple statements on one line) ________________________________________
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/frontend.py:143:13: E722 do not use bare 'except'
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/frontend.py:152:13: E722 do not use bare 'except'
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/frontend.py:284:14: E126 continuation line over-indented for hanging indent
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/frontend.py:286:14: E123 closing bracket does not match indentation of opening bracket's line
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/frontend.py:329:46: E226 missing whitespace around arithmetic operator

_______________________________________ FLAKE8-check(ignoring E701 // allows multiple statements on one line) ________________________________________
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/smm.py:82:43: E226 missing whitespace around arithmetic operator

_______________________________________ FLAKE8-check(ignoring E701 // allows multiple statements on one line) ________________________________________
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/util.py:25:38: E226 missing whitespace around arithmetic operator
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/util.py:26:41: E226 missing whitespace around arithmetic operator
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/util.py:90:26: E226 missing whitespace around arithmetic operator
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/util.py:90:37: E226 missing whitespace around arithmetic operator
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/util.py:94:26: E226 missing whitespace around arithmetic operator
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/util.py:94:37: E226 missing whitespace around arithmetic operator

_______________________________________ FLAKE8-check(ignoring E701 // allows multiple statements on one line) ________________________________________
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/tests/test_backend.py:164:42: E226 missing whitespace around arithmetic operator
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/tests/test_backend.py:205:42: E226 missing whitespace around arithmetic operator
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/tests/test_backend.py:224:42: E226 missing whitespace around arithmetic operator

_______________________________________ FLAKE8-check(ignoring E701 // allows multiple statements on one line) ________________________________________
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/tests/test_frontend.py:97:44: E226 missing whitespace around arithmetic operator
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/tests/test_frontend.py:127:44: E226 missing whitespace around arithmetic operator
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/tests/test_frontend.py:140:44: E226 missing whitespace around arithmetic operator
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/xesmf/tests/test_frontend.py:164:44: E226 missing whitespace around arithmetic operator

================================================================== warnings summary ==================================================================
/lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/heapdict.py:11
  /lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/heapdict.py:11: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    class heapdict(collections.MutableMapping):

data.py::FLAKE8
  :38: DeprecationWarning: invalid escape sequence \c

tests/test_backend.py::test_regrid
tests/test_frontend.py::test_build_regridder
tests/test_frontend.py::test_build_regridder
tests/test_frontend.py::test_build_regridder
tests/test_frontend.py::test_build_regridder
tests/test_frontend.py::test_existing_weights
tests/test_frontend.py::test_existing_weights
tests/test_frontend.py::test_build_regridder_from_dict
tests/test_frontend.py::test_regrid
tests/test_frontend.py::test_regrid_periodic_wrong
tests/test_frontend.py::test_regrid_periodic_correct
tests/test_frontend.py::test_regrid_with_1d_grid
  /lustre/soge1/users/chri4118/.conda/envs/esowc-drought/lib/python3.7/site-packages/ESMF/interface/cbindings.py:2037: DeprecationWarning: Call to deprecated function ESMP_FieldRegridStoreFile.
    def ESMP_FieldRegridStoreFile(srcField, dstField, filename,

-- Docs: https://docs.pytest.org/en/latest/warnings.html
================================================= 8 failed, 22 passed, 14 warnings in 19.91 seconds ==================================================
JiaweiZhuang commented 4 years ago

Aha, so seems like problem solved? I bet ESMF was linking the Intel MPI instead of the conda-installed MPI (I remember it is MPICH)...

JiaweiZhuang commented 4 years ago

MPI is required to compile ESMF, but xesmf is actually not using any MPI functionalities (either mpirun or MPI.Spawn ). So this MPI dependency unfortunately just complicates the installation process...

JiaweiZhuang commented 4 years ago

So I want to empty all those paths?

Side note: Ideally you should use higher-level commands like module purge (if available) instead of changing LD_LIBRARY_PATH directly

JiaweiZhuang commented 4 years ago

Close if no further issues