search

JiaweiZhuang / xESMF

Universal Regridder for Geospatial Data
http://xesmf.readthedocs.io/
MIT License
269 stars 49 forks source link

MPI error #68

Open rabernat opened 4 years ago

rabernat commented 4 years ago

I tried to run the curvilinear example from the docs on a relatively fresh conda environment on mac os.

When building the grid weights, I hit the same issue that came up in https://github.com/pydata/xarray/issues/2281#issuecomment-497129090.

Fatal error in MPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(572)..............: 
MPID_Init(224).....................: channel initialization failed
MPIDI_CH3_Init(105)................: 
MPID_nem_init(324).................: 
MPID_nem_tcp_init(178).............: 
MPID_nem_tcp_get_business_card(425): 
MPID_nem_tcp_init(384).............: gethostbyname failed, Ryans-MacBook-Pro.local (errno 1)
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=3191567
:
system msg for write_line failure : Bad file descriptor

causing my kernel to die.

I know this is an ESMF/ESMPy issue, but I thought I would raise it here first.

I installed everything from conda-forge.

``` $ conda list # packages in environment at /Users/rpa/miniconda3/envs/pangeo: # # Name Version Build Channel alabaster 0.7.12 pypi_0 pypi alembic 1.2.1 pypi_0 pypi appdirs 1.4.3 py_1 conda-forge appnope 0.1.0 py36_1000 conda-forge asciitree 0.3.3 py_2 conda-forge asn1crypto 0.24.0 py36_1003 conda-forge atomicwrites 1.3.0 pypi_0 pypi attrs 19.1.0 py_0 conda-forge babel 2.7.0 pypi_0 pypi backcall 0.1.0 py_0 conda-forge basemap 1.2.1 py36hd103db2_1 conda-forge binutils-meta 1.0.4 0 conda-forge bleach 3.1.0 py_0 conda-forge blinker 1.4 py_1 conda-forge bokeh 1.3.4 py36_0 conda-forge boost-cpp 1.70.0 h75728bb_2 conda-forge boto3 1.9.229 py_0 conda-forge botocore 1.12.229 py_0 conda-forge bottleneck 1.2.1 py36h917ab60_1001 conda-forge bqplot 0.11.8 py_0 conda-forge bzip2 1.0.8 h01d97ff_1 conda-forge ca-certificates 2019.9.11 hecc5488_0 conda-forge cachetools 2.1.0 py_0 conda-forge cairo 1.16.0 he1c11cd_1002 conda-forge cartopy 0.17.0 py36hd103db2_1006 conda-forge cctools 921 h5ba7a2e_4 conda-forge certifi 2019.6.16 py36_1 conda-forge cffi 1.12.3 py36hccf1714_0 conda-forge cfgrib 0.9.7.1 py_0 conda-forge cfitsio 3.470 h84d2f63_2 conda-forge cftime 1.0.3.4 py36h1e5eb4f_1001 conda-forge chardet 3.0.4 py36_1003 conda-forge clang 9.0.0 h28b9765_0 conda-forge click 7.0 py_0 conda-forge click-plugins 1.1.1 py_0 conda-forge cligj 0.5.0 py_0 conda-forge cloudpickle 1.2.2 py_0 conda-forge colorcet 2.0.1 py_0 conda-forge commonmark 0.9.0 pypi_0 pypi configobj 5.0.6 py_0 conda-forge cryptography 2.7 py36h212c5bf_0 conda-forge curl 7.65.3 h22ea746_0 conda-forge cycler 0.10.0 py_1 conda-forge cython 0.29.13 py36h6de7cb9_0 conda-forge cytoolz 0.10.0 py36h01d97ff_0 conda-forge dask 2.4.0 py_0 conda-forge dask-core 2.4.0 py_0 conda-forge datashader 0.7.0 py_1 conda-forge datashape 0.5.4 py_1 conda-forge dbus 1.13.6 h2f22bb5_0 conda-forge decorator 4.4.0 py_0 conda-forge defusedxml 0.5.0 py_1 conda-forge distributed 2.4.0 py_0 conda-forge doctr 1.8.0 pypi_0 pypi docutils 0.15.2 py36_0 conda-forge eccodes 2.13.1 h43bd019_1 conda-forge entrypoints 0.3 py36_1000 conda-forge esmf 7.1.0 h963e782_1008 conda-forge esmpy 7.1.0 py36h24bf2e0_3 conda-forge expat 2.2.5 h6de7cb9_1003 conda-forge fasteners 0.14.1 py_3 conda-forge fiona 1.8.6 py36h7eb7563_3 conda-forge fontconfig 2.13.1 h6b1039f_1001 conda-forge fortran-compiler 1.0.4 he991be0_0 conda-forge freetype 2.10.0 h24853df_1 conda-forge freexl 1.0.5 h1de35cc_1002 conda-forge fsspec 0.4.5 py_0 conda-forge future 0.17.1 pypi_0 pypi fuzzywuzzy 0.17.0 pypi_0 pypi gcsfs 0.3.0 py_1 conda-forge gdal 2.4.1 py36h39889d8_8 conda-forge geopandas 0.5.1 py_0 conda-forge geos 3.7.2 h6de7cb9_2 conda-forge geotiff 1.4.3 h113155d_1001 conda-forge geoviews 1.6.3 py_0 conda-forge geoviews-core 1.6.3 py_0 conda-forge gettext 0.19.8.1 h46ab8bc_1002 conda-forge gfortran_impl_osx-64 7.3.0 hf4212f2_2 conda-forge gfortran_osx-64 7.3.0 h22b1bf0_5 conda-forge giflib 5.1.7 h01d97ff_1 conda-forge glib 2.58.3 h9d45998_1002 conda-forge gmp 6.1.2 h0a44026_1000 conda-forge google-auth 1.6.3 py_0 conda-forge google-auth-oauthlib 0.4.0 py_0 conda-forge gsw 3.3.1 py36h01d97ff_0 conda-forge h5py 2.10.0 nompi_py36h6248fd5_100 conda-forge hdf4 4.2.13 hf3c6af0_1002 conda-forge hdf5 1.10.5 nompi_h15a436c_1103 conda-forge heapdict 1.0.0 py36_1000 conda-forge holoviews 1.12.5 py_0 pyviz icu 64.2 h6de7cb9_1 conda-forge idna 2.8 py36_1000 conda-forge imageio 2.5.0 py36_0 conda-forge imagesize 1.1.0 pypi_0 pypi importlib-metadata 0.23 pypi_0 pypi intake 0.5.3 py_0 conda-forge intake-xarray 0.3.1 py_0 conda-forge ipykernel 5.1.2 py36h5ca1d4c_0 conda-forge ipython 7.8.0 py36h5ca1d4c_0 conda-forge ipython_genutils 0.2.0 py_1 conda-forge ipywidgets 7.5.1 py_0 conda-forge isl 0.19 0 conda-forge jasper 1.900.1 h636a363_1006 conda-forge jedi 0.15.1 py36_0 conda-forge jinja2 2.10.1 py_0 conda-forge jmespath 0.9.4 py_0 conda-forge joblib 0.13.2 py_0 conda-forge jpeg 9c h1de35cc_1001 conda-forge json-c 0.13.1 h1de35cc_1001 conda-forge json5 0.8.5 py_0 conda-forge jsonschema 3.0.2 py36_0 conda-forge jupyter 1.0.0 pypi_0 pypi jupyter-book 0.5.2 pypi_0 pypi jupyter-contrib-core 0.3.3 pypi_0 pypi jupyter-contrib-nbextensions 0.5.1 pypi_0 pypi jupyter-highlight-selected-word 0.2.0 pypi_0 pypi jupyter-latex-envs 1.4.6 pypi_0 pypi jupyter-nbextensions-configurator 0.4.1 pypi_0 pypi jupyter_client 5.3.1 py_0 conda-forge jupyter_console 6.0.0 py_0 conda-forge jupyter_core 4.4.0 py_0 conda-forge jupyterlab 1.1.3 py_0 conda-forge jupyterlab_server 1.0.6 py_0 conda-forge kealib 1.4.10 h6659575_1005 conda-forge kiwisolver 1.1.0 py36h770b8ee_0 conda-forge krb5 1.16.3 hcfa6398_1001 conda-forge latexcodec 1.0.7 pypi_0 pypi ld64 409.12 h3c32e8a_4 conda-forge libaec 1.0.4 h0a44026_0 conda-forge libblas 3.8.0 12_openblas conda-forge libcblas 3.8.0 12_openblas conda-forge libcurl 7.65.3 h16faf7d_0 conda-forge libcxx 8.0.1 0 conda-forge libcxxabi 8.0.1 0 conda-forge libdap4 3.20.4 habf5908_0 conda-forge libedit 3.1.20170329 hcfe32e1_1001 conda-forge libffi 3.2.1 h6de7cb9_1006 conda-forge libgdal 2.4.1 h75d216d_8 conda-forge libgfortran 4.0.0 2 conda-forge libiconv 1.15 h01d97ff_1005 conda-forge libkml 1.3.0 hed7d534_1010 conda-forge liblapack 3.8.0 12_openblas conda-forge libllvm9 9.0.0 h770b8ee_2 conda-forge libnetcdf 4.6.2 h1a02027_1003 conda-forge libopenblas 0.3.7 h4bb4525_1 conda-forge libpng 1.6.37 h2573ce8_0 conda-forge libpq 11.5 h756f0eb_1 conda-forge libsodium 1.0.17 h01d97ff_0 conda-forge libspatialindex 1.9.0 h6de7cb9_1 conda-forge libspatialite 4.3.0a hb3e03d6_1030 conda-forge libssh2 1.8.2 hcdc9a53_2 conda-forge libtiff 4.0.10 hd08fb8f_1003 conda-forge libuv 1.30.1 h01d97ff_0 conda-forge libxml2 2.9.9 h12c6b28_5 conda-forge llvm-openmp 8.0.1 h770b8ee_0 conda-forge llvmlite 0.29.0 py36h3fea490_1 conda-forge locket 0.2.0 py_2 conda-forge lxml 4.4.1 pypi_0 pypi lz4-c 1.8.3 h6de7cb9_1001 conda-forge mako 1.1.0 pypi_0 pypi markdown 2.6.11 pypi_0 pypi markupsafe 1.1.1 py36h1de35cc_0 conda-forge matplotlib 3.1.1 py36_1 conda-forge matplotlib-base 3.1.1 py36h3a684a6_1 conda-forge mistune 0.8.4 py36h1de35cc_1000 conda-forge monotonic 1.5 py_0 conda-forge more-itertools 7.2.0 pypi_0 pypi mpc 1.1.0 h3aece09_1006 conda-forge mpfr 4.0.2 h44b798e_0 conda-forge mpi 1.0 mpich conda-forge mpich 3.3.1 hc856adb_1 conda-forge msgpack-numpy 0.4.4.3 py_0 conda-forge msgpack-python 0.6.1 py36h04f5b5a_0 conda-forge multipledispatch 0.6.0 py_0 conda-forge munch 2.3.2 py_0 conda-forge nbclean 0.3.2 pypi_0 pypi nbconvert 5.4.1 pypi_0 pypi nbformat 4.4.0 py_1 conda-forge nbgrader 0.6.0 pypi_0 pypi nbsphinx 0.4.2 pypi_0 pypi ncurses 6.1 h0a44026_1002 conda-forge netcdf-fortran 4.4.5 h1993a31_1004 conda-forge netcdf4 1.5.1.2 py36h876e625_1 conda-forge networkx 2.3 py_0 conda-forge nodejs 12.8.1 hec2bf70_0 conda-forge notebook 6.0.1 py36_0 conda-forge numba 0.45.1 py36h86efe34_0 conda-forge numcodecs 0.6.3 py36h0a44026_0 conda-forge numpy 1.17.2 py36h6b0580a_0 conda-forge oauthlib 3.0.1 py_0 conda-forge olefile 0.46 py_0 conda-forge openjpeg 2.3.1 hdc36067_1 conda-forge openssl 1.1.1c h01d97ff_0 conda-forge oset 0.1.3 pypi_0 pypi owslib 0.18.0 py_0 conda-forge packaging 19.0 py_0 conda-forge pandas 0.25.1 py36h86efe34_0 conda-forge pandoc 2.7.3 0 conda-forge pandocfilters 1.4.2 py_1 conda-forge panel 0.6.2 py_0 pyviz param 1.9.1 py_0 conda-forge parso 0.5.1 py_0 conda-forge partd 1.0.0 py_0 conda-forge patsy 0.5.1 py_0 conda-forge pcre 8.41 h0a44026_1003 conda-forge pexpect 4.7.0 py36_0 conda-forge pickleshare 0.7.5 py36_1000 conda-forge pillow 6.1.0 py36h75ffe9a_1 conda-forge pip 19.2.3 py36_0 conda-forge pixman 0.38.0 h01d97ff_1003 conda-forge pluggy 0.13.0 pypi_0 pypi poppler 0.67.0 hd5eb092_7 conda-forge poppler-data 0.4.9 1 conda-forge postgresql 11.5 h25afefd_1 conda-forge proj4 5.2.0 h6de7cb9_1006 conda-forge prometheus_client 0.7.1 py_0 conda-forge prompt_toolkit 2.0.9 py_0 conda-forge psutil 5.6.3 py36h01d97ff_0 conda-forge ptyprocess 0.6.0 py_1001 conda-forge py 1.8.0 pypi_0 pypi pyasn1 0.4.6 py_0 conda-forge pyasn1-modules 0.2.6 py_0 conda-forge pybtex 0.22.2 pypi_0 pypi pybtex-docutils 0.2.1 pypi_0 pypi pycparser 2.19 py36_1 conda-forge pyct 0.4.6 py_0 conda-forge pyct-core 0.4.6 py_0 conda-forge pyepsg 0.4.0 py_0 conda-forge pygments 2.4.2 py_0 conda-forge pyjwt 1.7.1 py_0 conda-forge pykdtree 1.3.1 py36h917ab60_1002 conda-forge pyopenssl 19.0.0 py36_0 conda-forge pyparsing 2.4.2 py_0 conda-forge pyproj 1.9.6 py36h01d97ff_1002 conda-forge pyqt 5.9.2 py36h2a560b1_4 conda-forge pyresample 1.13.0 py36h86efe34_0 conda-forge pyrsistent 0.15.4 py36h01d97ff_0 conda-forge pyshp 2.1.0 py_0 conda-forge pysocks 1.7.0 py36_0 conda-forge pytest 5.1.2 pypi_0 pypi python 3.6.7 h94afb7f_1005 conda-forge python-dateutil 2.8.0 py_0 conda-forge python-editor 1.0.4 pypi_0 pypi python-snappy 0.5.4 py36h1e06ddd_0 conda-forge pytz 2019.2 py_0 conda-forge pyviz_comms 0.7.2 py_0 conda-forge pywavelets 1.0.3 py36h1e5eb4f_1 conda-forge pyyaml 5.1.2 py36h01d97ff_0 conda-forge pyzmq 18.1.0 py36hee98d25_0 conda-forge qt 5.9.7 h8cf7e54_3 conda-forge qtconsole 4.5.5 py_0 conda-forge readline 8.0 hcfe32e1_0 conda-forge recommonmark 0.6.0 pypi_0 pypi requests 2.22.0 py36_1 conda-forge requests-oauthlib 1.2.0 py_0 conda-forge rise 5.5.1 py36_0 conda-forge rsa 3.4.2 py_1 conda-forge rtree 0.8.3 py36h666c49c_1002 conda-forge ruamel.yaml 0.16.5 py36h01d97ff_1 conda-forge ruamel.yaml.clib 0.1.2 py36h01d97ff_0 conda-forge s3fs 0.3.4 py_0 conda-forge s3transfer 0.2.1 py36_0 conda-forge scikit-image 0.15.0 py36h86efe34_2 conda-forge scikit-learn 0.21.3 py36hd4ffd6c_0 conda-forge scipy 1.3.1 py36h7e0e109_2 conda-forge seaborn 0.9.0 py_1 conda-forge send2trash 1.5.0 py_0 conda-forge setuptools 41.2.0 py36_0 conda-forge shapely 1.6.4 py36h0567c5e_1006 conda-forge sip 4.19.8 py36h0a44026_1000 conda-forge six 1.12.0 py36_1000 conda-forge snappy 1.1.7 h6de7cb9_1002 conda-forge snowballstemmer 1.9.1 pypi_0 pypi sortedcontainers 2.1.0 py_0 conda-forge sphinx 1.6.5 pypi_0 pypi sphinx-bootstrap-theme 0.7.1 pypi_0 pypi sphinx-copybutton 0.2.5 pypi_0 pypi sphinx-jinja 1.1.0 pypi_0 pypi sphinx-markdown-tables 0.0.9 pypi_0 pypi sphinx-nbexamples 0.4.0 pypi_0 pypi sphinx-pangeo-theme 0.1.0 pypi_0 pypi sphinxcontrib-bibtex 0.4.2 pypi_0 pypi sphinxcontrib-srclinks 0.2.4 pypi_0 pypi sphinxcontrib-websupport 1.1.2 pypi_0 pypi sqlalchemy 1.3.8 pypi_0 pypi sqlite 3.29.0 hb7d70f7_1 conda-forge statsmodels 0.10.1 py36heacc8b8_0 conda-forge tapi 1000.10.8 h770b8ee_3 conda-forge tblib 1.4.0 py_0 conda-forge terminado 0.8.2 py36_0 conda-forge testpath 0.4.2 py_1001 conda-forge tk 8.6.9 h2573ce8_1003 conda-forge toolz 0.10.0 py_0 conda-forge tornado 6.0.3 py36h01d97ff_0 conda-forge tqdm 4.35.0 py_0 conda-forge traitlets 4.3.2 py36_1000 conda-forge traittypes 0.2.1 py_1 conda-forge tzcode 2019a h01d97ff_1002 conda-forge urllib3 1.25.3 py36_0 conda-forge wcwidth 0.1.7 py_1 conda-forge webencodings 0.5.1 py_1 conda-forge wheel 0.33.6 py36_0 conda-forge widgetsnbextension 3.5.1 py36_0 conda-forge xarray 0.12.3+71.gf7c12a9a dev_0 xcape 0+untagged.15.gfe33fdd.dirty dev_0 xerces-c 3.2.2 hbda6038_1004 conda-forge xesmf 0.2.1 py_0 conda-forge xhistogram 0.1.1 dev_0 xz 5.2.4 h1de35cc_1001 conda-forge yaml 0.1.7 h1de35cc_1001 conda-forge zarr 2.3.2 py36_0 conda-forge zeromq 4.3.2 h6de7cb9_2 conda-forge zict 1.0.0 py_0 conda-forge zipp 0.6.0 pypi_0 pypi zlib 1.2.11 h01d97ff_1006 conda-forge zstd 1.4.0 ha9f0a20_0 conda-forge ```
JiaweiZhuang commented 4 years ago

This error can happen on HPC clusters with pre-installed, incompatible ESMF/NetCDF/MPI, and can be solved by unset LD_LIBRARY_PATH or module purge before conda install (https://github.com/JiaweiZhuang/xESMF/issues/55#issuecomment-514298498) But on Mac OS there shouldn't be such an error, unless you have some MPI installed...

rabernat commented 4 years ago

Thanks for the quick reply!

But on Mac OS there shouldn't be such an error, unless you have some MPI installed...

Nothing special here, just a pretty vanilla macbook with a fresh conda environment.

These dependency problems are very hard / annoying to debug.

JiaweiZhuang commented 4 years ago

People do get the same error with Mac OS + Conda + MPI and there seems to be a solution: https://github.com/conda-forge/fenics-feedstock/issues/44#issuecomment-338164343