search

E3SM-Project / e3sm-unified

A metapackage for a unified anaconda environment for analyzing results from the Energy Exascale Earth System Model (E3SM).
BSD 3-Clause "New" or "Revised" License
8 stars 8 forks source link

e3sm_diags doesn't work in sysmpi environment #61

Closed xylar closed 4 years ago

xylar commented 4 years ago

At least on compy, no luck with e3sm_diags in the sysmpi envrionments.

From @chengzhuzhang:

e3sm_unified_1.3.0_sysmpi: it seems this environment doesn’t work with esmf. I tested e3sm_diags in an interactive session. I got following:

“Traceback (most recent call last):
File "/compyfs/software/e3sm-unified/base/envs/e3sm_unified_1.3.0_py3.7_sysmpi/lib/python3.7/site-packages/ESMF/interface/loadESMF.py", line 118, in <module>
mode=ct.RTLD_GLOBAL)
File "/compyfs/software/e3sm-unified/base/envs/e3sm_unified_1.3.0_py3.7_sysmpi/lib/python3.7/ctypes/__init__.py", line 356, in __init__
self._handle = _dlopen(self._name, mode)
OSError: /compyfs/software/e3sm-unified/base/envs/e3sm_unified_1.3.0_py3.7_sysmpi/lib/./libmpicxx.so.12: undefined symbol: MPII_Errhandler_set_cxx
Traceback (most recent call last):
File "/compyfs/software/e3sm-unified/base/envs/e3sm_unified_1.3.0_py3.7_sysmpi/lib/python3.7/site-packages/ESMF/interface/loadESMF.py", line 118, in <module>
mode=ct.RTLD_GLOBAL)
File "/compyfs/software/e3sm-unified/base/envs/e3sm_unified_1.3.0_py3.7_sysmpi/lib/python3.7/ctypes/__init__.py", line 356, in __init__
self._handle = _dlopen(self._name, mode)
OSError: /compyfs/software/e3sm-unified/base/envs/e3sm_unified_1.3.0_py3.7_sysmpi/lib/./libmpicxx.so.12: undefined symbol: MPII_Errhandler_set_cxx”

The diagnostics run was completed any way because cdms is able to pick up another regridder in this case.

These errors likely result from the custom-built esmf I included in this environment to remove MPI dependence. It may be that no packages that rely on esmf will work in the sysmpi environment or the problem may just lie with esmpy. More testing is needed.

xylar commented 4 years ago

I have decided not to pursue the sysmpi environment any further. It doesn't seem to work well for most of our workflow. It works with mpi4py, which can be built from source against the system MPI with relatively little difficulty but not for packages like libhdf5, libnetcdf and esmf that need to be compiled against system MPI and have a large number of other dependencies that need to be built with the same compilers.