search

paudetseis / Telewavesim

Teleseismic body wave modeling through stacks of (submarine/anisotropic) layers
https://paudetseis.github.io/Telewavesim/
MIT License
66 stars 18 forks source link

[Review] dependency issues with installation #31

Closed brmather closed 4 years ago

brmather commented 5 years ago

https://github.com/openjournals/joss-reviews/issues/1818

I am running into the following error with pip install . in the source directory:

    /usr/bin/ld: cannot find -llapack
    collect2: error: ld returned 1 exit status

which suggests that it cannot find my lapack installation. However, if I run pip install telewavesim then it installs fine. Is there something new in the repo that is preventing me from installing this properly?

In addition I run into the following issue when running the test suite:

tests/test_utils.py:2: in <module>
    from telewavesim import utils as ut
utils.py:33: in <module>
    from obspy.core import Trace, Stream
../../../.local/lib/python3.6/site-packages/obspy/__init__.py:40: in <module>
    from obspy.core.util import _get_version_string
../../../.local/lib/python3.6/site-packages/obspy/core/__init__.py:126: in <module>
    from obspy.core.trace import Stats, Trace  # NOQA
../../../.local/lib/python3.6/site-packages/obspy/core/trace.py:26: in <module>
    from obspy.core.util import AttribDict, create_empty_data_chunk
../../../.local/lib/python3.6/site-packages/obspy/core/util/__init__.py:27: in <module>
    from obspy.core.util.base import (ALL_MODULES, DEFAULT_MODULES,
../../../.local/lib/python3.6/site-packages/obspy/core/util/base.py:397: in <module>
    PROJ4_VERSION = get_proj_version()
../../../.local/lib/python3.6/site-packages/obspy/core/util/base.py:383: in get_proj_version
    version_string = str(Proj(proj='utm', zone=10, ellps='WGS84').proj_version)
E   AttributeError: 'Proj' object has no attribute 'proj_version'

which looks to be a dependency issue with obspy 1.1.1. Switching to conda may resolve these dependency issues, but I try to avoid Anaconda as it has the tendency to mess up my system python configuration. In light of this, I would suggest bundling a Docker image with Telewavesim - what do you think?

trichter commented 5 years ago

The package on pypi is outdated and I think it is reasonable to publish a new version only after the review. Therefore, please install from source. Are you running on Linux or OSX? The tests on travis-ci indicate it should work with recent versions of numpy both on Linux and OSX. Does it work for you if you delete the option libraries=['lapack'] in line 18 in setup.py?

I had a similar issue in #2.

brmather commented 5 years ago

I'm running Linux. Deleting libraries=['lapack'] line meant that I could install the package from source, but now I run into this error:

Traceback:
telewavesim/tests/test_utils.py:2: in <module>
    from telewavesim import utils as ut
telewavesim/utils.py:36: in <module>
    from telewavesim.rmat_f import conf as cf_f
E   ModuleNotFoundError: No module named 'telewavesim.rmat_f'

which looks like it's not compiling the fortran source correctly? The second issue I was having is not related to Telewavesim but rather a conflict in obspy with a later version of pyproj (https://github.com/obspy/obspy/issues/2445 and https://github.com/obspy/obspy/issues/2449).

trichter commented 5 years ago

It would not have surprised me, if you ran into ImpoertError - undefined symbol: dgeev_, but this error seems strange. Are you still in the source directory? Maybe it is trying to import from there. Can you try to run the tests from an empty folder?

trichter commented 5 years ago

In light of this, I would suggest bundling a Docker image with Telewavesim - what do you think?

In my opinion, this would be overkill. Hopefully, in the end, there will be package on pypi for which the installation is simply working out of the box.

brmather commented 5 years ago

I have compiled lapack from source with shared libraries using cmake. When I run

f2py -c *.f90 -L/usr/local/lib64 -llapack -m rmat_f

from the src folder it seems to compile fine but it returns an ImportError:

ImportError                               Traceback (most recent call last)
<ipython-input-1-553ea2ea12ab> in <module>
----> 1 import rmat_f

ImportError: liblapack.so.3: cannot open shared object file: No such file or directory

/usr/local/lib64/liblapack.so.3 is there, but somehow f2py fails to link to it.

$ ldd rmat_f.cpython-36m-x86_64-linux-gnu.so
    linux-vdso.so.1 (0x00007ffeb1569000)
    liblapack.so.3 => not found
    libpython3.6m.so.1.0 => /usr/lib/libpython3.6m.so.1.0 (0x00007f3987c75000)
    libgfortran.so.5 => /usr/lib/libgfortran.so.5 (0x00007f3987668000)
    libm.so.6 => /usr/lib/haswell/libm.so.6 (0x00007f3987528000)
    libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f398750d000)
    libquadmath.so.0 => /usr/lib/libquadmath.so.0 (0x00007f39874bd000)
    libc.so.6 => /usr/lib/haswell/libc.so.6 (0x00007f39872d4000)
    libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f39872b3000)
    libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f39872ad000)
    libutil.so.1 => /usr/lib/libutil.so.1 (0x00007f39872a8000)
    /usr/lib64/ld-linux-x86-64.so.2 (0x00007f3988030000)

Any advice on how to proceed would be appreciated.

trichter commented 5 years ago

I also do not know, what's wrong, but I have some ideas how to proceed:

brmather commented 5 years ago

creating /tmp/tmpiwu4ku4f/tmp creating /tmp/tmpiwu4ku4f/tmp/tmpiwu4ku4f creating /tmp/tmpiwu4ku4f/tmp/tmpiwu4ku4f/src.linux-x86_64-3.6 compile options: '-I/tmp/tmpiwu4ku4f/src.linux-x86_64-3.6 -I/home/ben/.local/lib/python3.6/site-packages/numpy/core/include -I/usr/include/python3.6m -c' x86_64-solus-linux-gcc: /tmp/tmpiwu4ku4f/src.linux-x86_64-3.6/fortranobject.c x86_64-solus-linux-gcc: /tmp/tmpiwu4ku4f/src.linux-x86_64-3.6/rmat_fmodule.c In file included from /home/ben/.local/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1830, from /home/ben/.local/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:12, from /home/ben/.local/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4, from /tmp/tmpiwu4ku4f/src.linux-x86_64-3.6/fortranobject.h:13, from /tmp/tmpiwu4ku4f/src.linux-x86_64-3.6/fortranobject.c:2: /home/ben/.local/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] 17 | #warning "Using deprecated NumPy API, disable it with " \ | ^~~ In file included from /home/ben/.local/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1830, from /home/ben/.local/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:12, from /home/ben/.local/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4, from /tmp/tmpiwu4ku4f/src.linux-x86_64-3.6/fortranobject.h:13, from /tmp/tmpiwu4ku4f/src.linux-x86_64-3.6/rmat_fmodule.c:16: /home/ben/.local/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] 17 | #warning "Using deprecated NumPy API, disable it with " \ | ^~~ /tmp/tmpiwu4ku4f/src.linux-x86_64-3.6/rmat_fmodule.c:144:12: warning: ‘f2py_size’ defined but not used [-Wunused-function] 144 | static int f2py_size(PyArrayObject* var, ...) | ^~~~~ compiling Fortran 90 module sources Fortran f77 compiler: /usr/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops Fortran f90 compiler: /usr/bin/gfortran -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops Fortran fix compiler: /usr/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops compile options: '-I/tmp/tmpiwu4ku4f/src.linux-x86_64-3.6 -I/home/ben/.local/lib/python3.6/site-packages/numpy/core/include -I/usr/include/python3.6m -c' extra options: '-J/tmp/tmpiwu4ku4f/ -I/tmp/tmpiwu4ku4f/' gfortran:f90: rmat.f90 compiling Fortran sources Fortran f77 compiler: /usr/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops Fortran f90 compiler: /usr/bin/gfortran -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops Fortran fix compiler: /usr/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops compile options: '-I/tmp/tmpiwu4ku4f/src.linux-x86_64-3.6 -I/home/ben/.local/lib/python3.6/site-packages/numpy/core/include -I/usr/include/python3.6m -c' extra options: '-J/tmp/tmpiwu4ku4f/ -I/tmp/tmpiwu4ku4f/' gfortran:f90: rmat_sub.f90 gfortran:f90: /tmp/tmpiwu4ku4f/src.linux-x86_64-3.6/rmat_f-f2pywrappers2.f90 /usr/bin/gfortran -Wall -g -Wall -g -shared /tmp/tmpiwu4ku4f/tmp/tmpiwu4ku4f/src.linux-x86_64-3.6/rmat_fmodule.o /tmp/tmpiwu4ku4f/tmp/tmpiwu4ku4f/src.linux-x86_64-3.6/fortranobject.o /tmp/tmpiwu4ku4f/rmat.o /tmp/tmpiwu4ku4f/rmat_sub.o /tmp/tmpiwu4ku4f/tmp/tmpiwu4ku4f/src.linux-x86_64-3.6/rmat_f-f2pywrappers2.o -L/usr/local/lib64 -L/usr/lib64/gcc/x86_64-solus-linux/9/../../../../lib64 -L/usr/lib64/gcc/x86_64-solus-linux/9/../../../../lib64 -L/usr/lib64 -llapack -lpython3.6m -lgfortran -o ./rmat_f.cpython-36m-x86_64-linux-gnu.so Removing build directory /tmp/tmpiwu4ku4f

trichter commented 5 years ago

The f2py output does not look suspicious.

I am not an expert in this linking stuff. Nevertheless, maybe answers to this stackoverflow question are helpful?

Does ldconfig -p | grep lapack output your freshly compiled library? Does cat /etc/ld.so.conf.d/*.conf | grep /usr/local/lib64 output your lib directory?

If not that might be the reason for the problems. I think you could solve this problem by adding a new file to the /etc/ld.so.conf.d folder with sudo echo "/usr/local/lib64" > /etc/ld.so.conf.d/usrlocallib64.conf running sudo ldconfig afterwards. (Alternatively check and set your $LD_LIBRARY_PATH environment variable).

brmather commented 5 years ago

That worked! I had to manually add /usr/local/lib64/ to my library path for rmat_f to link against. Also, the fix in https://github.com/paudetseis/Telewavesim/issues/2#issuecomment-538487241 was required otherwise the fortran sources do not detect lapack (even if it's in my library path). So long as other installation methods work with these changes to setup.py, they should be commited to master.

trichter commented 5 years ago

I pushed a similar fix to master. I would appreciate if you could test it with your setup.

paudetseis commented 4 years ago

Thanks @trichter . @brmather , did you have a chance to test the fix? Hopefully it's all good now and we can move on.

brmather commented 4 years ago

Hi @trichter and @paudetseis,

The workaround in #33 did not work for me. The function get_library_dirs in setup.py could not locate lapack on my system so I submitted PR #34. This re-implements numpy's get_info function to link lapack on compiling the fortran source code, and ensures the shared libraries are loaded on importing telewavesim. I found this forum thread quite helpful.

This will likely not be an issue for the majority of users on Anaconda, but it should harden the installation procedure for people like me on native installs.

Feel free to modify the pull request as you see fit. As it stands I'm happy to close this issue and move on.