ImpoertError - undefined symbol: dgeev_

[trichter]

Hi again, nice work!

I tried to install telewavesim into a conda environment. The Fortran code compiled successfully. But I got the following ImportError:

In [2]: import telewavesim.rmat_f                                                                                                                                                                                                             
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-2-f302e8277083> in <module>
----> 1 import telewavesim.rmat_f

ImportError: /home/eule/dev/other/Telewavesim/telewavesim/rmat_f.cpython-37m-x86_64-linux-gnu.so: undefined symbol: dgeev_

This appears to be a linking issue to lapack, but I do not know how to solve it. Maybe anybody has a clue. I am on 64bit Linux.

Other small things.

• Testing paragraph: git checkout should be git clone.
• The command conda env create -f tws_env.yml is not working for me: Solving environment: failed, ResolvePackageNotFound

[paudetseis]

Hi @trichter,

Thanks for flagging these issues. Did you install from source or using the PyPi package? I just tested both version on a Mac and both worked fine.

The lapack library is packaged with numpy and in theory you don't need the library installed separately. Unless this is not true for Linux. Can you post the result of conda list? Mine has liblapack=3.8.0 showing.

The default install works well with numpy=1.15.4, so perhaps you could try that version?

I'll fix those other bugs you found.

Thanks again!

[paudetseis]

I just made a new two_env.yml - maybe that can fix your install problem?

[trichter]

I tested both from source and from pypi. The output from conda list is

# packages in environment at /home/eule/anaconda/envs/telewavesim:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
asn1crypto                0.24.0                py37_1003    conda-forge
basemap                   1.2.0            py37hd759880_4    conda-forge
bzip2                     1.0.8                h516909a_1    conda-forge
ca-certificates           2019.9.11            hecc5488_0    conda-forge
certifi                   2019.9.11                py37_0    conda-forge
cffi                      1.12.3           py37h8022711_0    conda-forge
chardet                   3.0.4                 py37_1003    conda-forge
cryptography              2.7              py37h72c5cf5_0    conda-forge
cycler                    0.10.0                     py_1    conda-forge
dbus                      1.13.6               he372182_0    conda-forge
decorator                 4.4.0                      py_0    conda-forge
expat                     2.2.5             he1b5a44_1003    conda-forge
fontconfig                2.13.1            h86ecdb6_1001    conda-forge
freetype                  2.10.0               he983fc9_1    conda-forge
future                    0.17.1                py37_1000    conda-forge
geos                      3.7.2                he1b5a44_2    conda-forge
gettext                   0.19.8.1          hc5be6a0_1002    conda-forge
glib                      2.58.3            h6f030ca_1002    conda-forge
gst-plugins-base          1.14.5               h0935bb2_0    conda-forge
gstreamer                 1.14.5               h36ae1b5_0    conda-forge
icu                       64.2                 he1b5a44_1    conda-forge
idna                      2.8                   py37_1000    conda-forge
jpeg                      9c                h14c3975_1001    conda-forge
kiwisolver                1.1.0            py37hc9558a2_0    conda-forge
libblas                   3.8.0               12_openblas    conda-forge
libcblas                  3.8.0               12_openblas    conda-forge
libffi                    3.2.1             he1b5a44_1006    conda-forge
libgcc-ng                 9.1.0                hdf63c60_0  
libgfortran-ng            7.3.0                hdf63c60_0  
libiconv                  1.15              h516909a_1005    conda-forge
liblapack                 3.8.0               12_openblas    conda-forge
libopenblas               0.3.7                h6e990d7_1    conda-forge
libpng                    1.6.37               hed695b0_0    conda-forge
libstdcxx-ng              9.1.0                hdf63c60_0  
libuuid                   2.32.1            h14c3975_1000    conda-forge
libxcb                    1.13              h14c3975_1002    conda-forge
libxml2                   2.9.9                hee79883_5    conda-forge
libxslt                   1.1.33               h31b3aaa_0    conda-forge
lxml                      4.4.1            py37h7ec2d77_0    conda-forge
matplotlib                3.1.1                    py37_1    conda-forge
matplotlib-base           3.1.1            py37he7580a8_1    conda-forge
ncurses                   6.1               hf484d3e_1002    conda-forge
numpy                     1.15.4          py37h8b7e671_1002    conda-forge
obspy                     1.1.1            py37h3010b51_1    conda-forge
openssl                   1.1.1c               h516909a_0    conda-forge
pcre                      8.41              hf484d3e_1003    conda-forge
pip                       19.2.3                   py37_0    conda-forge
proj4                     4.9.3                h516909a_9    conda-forge
pthread-stubs             0.4               h14c3975_1001    conda-forge
pycparser                 2.19                     py37_1    conda-forge
pyfftw                    0.11.1          py37h3010b51_1001    conda-forge
pyopenssl                 19.0.0                   py37_0    conda-forge
pyparsing                 2.4.2                      py_0    conda-forge
pyproj                    1.9.5.1         py37h2944ce7_1006    conda-forge
pyqt                      5.9.2            py37hcca6a23_4    conda-forge
pyshp                     2.1.0                      py_0    conda-forge
pysocks                   1.7.1                    py37_0    conda-forge
python                    3.7.3                h33d41f4_1    conda-forge
python-dateutil           2.8.0                      py_0    conda-forge
qt                        5.9.7                h0c104cb_3    conda-forge
readline                  8.0                  hf8c457e_0    conda-forge
requests                  2.22.0                   py37_1    conda-forge
scipy                     1.3.1            py37h921218d_2    conda-forge
setuptools                41.2.0                   py37_0    conda-forge
sip                       4.19.8          py37hf484d3e_1000    conda-forge
six                       1.12.0                py37_1000    conda-forge
sqlalchemy                1.3.8            py37h516909a_0    conda-forge
sqlite                    3.29.0               hcee41ef_1    conda-forge
telewavesim               0.1.0                     dev_0    <develop>
tk                        8.6.9             hed695b0_1003    conda-forge
tornado                   6.0.3            py37h516909a_0    conda-forge
urllib3                   1.25.6                   py37_0    conda-forge
wheel                     0.33.6                   py37_0    conda-forge
xorg-libxau               1.0.9                h14c3975_0    conda-forge
xorg-libxdmcp             1.1.3                h516909a_0    conda-forge
xz                        5.2.4             h14c3975_1001    conda-forge
zlib                      1.2.11            h516909a_1006    conda-forge

numpy and liblapack are the same versions as you use. I come back here once I have a solution.

For reference, the output from pip install -e . -v is:

(telewavesim) eule@kiste:~/dev/other/Telewavesim$  pip install -e . -v
Created temporary directory: /tmp/pip-ephem-wheel-cache-8pxhu3ys
Created temporary directory: /tmp/pip-req-tracker-rh7mjymh
Created requirements tracker '/tmp/pip-req-tracker-rh7mjymh'
Created temporary directory: /tmp/pip-install-9yvimatm
Obtaining file:///home/eule/dev/other/Telewavesim
  Added file:///home/eule/dev/other/Telewavesim to build tracker '/tmp/pip-req-tracker-rh7mjymh'
    Running setup.py (path:/home/eule/dev/other/Telewavesim/setup.py) egg_info for package from file:///home/eule/dev/other/Telewavesim
    Running command python setup.py egg_info
    running egg_info
    running build_src
    build_src
    building extension "telewavesim.rmat_f" sources
    f2py options: []
      adding 'build/src.linux-x86_64-3.7/build/src.linux-x86_64-3.7/telewavesim/fortranobject.c' to sources.
      adding 'build/src.linux-x86_64-3.7/build/src.linux-x86_64-3.7/telewavesim' to include_dirs.
      adding 'build/src.linux-x86_64-3.7/telewavesim/rmat_f-f2pywrappers2.f90' to sources.
    build_src: building npy-pkg config files
    writing telewavesim.egg-info/PKG-INFO
    writing dependency_links to telewavesim.egg-info/dependency_links.txt
    writing requirements to telewavesim.egg-info/requires.txt
    writing top-level names to telewavesim.egg-info/top_level.txt
    reading manifest file 'telewavesim.egg-info/SOURCES.txt'
    writing manifest file 'telewavesim.egg-info/SOURCES.txt'
  Source in /home/eule/dev/other/Telewavesim has version 0.1.0, which satisfies requirement telewavesim==0.1.0 from file:///home/eule/dev/other/Telewavesim
  Removed telewavesim==0.1.0 from file:///home/eule/dev/other/Telewavesim from build tracker '/tmp/pip-req-tracker-rh7mjymh'
Requirement already satisfied: numpy>=1.15 in /home/eule/anaconda/envs/telewavesim/lib/python3.7/site-packages (from telewavesim==0.1.0) (1.15.4)
Requirement already satisfied: obspy>=1.0.0 in /home/eule/anaconda/envs/telewavesim/lib/python3.7/site-packages (from telewavesim==0.1.0) (1.1.1)
Requirement already satisfied: pyfftw>=0.11.1 in /home/eule/anaconda/envs/telewavesim/lib/python3.7/site-packages (from telewavesim==0.1.0) (0.11.1)
Requirement already satisfied: matplotlib in /home/eule/anaconda/envs/telewavesim/lib/python3.7/site-packages (from telewavesim==0.1.0) (3.1.1)
Requirement already satisfied: future>=0.12.4 in /home/eule/anaconda/envs/telewavesim/lib/python3.7/site-packages (from obspy>=1.0.0->telewavesim==0.1.0) (0.17.1)
Requirement already satisfied: scipy>=0.9.0 in /home/eule/anaconda/envs/telewavesim/lib/python3.7/site-packages (from obspy>=1.0.0->telewavesim==0.1.0) (1.3.1)
Requirement already satisfied: lxml in /home/eule/anaconda/envs/telewavesim/lib/python3.7/site-packages (from obspy>=1.0.0->telewavesim==0.1.0) (4.4.1)
Requirement already satisfied: setuptools in /home/eule/anaconda/envs/telewavesim/lib/python3.7/site-packages (from obspy>=1.0.0->telewavesim==0.1.0) (41.2.0)
Requirement already satisfied: sqlalchemy in /home/eule/anaconda/envs/telewavesim/lib/python3.7/site-packages (from obspy>=1.0.0->telewavesim==0.1.0) (1.3.8)
Requirement already satisfied: decorator in /home/eule/anaconda/envs/telewavesim/lib/python3.7/site-packages (from obspy>=1.0.0->telewavesim==0.1.0) (4.4.0)
Requirement already satisfied: requests in /home/eule/anaconda/envs/telewavesim/lib/python3.7/site-packages (from obspy>=1.0.0->telewavesim==0.1.0) (2.22.0)
Requirement already satisfied: python-dateutil>=2.1 in /home/eule/anaconda/envs/telewavesim/lib/python3.7/site-packages (from matplotlib->telewavesim==0.1.0) (2.8.0)
Requirement already satisfied: cycler>=0.10 in /home/eule/anaconda/envs/telewavesim/lib/python3.7/site-packages (from matplotlib->telewavesim==0.1.0) (0.10.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /home/eule/anaconda/envs/telewavesim/lib/python3.7/site-packages (from matplotlib->telewavesim==0.1.0) (2.4.2)
Requirement already satisfied: kiwisolver>=1.0.1 in /home/eule/anaconda/envs/telewavesim/lib/python3.7/site-packages (from matplotlib->telewavesim==0.1.0) (1.1.0)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /home/eule/anaconda/envs/telewavesim/lib/python3.7/site-packages (from requests->obspy>=1.0.0->telewavesim==0.1.0) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/eule/anaconda/envs/telewavesim/lib/python3.7/site-packages (from requests->obspy>=1.0.0->telewavesim==0.1.0) (1.25.6)
Requirement already satisfied: idna<2.9,>=2.5 in /home/eule/anaconda/envs/telewavesim/lib/python3.7/site-packages (from requests->obspy>=1.0.0->telewavesim==0.1.0) (2.8)
Requirement already satisfied: certifi>=2017.4.17 in /home/eule/anaconda/envs/telewavesim/lib/python3.7/site-packages (from requests->obspy>=1.0.0->telewavesim==0.1.0) (2019.9.11)
Requirement already satisfied: six>=1.5 in /home/eule/anaconda/envs/telewavesim/lib/python3.7/site-packages (from python-dateutil>=2.1->matplotlib->telewavesim==0.1.0) (1.12.0)
Installing collected packages: telewavesim
  Running setup.py develop for telewavesim
    Running command /home/eule/anaconda/envs/telewavesim/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/eule/dev/other/Telewavesim/setup.py'"'"'; __file__='"'"'/home/eule/dev/other/Telewavesim/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
    running develop
    running build_scripts
    running egg_info
    running build_src
    build_src
    building extension "telewavesim.rmat_f" sources
    f2py options: []
      adding 'build/src.linux-x86_64-3.7/build/src.linux-x86_64-3.7/telewavesim/fortranobject.c' to sources.
      adding 'build/src.linux-x86_64-3.7/build/src.linux-x86_64-3.7/telewavesim' to include_dirs.
      adding 'build/src.linux-x86_64-3.7/telewavesim/rmat_f-f2pywrappers2.f90' to sources.
    build_src: building npy-pkg config files
    writing telewavesim.egg-info/PKG-INFO
    writing dependency_links to telewavesim.egg-info/dependency_links.txt
    writing requirements to telewavesim.egg-info/requires.txt
    writing top-level names to telewavesim.egg-info/top_level.txt
    reading manifest file 'telewavesim.egg-info/SOURCES.txt'
    writing manifest file 'telewavesim.egg-info/SOURCES.txt'
    running build_ext
    customize UnixCCompiler
    C compiler: /home/eule/anaconda/envs/telewavesim/bin/x86_64-conda_cos6-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -Wstrict-prototypes -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -fPIC

    creating /tmp/tmp3fobdqbv/tmp
    creating /tmp/tmp3fobdqbv/tmp/tmp3fobdqbv
    compile options: '-MMD -MF /tmp/tmp3fobdqbv/file.c.d -c'
    x86_64-conda_cos6-linux-gnu-cc: /tmp/tmp3fobdqbv/file.c
    customize UnixCCompiler using build_ext
    get_default_fcompiler: matching types: '['gnu95', 'intel', 'lahey', 'pg', 'absoft', 'nag', 'vast', 'compaq', 'intele', 'intelem', 'gnu', 'g95', 'pathf95', 'nagfor']'
    customize Gnu95FCompiler
    Found executable /home/eule/anaconda/envs/telewavesim/bin/x86_64-conda_cos6-linux-gnu-gfortran
    Found executable /home/eule/anaconda/envs/telewavesim/bin/x86_64-conda_cos6-linux-gnu-ld
    Found executable /home/eule/anaconda/envs/telewavesim/bin/x86_64-conda_cos6-linux-gnu-ar
    Found executable /home/eule/anaconda/envs/telewavesim/bin/x86_64-conda_cos6-linux-gnu-ranlib
    customize Gnu95FCompiler
    customize Gnu95FCompiler using build_ext
    Creating /home/eule/anaconda/envs/telewavesim/lib/python3.7/site-packages/telewavesim.egg-link (link to .)
    Adding telewavesim 0.1.0 to easy-install.pth file

    Installed /home/eule/dev/other/Telewavesim
Successfully installed telewavesim
Cleaning up...
Removed build tracker '/tmp/pip-req-tracker-rh7mjymh'

I just made a new two_env.yml - maybe that can fix your install problem?

I still have the same error. I think this combination of versions is simply not available for my distribution. In ContinuumIO/anaconda-issues#9480 similar problem were discussed, but don't mind, specifying the dependencies on the command line is good enough.

[paudetseis]

Not sure if this is relevant but your gfortran executable is found in a different environment (workhorse) - you could try installing gfortran within the telewavesim environment and see if that correctly links to liblapack in telewavesim.

[trichter]

That was a good catch. Unfortunately installing gfortan correctly inside the environment does not change anything. I updated the above code block.

[trichter]

I tried to run f2py manually inside the src folder. The default is not working for me:

f2py -c *.f90 -m test
python -c 'import test'

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: /home/eule/dev/other/Telewavesim/src/test.cpython-37m-x86_64-linux-gnu.so: undefined symbol: dgeev_

Explicitly requesting to link against lapack and specifying the lib folder is working:

f2py -c *.f90 -L/home/eule/anaconda/envs/telewavesim/lib -llapack -m test
python -c 'import test'

Leaving away either '-L' or '-l' flag does not work.

The call to gfortran for the first version is

/home/eule/anaconda/envs/telewavesim/bin/x86_64-conda_cos6-linux-gnu-gfortran -Wall -g -Wall -g -shared -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections /tmp/tmppks606nv/tmp/tmppks606nv/src.linux-x86_64-3.7/testmodule.o /tmp/tmppks606nv/tmp/tmppks606nv/src.linux-x86_64-3.7/fortranobject.o /tmp/tmppks606nv/rmat.o /tmp/tmppks606nv/rmat_sub.o /tmp/tmppks606nv/tmp/tmppks606nv/src.linux-x86_64-3.7/test-f2pywrappers2.o -L/home/eule/anaconda/envs/telewavesim/bin/../x86_64-conda_cos6-linux-gnu/sysroot/lib/../lib -L/home/eule/anaconda/envs/telewavesim/bin/../x86_64-conda_cos6-linux-gnu/sysroot/lib/../lib -lgfortran -o ./test.cpython-37m-x86_64-linux-gnu.so

Interestingly, it already includes the folder telewavesim/bin/../x86_64-conda_cos6-linux-gnu/sysroot/lib twice, but inside this folder there is no liblapack.so. This file is only located in the telewavesim/lib folder.

[paudetseis]

Ok thanks. I'm not sure I understand why this is happening since the lapack lib does exist in the telewavesim env and correctly works when you link to it explicitly. I suspect this is a problem with f2py (perhaps on the linux dist of numpy?). You could work around by specifying the extra_link_args in the setup.py Extension, although it's kind of a hack:

ext = [Extension(name='telewavesim.rmat_f',
                 sources=['src/rmat.f90', 'src/rmat_sub.f90'],
                 extra_link_args=["-L/home/eule/anaconda/envs/telewavesim/lib", "-llapack"])]

[trichter]

I also expect, this is a problem with f2py and/or conda. I already tried the code you suggest, and also the f2py_options, but both produced extension modules do not correctly link against lapack. I guess I'll try to copy the file I produced with f2py.

[trichter]

It is working.

Here is how I got it installed:

cd Telewavesim
pip install -e .
cd telewavesim
f2py -c ../src/*.f90 -L/home/eule/anaconda/envs/telewavesim/lib -llapack -m rmat_f

Thanks for your help. Hopefully others will suggest a better solution.

[paudetseis]

Ok great. I might post a Known Issues rubric if others come across the same problem. Thanks for the solution!

[trichter]

This diff finally solved it:

diff --git a/setup.py b/setup.py
index aaf9979..97efe24 100644
--- a/setup.py
+++ b/setup.py
@@ -1,8 +1,8 @@
-import setuptools
 from numpy.distutils.core import setup, Extension

 ext = [Extension(name='telewavesim.rmat_f',
-                sources=['src/rmat.f90', 'src/rmat_sub.f90'])]
+                sources=['src/rmat.f90', 'src/rmat_sub.f90'],
+                libraries=['lapack'])]

Edit: Sorry I was wrong, it did not solve the issue. :unamused:

[trichter]

But this diff really solves the issue for me. Still, I don't know why it is necessary.

diff --git a/setup.py b/setup.py
index aaf9979..704f8bd 100644
--- a/setup.py
+++ b/setup.py
@@ -1,8 +1,10 @@
-import setuptools
 from numpy.distutils.core import setup, Extension
+from numpy.distutils.system_info import get_info

 ext = [Extension(name='telewavesim.rmat_f',
-                sources=['src/rmat.f90', 'src/rmat_sub.f90'])]
+                sources=['src/rmat.f90', 'src/rmat_sub.f90'],
+                library_dirs=get_info('lapack')['library_dirs'],
+                libraries=['lapack'])]

[paudetseis]

Awesome! Thanks for putting the time to find a solution. I'll direct people with similar problems to this thread.

[trichter]

Interesting, in a fresh up-to-date conda environment, I only need the libraries=['lapack'] option and not the library_dirs option and the lapack package. This makes sense. At the moment it is working as it is. For the future, I suggest to wait for the new obspy release and after that see if CI gives green light only with the libraries=['lapack'] option.