ImportError: undefined symbol: cblas_daxpy

[Fhechene]

Hi! I've managed to install HORTON, but I have an issue. When I try to import it to python I get the following error ImportError: /usr/local/lib/python2.7/dist-packages/horton/gbasis/cext.so: undefined symbol: cblas_daxpy

Any idea why could this be?

[tovrstra]

There could be several reasons, depending on the operating system you are using and other factors. Which OS are you working on? Which steps did you follow to install HORTON, i.e. which commands did you use on the command line, including options given to ./setup.py?

In essence, the problem is that the file /usr/local/lib/python2.7/dist-packages/horton/gbasis/cext.so was dynamically linked against a BLAS library at compile time but that, at the time you import horton, the BLAS library can no longer be found. Most of the workarounds are explained here: http://theochem.github.io/horton/2.1.0b1/user_download_and_install_setup_cfg.html

[Fhechene]

I have Linux Mint 64 It seems that I screwed up even more. I installed horton a few weeks ago, so I can't remember what exactly did I do. Because of that, I ran again the setup as ./setup.py install --user. Then, I remembered I had to edit some files, to correct the libint2 path and finally, after a few minutes, I imported horton and got a No module named cext error.

[tovrstra]

Oops. Linux Mint is similar to Ubuntu, so the Ubuntu instructions should work reasonably well, not sure which Ubuntu version though. The file to be edited is setup.cfg. There are a few examples in data/setup_cfgs to get you started. If you run one of the distributions in that directory, the right file is picked up automatically but we don't cover all distributions.

[Fhechene]

But now cext isn't even recognized. Anyway, LM uses the Ubuntu core, so there should be no problem, but one never knows...

[kimt33]

@Fhechene Could you try to install it again and post the first few messages that pop up when you install it? It should look something like the following

MACHINE=Linux-Fedora-25-x86_64
LIBXC Configuration
  From setup.cfg
      -
  From environment variables
      -
  Static lib config in ${QAWORKDIR}
      extra_objects: ['qaworkdir/cached/libxc-2.2.2/lib/libxc.a']
      include_dirs: ['qaworkdir/cached/libxc-2.2.2/include']
  Final
      extra_objects: ['qaworkdir/cached/libxc-2.2.2/lib/libxc.a']
      include_dirs: ['qaworkdir/cached/libxc-2.2.2/include']
LIBINT2 Configuration
  From setup.cfg
      include_dirs: ['/usr/include/libint2']
      libraries: ['int2']
  From environment variables
      -
  Final
      include_dirs: ['/usr/include/libint2']
      libraries: ['int2']
BLAS Configuration
  From setup.cfg
      include_dirs: ['/usr/include/openblas/']
      libraries: ['openblas']
      library_dirs: ['/lib64/']
  From environment variables
      -
  Final
      include_dirs: ['/usr/include/openblas/']
      libraries: ['openblas']
      library_dirs: ['/lib64/']
BLAS precompiler directive: -DBLAS_OPENBLAS
Version of dependencies:
            breathe: -- not found --
            coverage: /bin/sh: coverage: command not found
            cppcheck: -- not found --
                curl: curl 7.50.3 (x86_64-redhat-linux-gnu) libcurl/7.50.3 NSS/3.26 zlib/1.2.8 libidn/1.33 libpsl/0.14.0 (+libidn/1.33) libssh2/1.8.0 nghttp2/1.13.0
              cython: Cython version 0.25.2
            docutils: -- not found --
            doxygen: -- not found --
                gcc: gcc (GCC) 6.2.1 20160916 (Red Hat 6.2.1-2)
                git: git version 2.9.3
          gitpython: -- not found --
                h5py: 2.7.0
          matplotlib: 2.0.0
                nose: nosetests version 1.3.7
              numpy: 1.12.1
                pip: pip 9.0.1 from /usr/lib/python2.7/site-packages (python 2.7)
        pycodestyle: -- not found --
          pydocstyle: -- not found --
              pylint: 
              python: Python 2.7.13
              scipy: 0.19.0
              sphinx: -- not found --
    sphinx_rtd_theme: -- not found --
              sympy: 1.0

[Fhechene]

I did create a LinuxMint-18-x86_64 cfg file. Copied the ubuntu 16.04 86_64 file and changed the name. MACHINE=Linux-LinuxMint-18-x86_64 LIBXC Configuration File setup.cfg not found. Skipping. From environment variables

Static lib not found in ${QAWORKDIR} extra_objects: ['qaworkdir/cached/libxc-2.2.2/lib/libxc.a'] include_dirs: ['qaworkdir/cached/libxc-2.2.2/include'] pkg-config failed. File data/setup_cfgs/setup.Linux-LinuxMint-18-x86_64.cfg not found. Skipping. Last resort fallback plan libraries: ['xc'] Final libraries: ['xc'] LIBINT2 Configuration File setup.cfg not found. Skipping. From environment variables

Static lib not found in ${QAWORKDIR} extra_objects: ['qaworkdir/cached/libint-2.0.3/lib/libint2.a'] include_dirs: ['qaworkdir/cached/libint-2.0.3/include/libint2'] pkg-config failed. File data/setup_cfgs/setup.Linux-LinuxMint-18-x86_64.cfg not found. Skipping. Last resort fallback plan libraries: ['int2'] Final libraries: ['int2'] BLAS Configuration File setup.cfg not found. Skipping. From environment variables

No static config available for this library pkg-config failed. File data/setup_cfgs/setup.Linux-LinuxMint-18-x86_64.cfg not found. Skipping. Last resort fallback plan libraries: ['atlas'] Final libraries: ['atlas'] BLAS precompiler directive: -DBLAS_ATLAS Version of dependencies: breathe: -- not found -- coverage: /bin/sh: 1: coverage: not found cppcheck: -- not found -- curl: curl 7.47.0 (x86_64-pc-linux-gnu) libcurl/7.47.0 GnuTLS/3.4.10 zlib/1.2.8 libidn/1.32 librtmp/2.3 cython: Cython version 0.23.4 docutils: -- not found -- doxygen: -- not found -- gcc: gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609 git: git version 2.7.4 gitpython: -- not found -- h5py: 2.6.0 matplotlib: 1.5.1 nose: nosetests version 1.3.7 numpy: 1.11.0 pip: pip 9.0.1 from /usr/local/lib/python2.7/dist-packages (python 2.7) pycodestyle: -- not found -- pydocstyle: -- not found -- pylint: python: Python 2.7.12 scipy: 0.17.0 sphinx: -- not found -- sphinx_rtd_theme: -- not found -- sympy: 0.7.6.1

[kimt33]

@Fhechene I'm not too sure about Mint Linux, but atlas normally has its own include directory (especially if it doesn't ship with the OS). Could you double check that you have the header files for atlas somewhere? It should be somewhere like /usr/include/atlas. Once you find it, add it to the blas section of your cfg file (e.g. include_dirs = /usr/include/atlas). You might also need a fresh installation (delete ~/.local/lib/python2.7/site-packages/horton, ~/.local/share/horton, ~/.local/bin/horton* or something like this), just in case.

[Fhechene]

Managed to install again HORTON. Turns out the cfg file had a cfg.cfg file. I guess I should be more careful. Any way, I checked the atlas headers folder. It's filled with .h files, so that should be all right. Added it to the blas section in the cfg file.

Now again, I arrived to ImportError: /usr/local/lib/python2.7/dist-packages/horton/gbasis/cext.so: undefined symbol: cblas_daxpy

Edit: I found a CBLAS installation folder, which I downloaded a few years ago. Tried to install it. Installed HORTON again. Returned to No module named cext error.

[kimt33]

Could you double check that when you install HORTON, the configuration points to the appropriate directories: libxc should have include_dirs = qaworkdir/cached/libxc-2.2.2/include, libraries = xc; libint should have include_dirs = /usr/include/libint2 or include_dirs = /qaworkdir/cached/libint-2.0.3/include/libint2 and libraries = int2; and blas should have include_dirs = atlas_directory, library_dirs = library_directory and libraries = atlas. You should double check that your blas include directory contains a file called cblas.h (what your horton cannot seem to find). You can't just reference a cblas library b/c you need the rest of the blas library (I think) to build horton.

If this fails, you can always use some other blas package (Atlas seems to give a lot of headache to non-Fedora users), for example, OpenBlas. You need to find OpenBlas, e.g. ldconfig -p | grep openblas (or some other method of finding libraries), and link it up like Atlas. See http://theochem.github.io/horton/2.1.0b1/user_download_and_install_setup_cfg.html for more details.

If that also fails, you can use conda to install horton. First, download and install conda for Python2.7 (https://conda.io/docs/get-started.html). Create an environment for horton and install it, conda create --name horton-env -c pyqc horton. To activate the environment, source activate horton-env.

[Fhechene]

I had to recruit my computing engineer cousin. We managed to pinpont the error to having two versions of numpy installed: one from anaconda and the other from pip. Now, I have the following error undefined symbol: PyFPE_jbuf No matter if I install or uninstall numpy or python or cython, nothing happens. I keep getting that error.

[Fhechene]

Trying to install the different packages made my Linux distro to not start. Had to format my notebook. I've downloaded again the horton-2.0.1.tar.gz file. I've found that there is no /tools/qa/install_libxc-2.2.2.sh file. I couldn't find libint.sh either, so I downloaded the horton-master. Let's see what happens. Edit 1: used those both files, but when I tried to install horton from the master folder, couldn't find some .h files. Tried to instal from the 2.0.1 folder and there is no json file. Edit 2: also, there are many erros like warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++

[kimt33]

When you are installing things, you should only give sudo to your main package manager (apt-get in your case I think). Otherwise, you really should install things locally (for pip, pip install --user package) or within virtual environments (this is quite convenient with conda). If something goes wrong, you don't have to reinstall everything and you can just delete the directory (~/.local/lib/python2.7/site-packages or your virtual environment). Also, do not to give conda sudo, because it wreaks havoc on all your paths and whatnot.

If you suspect that you have the same module installed multiple times and you don't know which you are using, enter python interpreter, import module and module.__file__.

@matt-chan @tovrstra There isn't install_libxc or install_libint in the link provided in the horton 2.0.1 documentation (https://github.com/theochem/horton/releases/download/2.0.1/horton-2.0.1.tar.gz). It is also missing the dependencies.json.

To download and install libint,

mkdir qaworkdir
cd qaworkdir
curl -OL 'http://downloads.sourceforge.net/project/libint/libint-for-mpqc/libint-2.0.3-stable.tgz'
tar -xzf libint-2.0.3-stable.tgz
cd libint-2.0.3-stable
CONFIG_SHELL=$(which sh) CFLAGS='-fPIC' CPPFLAGS='-fPIC' ./configure --with-cxx-optflags='-O1' --prefix=../cached/libint-2.0.3-stable/
make install

To download and install libxc,

mkdir -p qaworkdir
cd qaworkdir
curl -OL 'http://www.tddft.org/programs/octopus/down.php?file=libxc/libxc-2.2.2.tar.gz'
tar -xzf libxc-2.2.2.tar.gz
cd libxc-2.2.2
CFLAGS='-fPIC' CPPFLAGS='-fPIC' FCCPP=-ffreestanding ./configure --prefix=../cached/libxc-2.2.2/
make install

[kimt33]

@Fhechene It seems that there are some files missing in the https://github.com/theochem/horton/releases/download/2.0.1/horton-2.0.1.tar.gz, so for now just work off of horton master (https://github.com/theochem/horton/archive/master.zip). If you are missing header files, you need to check that your setup.cfg is set up properly, CPATH variable is properly set, and that the dependencies are installed. Normally, these things are set up right away, but you can get some crazy behaviours with multiple package managers (and other stuff). Have you tried installing horton using conda? I'll try to use Mint to see if I can reproduce your errors at some point.

[Fhechene]

I managed to find an old horton-2.0.1 tar.gz, which has all of the above files (.h's, dependencies.json, libxc and libint install files) and didn't give me warnings about c++. Now, I'm back to step one: ImportError: No module named cext

Edit: I tried to use the master folder, but it gave lot's of .h missing files and warning: command line option ‘stuffs’ is valid for C/ObjC but not for C++ warnings, which the file I've just found doesn't give me.

[tczorro]

This problem import error problem happened to centOS before because the new version of Atlas(3.10.1) on CentOS does not contain libcblas.so.3 while the old one(3.8.4) does. Here are some old tickets https://github.com/jovigb/nwatechsummit-2015/issues/1 https://github.com/fommil/netlib-java/issues/66

I also met this problem when I tried to install HORTON on my raspberry pi(Raspbian). I replaced atlas with openblas and the problem solved since I couldn't find the old version.

To double check, @Fhechene could you run ldconfig -p | grep blas and paste the output here

[Fhechene]

@tczorro So, after make, what do I do?

[tczorro]

@Fhechene Could you please do ldconfig -p | grep blas and paste the result. I am curious about does mint have the same problem with centOS

[Fhechene]

libf77blas.so.3 (libc6,x86-64) => /usr/lib/libf77blas.so.3 libf77blas.so (libc6,x86-64) => /usr/lib/libf77blas.so libcblas.so.3 (libc6,x86-64) => /usr/lib/libcblas.so.3 libcblas.so (libc6,x86-64) => /usr/lib/libcblas.so libblas.so.3 (libc6,x86-64) => /usr/lib/libblas.so.3 libblas.so (libc6,x86-64) => /usr/lib/libblas.so By the way, I installed OpenBLAS with make PATH=/stuff and then changed the LinuxMint Config file. As I executed setup.py, I got `LAS Configuration File setup.cfg not found. Skipping. From environment variables

No static config available for this library pkg-config failed. From data/setup_cfgs/setup.Linux-LinuxMint-18.1-x86_64.cfg include_dirs: ['/home/federico/OpenBLAS-0.2.19'] libraries: ['blas'] Final include_dirs: ['/home/federico/OpenBLAS-0.2.19'] libraries: ['blas'] Unknown BLAS implementation. Assuming Netlib-compatible headers. `

[tczorro]

@Fhechene Ah OK, you have installed openblas already. libcblas.so.3 is there. That's is good. Could you change the blas in your config into openblas and run it again

[Fhechene]

Done. No luck.

[tczorro]

@Fhechene If it works, could you run the nosetests, just make sure anything function properly.

[Fhechene]

`Failure: ImportError (/home/federico/.local/lib/python2.7/site-packages/horton/gbasis/cext.so: undefined symbol: cblas_daxpy) ... ERROR

====================================================================== ERROR: Failure: ImportError (/home/federico/.local/lib/python2.7/site-packages/horton/gbasis/cext.so: undefined symbol: cblas_daxpy)

Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/nose/loader.py", line 407, in loadTestsFromName module = resolve_name(addr.module) File "/usr/lib/python2.7/dist-packages/nose/util.py", line 312, in resolve_name module = import('.'.join(parts_copy)) File "/home/federico/.local/lib/python2.7/site-packages/horton/init.py", line 33, in from horton.part import File "/home/federico/.local/lib/python2.7/site-packages/horton/part/init.py", line 31, in from horton.part.mulliken import File "/home/federico/.local/lib/python2.7/site-packages/horton/part/mulliken.py", line 26, in from horton.gbasis.cext import get_shell_nbasis File "/home/federico/.local/lib/python2.7/site-packages/horton/gbasis/init.py", line 24, in from horton.gbasis.cext import * ImportError: /home/federico/.local/lib/python2.7/site-packages/horton/gbasis/cext.so: undefined symbol: cblas_daxpy

---

Ran 1 test in 0.001s

FAILED (errors=1)` I suspect that cython has something to do with it, but I'm not sure.

[tczorro]

@Fhechene Hmm, not sure why this keep happening. David and I will try to solve it. Some suggestion might help, clean/remove installed openblas or atlas use apt-get install libopenblas-dev change config file libraries=openblas then run setup.py by the way, try build_ext -i other than install

[Fhechene]

With build_ext -i I get } /horton/gbasis/libint2.h:31:50: fatal error: /usr/include/libint2/intrinsic_types.h: File or folder doesn't exist compilation terminated. error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 besides a bunch of warnings like warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++

I installed libopenblas-dev, and nothing

[tczorro]

Execute /tools/qa/install_libxc-2.2.2.sh and /tools/qa/install_libint-2.0.3.sh Forget about build_ext. Just use install.

[Fhechene]

Input: ./tools/qa/install_libxc-2.2.2.sh Output: Using Cached libxc-2.2.2

Input: ./tools/qa/install_libint-2.0.3.sh Output: Using Cached libint-2.0.3

After building and installing again, I still get undefined symbol: cblas_daxpy

[tczorro]

@Fhechene change config file libraries=openblas and run .setup install again. If failed, please paste the configuration info at the beginning of installation.

[Fhechene]

MACHINE=Linux-LinuxMint-18.1-x86_64 LIBXC Configuration File setup.cfg not found. Skipping. From environment variables

Static lib config in ${QAWORKDIR} extra_objects: ['qaworkdir/cached/libxc-2.2.2/lib/libxc.a'] include_dirs: ['qaworkdir/cached/libxc-2.2.2/include'] Final extra_objects: ['qaworkdir/cached/libxc-2.2.2/lib/libxc.a'] include_dirs: ['qaworkdir/cached/libxc-2.2.2/include'] LIBINT2 Configuration File setup.cfg not found. Skipping. From environment variables

Static lib config in ${QAWORKDIR} extra_objects: ['qaworkdir/cached/libint-2.0.3/lib/libint2.a'] include_dirs: ['qaworkdir/cached/libint-2.0.3/include/libint2'] Final extra_objects: ['qaworkdir/cached/libint-2.0.3/lib/libint2.a'] include_dirs: ['qaworkdir/cached/libint-2.0.3/include/libint2'] BLAS Configuration File setup.cfg not found. Skipping. From environment variables

No static config available for this library pkg-config failed. From data/setup_cfgs/setup.Linux-LinuxMint-18.1-x86_64.cfg include_dirs: ['/home/federico/OpenBLAS-0.2.19'] libraries: ['openblas'] Final include_dirs: ['/home/federico/OpenBLAS-0.2.19'] libraries: ['openblas'] BLAS precompiler directive: -DBLAS_OPENBLAS /usr/lib/python2.7/distutils/extension.py:133: UserWarning: Unknown Extension options: 'cython_directives' warnings.warn(msg) running install

[tczorro]

@Fhechene Could you please change include_dirs: ['/home/federico/OpenBLAS-0.2.19'] from your cfg. to /usr/lib/openblas-baseand then, run ./setup.py install again. If it install properly, open a new tab and run python, then import horton

[Fhechene]

@tczorro I changed it and still get the same error. Now, I'll add something that may be of use. If I execute python setup.py build or ./setup.py build I get the same messages from the terminal, BUT, there is a difference between python setup.py install and ./setup.py install. The later gives me the following error error: [Errno 1] Operation not permitted: '/usr/local/bin/horton-esp-test.py'

[tovrstra]

I'm going to close this issue because the problem is user configuration of dynamic link libraries, not a specific problem in HORTON. Some of the problems mentioned above have been solved in the development branch and are being backported to 2.0.1. See #232