error linking against numpy libraries

[satra]

when linking against numpy/scipy libraries compiled with mkl support we run into the issue that the libraries appended to the link flag includes 'mkl_lapack95_lp64` (see below), which we don't have access to (without mkl dev being available).

this means that we either have to remove mkl support and install the package and then enable mkl support, or have to go in and hack the package's setup function to not include mkl_lapack95_lp64. since this is specific to numpy compiled with mkl support distributed by anaconda, it would be good to figure out a solution at the anaconda level.

>>> numpy.show_config()
lapack_opt_info:
    libraries = ['mkl_lapack95_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'iomp5', 'pthread']
    library_dirs = ['/om/user/satra/miniconda/lib']
    define_macros = [('SCIPY_MKL_H', None)]
    include_dirs = ['/om/user/satra/miniconda/include']
blas_opt_info:
    libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'iomp5', 'pthread']
    library_dirs = ['/om/user/satra/miniconda/lib']
    define_macros = [('SCIPY_MKL_H', None)]
    include_dirs = ['/om/user/satra/miniconda/include']
openblas_lapack_info:
  NOT AVAILABLE
lapack_mkl_info:
    libraries = ['mkl_lapack95_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'iomp5', 'pthread']
    library_dirs = ['/om/user/satra/miniconda/lib']
    define_macros = [('SCIPY_MKL_H', None)]
    include_dirs = ['/om/user/satra/miniconda/include']
blas_mkl_info:
    libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'iomp5', 'pthread']
    library_dirs = ['/om/user/satra/miniconda/lib']
    define_macros = [('SCIPY_MKL_H', None)]
    include_dirs = ['/om/user/satra/miniconda/include']
mkl_info:
    libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'iomp5', 'pthread']
    library_dirs = ['/om/user/satra/miniconda/lib']
    define_macros = [('SCIPY_MKL_H', None)]
    include_dirs = ['/om/user/satra/miniconda/include']

[WarrenWeckesser]

@satra, have you found a good solution for this?

[satra]

@WarrenWeckesser - nothing seemless.

one option is for libraries using this to do so through scipy functions. the other is as noted above to hack the build process to exclude libraries like mkl_lapack95_lp64

[WarrenWeckesser]

I ran into the problem when trying to build scipy from source. For simply building and testing scipy, I'm not concerned about performance, so my quick solution was to install the nomkl package, as explained at the end of this blog post by Ilan: https://www.continuum.io/blog/developer-blog/anaconda-25-release-now-mkl-optimizations

[WarrenWeckesser]

I finally got back to looking into this. After installing the mkl version of numpy 1.11.1 in a python 3 miniconda installation (installed in /Users/warren/miniconda3scipy), I was able to build scipy by creating the file site.cfg in the top scipy directory that contains

[mkl]
library_dirs = /Users/warren/miniconda3scipy/lib
lapack_libs = mkl_core

[rgommers]

We should add both nomkl and site.cfg solutions to the Linux and OS X parts of http://scipy.org/scipylib/building/index.html. Also, one might prefer ~/.numpy-site.cfg over site.cfg inside the package, because that gets it right for the whole stack.

[justbennet]

@WarrenWeckesser and @rgommers I have been using the following site.cfg on Linux with what seems good success.

$ cat << __EOF__ > site.cfg
[mkl]
library_dirs = ${MKL_LIB}
include_dirs = ${MKL_INCLUDE}
mkl_libs = mkl_rt
lapack_libs = 
__EOF__

The libmkl_rt.so (or .dylib on OS X) is, I believe, the runtime library stubfile, and it should contain all the symbols needed for MKL's runtime distribution along with pointers into the correct libraries. From my OS X 10.11.5 machine,

$ strings libmkl_rt.dylib | grep .dylib
libmkl_core.dylib
libmkl_tbb_thread.dylib
libmkl_sequential.dylib
libmkl_intel_thread.dylib
libmkl_intel_lp64.dylib
libmkl_intel_ilp64.dylib
libmkl_cdft_core.dylib

I hope that's germane.

[WarrenWeckesser]

Thanks Bennet and Ralf. My "solutions" should not be considered thoroughly researched and definitive. I was just tweaking my configuration enough to get scipy to build. More information and better solutions are appreciated!

[rgommers]

@justbennet do you know if MKL always set those env vars, or is it Anaconda/Miniconda specific?

[justbennet]

Ahh, Ralf, sorry about that. No, those are names we assign here to make things convenient. If you source the mklvars.[x]sh file that comes with the compilers it will set MKLROOT, and other libraries will be set from there. In this case, you would want to set MKL_INCLUDE to wherever the MKL header files are located and the MKL_LIB to wherever the .dylib files are. I am not, at the moment, at an machine with Anaconda on it, but I will look when I get home and post what I think is correct.