search

conda-forge / openblas-feedstock

A conda-smithy repository for openblas.
BSD 3-Clause "New" or "Revised" License
9 stars 38 forks source link

NumPy dev build + OpenBLAS incompatibility #98

Closed rgommers closed 4 years ago

rgommers commented 4 years ago

It's not (or no longer) possible to build NumPy directly against OpenBLAS in a conda-forge based dev environment.

conda create -n blas
conda activate blas
conda install -c conda-forge cython numpy  # will pull in libopenblas
# cd into numpy git repo
python setup.py build_ext -i  # will not detect openblas and fail

This is a numpy.distutils / conda-forge incompatibility. The conda-forge packaging seems fine in principle, and OpenBLAS is installed as libopenblas.0.dylib. That lib isn’t picked up by the numpy.distutils detection code though; when I create a symlink right next to it with .0 removed then it does get picked up, and NumPy builds and tests fine. The normal convention is to have both a “base name” for the shared library, and a versioned library. This is the case for libblas for example:

$ ls *blas*
libblas.3.dylib         libcblas.3.dylib        libopenblas.0.dylib
libblas.dylib           libcblas.dylib          libopenblasp-r0.3.10.dylib

However it’s not for libopenblas, likely because everything in the conda-forge channel has to link to the mutex blas package instead, and letting users build directly against OpenBLAS wasn't considered.

There's two possible fixes:

  1. install a libopenblas.dylib
  2. special-case conda-forge in numpy.distutils

Given that (2) won't fix the issue for users/packages that don't use numpy.distutils, I'd prefer (1) I think. Also looking at my /usr/lib and /usr/local/lib I cannot see a single library that only provides a versioned .X.dylib library - so arguably this is an OpenBLAS packaging bug.

Also, I checked BLIS for comparison, which does have the base name:

$ ls *blis*
libblis.3.0.0.dylib libblis.3.dylib     libblis.a       libblis.dylib
$ otool -L libblis.dylib 
libblis.dylib:
    @rpath/libblis.3.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
isuruf commented 4 years ago

You have to install openblas package which gives you the library without the version and the headers.

isuruf commented 4 years ago

You can also install blas-devel which installs openblas for the openblas variant and the corresponding package for others as well.

rgommers commented 4 years ago

Thanks @isuruf. Somehow that didn't cross my mind - makes perfect sense. Sorry for the noise.