pytensor >= 2.17.4 doesn't detect system blas

[beew]

Describe the issue:

As the title says pytensor >=2.17.4 doesn't detect system blas

Reproducable code example:

for pytensor >= 2.17.4

python -c "import pytensor; print(pytensor.config.blas__ldflags)"
WARNING (pytensor.tensor.blas): Using NumPy C-API based implementation for BLAS functions.

pytensor 2.16.3 finds the default blas correctly

python -c "import pytensor; print(pytensor.config.blas__ldflags)"
-L/opt/intel/oneapi/mkl/latest/lib/intel64 -lmkl_rt -lpthread -lm -lm

PyTensor version information:

pytensor >= 2.17.4

Context for the issue:

pytensor is installed as a dependency for pymc

OS is Ubuntu 22.04, default blas is mkl, set by update-alternatives. python 3.10.9, numpy 1.23.5. Python is compiled from source and invoked by sourcing a script that sets the environmental variables :

export PREFIX=/home/beew/opt/python310

export HOME=$PREFIX #for spyder to create seperate config than system

export PYTHONHOME=$PREFIX

export MKL_THREADING_LAYER=GNU #required by pymc

export PATH=$PYTHONHOME/bin:$PATH

export LD_LIBRARY_PATH=$PYTHONHOME/lib:$PYTHONHOME/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH

export PKG_CONFIG_PATH=$PREFIX/lib/pkgconfig:$PKG_CONFIG_PATH

export PYTHONUSERBASE=$PYTHONHOME

export PYTHONPATH=$PYTHONHOME/lib/python3.10/site-packages

some of these might not be necessary but it works.

[digicosmos86]

I am having the same issue. This only happens when pytensor>=2.17.4. OS is MacOS Arm64.

[twiecki]

Did you install via pip or conda/mamba?

[beew]

Did you install via pip or conda/mamba?

pip

[digicosmos86]

Pip for us as well. I know that installation from PyPI is not officially supported by PyMC, but we have some use cases where some secure computing environments only allow access to private mirrors of PyPI, so packages can only be installed via pip

[twiecki]

We definitely want to support that case if you know what you're doing.

[lucianopaz]

Thanks for the report.

The reason this used to work in past versions was that numpy stored the information of the MKL or BLAS libraries with which it was built on the host system. The newer versions of numpy don't provide this information anymore and we had to resort to other means to determine some default blas flags.

The current behavior of pytensor is to try to find the libraries in the python lib dir or in the C compiler's -print-search-dirs library directories. These folders do not include /opt/intel/mkl/, which seemed to be the default installation directory of mkl some years ago. One way to go around this problem and use pytensor now is to configure the blas flags following these instructions.

That being said, I don't think that we can safely assume that future MKL installations will go into /opt/intel. The manual mentions an <mkl directory> that seems to be where mkl gets installed but it doesn't say anything about defaults.

@beew, @digicosmos86, I would like to ask you one thing though. Do your environments define the MKLROOT variable? You can check by opening a terminal and running:

echo $MKLROOT

If yes, then we could try to look for that variable to see if mkl is installed at the system level somewhere.

As a side comment, it might still be possible to find out the BLAS layer that numpy uses, but it will be a bit hard to get

[digicosmos86]

Thank you @twiecki and @lucianopaz! Looks like the MKLROOT in my environment is not set

[twiecki]

@digicosmos86 Does setting it fix it?

[digicosmos86]

@twiecki how do I set this on MacOS? There is no /opt/intel directory on this machine

I saw your post about a potential speed-up on M1/M2 macs. Is there a way to set it up so that pytensor uses Apple's accelerate library? I'm not using conda though

[beew]

@lucianopaz

At Present MKLROOT is not defined but I can easily add it to .profile.

On Ubuntu systems if install MKL from Intel's repository or downloading standalone it is at

/opt/intel/oneapi/mkl/latest

If install MKL from the standard Ubuntu repo, MKLROOT is simpler

/usr

I use these values to compile Octave from source.

However, no matter where MKL is actually installed, I think in Debian systems (definitely for Ubuntu) the default blas is at

/usr/lib/x86_64-linux-gnu/libblas.so

This is a symlink to whatever blas the user sets as default via update-alternatives (it can be mkl, openblas, atlas or the useless blas shipped by default)

So at least in this case maybe pytensor could just link to /usr/lib/x86_64-linux-gnu/libblas.so without worrying where mkl or openblas is (In some of my computers openblas is in /opt since I also compiled it from source)?

P.S I also compiled numpy against MKL, but I have only 1.23.5.

[lucianopaz]

Thanks @digicosmos86 and @beew.

Just to be clear, defining $MKLROOT wont fix the blas flags problem at the moment. The way in which you can fix the problem now is to either create a .pytensorrc on your home folder or define a specific value for the PYTENSOR_FLAGS environment variable as is described here. If you want to define the default blas flags via the .pytensorrc file, you need to create the file with the following lines:

[blas]
ldflags=-Lwherever/mkl/is -lmkl_core -lmkl_rt -lpthread -lm

The reason I was asking about the MKLROOT variable is that I'm trying to think of some way for pytensor to detect system level mkl installations automatically. Since neither of you had that variable defined, it seems like we can't expect to rely on it to detect MKL.

@beew, about your comment on Ubuntu BLAS, the default blas implementations that come with Ubuntu should be detected automatically. The way in which this works is that pytensor queries the compiler's library search dirs like this:

gcc -print-search-dirs

It reads and parses the directories under the libraries: = header and then looks for the libraries it needs there. On my Ubuntu machine, those directories are:

/usr/lib/gcc/x86_64-linux-gnu/7
/usr/x86_64-linux-gnu/lib/x86_64-linux-gnu/7
/usr/x86_64-linux-gnu/lib/x86_64-linux-gnu
/usr/x86_64-linux-gnu/lib
/usr/lib/x86_64-linux-gnu/7
/usr/lib/x86_64-linux-gnu
/usr/lib
/lib/x86_64-linux-gnu/7
/lib/x86_64-linux-gnu
/lib
/usr/lib/x86_64-linux-gnu/7
/usr/lib/x86_64-linux-gnu
/usr/lib
/usr/x86_64-linux-gnu/lib
/usr/lib
/lib
/usr/lib

which includes /usr/lib/x86_64-linux-gnu and hence blas. I'm not sure why pytensor doesn't use your libblas.so though. Debugging this remotely is a bit of a pain. @michaelosthege suggested we add many logging.debug statements to this part of pytensor so that it can ease the process of identifying why some flags fail to work and the blas flags are left empty.

[lucianopaz]

@twiecki how do I set this on MacOS? There is no /opt/intel directory on this machine

I saw your post about a potential speed-up on M1/M2 macs. Is there a way to set it up so that pytensor uses Apple's accelerate library? I'm not using conda though

Yes, it's possible to link to Apple's accelerate library. I thought that we had already tested that this worked with @twiecki? If it doesn't detect accelerate by default then we could either:

1. Try to add the Accelerate framework's library dirs to our search paths for linking. This seems possible given that the blas feedstock already handles it somehow.
2. Include -framework accelerate flag on MacOS after trying to get MKL. This would put an additional requirement on the compiler since only clang can handle the -framework argument and gcc can't, but it if it works, it would be the easiest to implement.

[maresb]

I thought that we had already tested that this worked

We just did this for conda with libblas=*=*accelerate.

[digicosmos86]

Thanks @lucianopaz and @maresb! Your instruction assumes conda though right? Is there a way to do this without conda? As mentioned there's some use cases where only pip can be used...

[lucianopaz]

Thanks @lucianopaz and @maresb! Your instruction assumes conda though right? Is there a way to do this without conda? As mentioned there's some use cases where only pip can be used...

No, what I said works for any kind of installation. It’s a way in which you can customise pytensor to deal with your specific setup. The difference with conda is that we try to support it by default and try to look for libraries in the directories it regularly uses.

[beew]

@lucianopaz

It reads and parses the directories under the libraries: = header and then looks for the libraries it needs there. On my Ubuntu machine, those directories are:

/usr/lib/gcc/x86_64-linux-gnu/7
/usr/x86_64-linux-gnu/lib/x86_64-linux-gnu/7
/usr/x86_64-linux-gnu/lib/x86_64-linux-gnu
/usr/x86_64-linux-gnu/lib
/usr/lib/x86_64-linux-gnu/7
/usr/lib/x86_64-linux-gnu
/usr/lib
/lib/x86_64-linux-gnu/7
/lib/x86_64-linux-gnu
/lib
/usr/lib/x86_64-linux-gnu/7
/usr/lib/x86_64-linux-gnu
/usr/lib
/usr/x86_64-linux-gnu/lib
/usr/lib
/lib
/usr/lib

which includes /usr/lib/x86_64-linux-gnu and hence blas. I'm not sure why pytensor doesn't use your libblas.so though. Debugging this remotely is a bit of a pain. @michaelosthege suggested we add many logging.debug statements to this part of pytensor so that it can ease the process of identifying why some flags fail to work and the blas flags are left empty.

Hi, maybe I should clarify, there are two meanings of "default blas" in this discussion.

1. libblas shipped with Ubuntu, which is single thread and no good, It is called libblas3 in Ubuntu 22.04 and it is in /usr/lib/x86_64-linux-gnu/blas/libblas.so.3 (note this is in a blas subfolder of /usr/lib/x86_64-linux-gnu/

2. If you install multiple versions of blas you can choose a new default by using update-alternatives to override the blas shipped with Ubuntu. Update-alternatives creates a symlink /usr/lib/x86_64-linux-gnu/libblas.so. The taget of the symlink is /etc/alternatives/libblas.so-x86_64-linux-gnu.

   In this case it shouldn't matter where your alternative blas is (are) if update-alternatives is set up correctly it would work because the program should just check /usr/lib/x86_64-linux-gnu/libblas.so.

   Sorry for the confusing use of "default".