Closed AgilentGCMS closed 1 month ago
I've explicitly tried to exclude openblas
by excluding it from lapack-order
and blas-order
as follows
-Dblas-order=mkl -Dlapack-order=mkl
but it still builds and links against openblas
.
Update: I realized that I was testing a different build of numpy than I was building. However, for the life of me I can't figure out how to install into a specific folder with spin build
. It always installs into build-install
, and Google has failed me on this problem.
Not sure how to install into a specific directory, but you can use NUMPY_SITE
env variable to pick that version of numpy:
export NUMPY_SITE=$(realpath build-install/usr/lib/python*/site-packages/)
I use pip
to install a development build of numpy if I need to use it in a python environment that depends on numpy. Something like:
python -m pip install -v . --no-build-isolation -Cbuilddir=build -C'compile-args=-v' -C'setup-args=-Dbuildtype=debug'
This will build a new install of numpy
in the build
folder (same as spin
, and spin
can re-use this install) and then install it to site-packages
as normal. You also don't need to pass the compile-args
or setup-args
, see the meson-python
docs for more about passing arguments to meson
.
spin
is really only for working on numpy itself. If whatever you're testing only needs numpy, you can use e.g. spin python
or spin ipython
to directly use the version of numpy spin
builds.
@ngoldbaum Let's say I've already built numpy with spin build
. Is there a way to install it to a prefix of my choice? I could use the NUMPY_SITE
environment variable, but I'm trying to install for multiple people, so I want to avoid kludges like setting an environment variable for all users that point to my home directory.
If you are providing a package for multiple people, then you should be producing a wheel and publishing that wheel for them to install
You can pass arguments to meson from a pip
invocation. So, to pass the MKL options, you'd do:
pip install python -m pip install -v . --no-build-isolation -C'setup-args=-Dblas-order=mkl' -C'setup-args=-Dlapack-order=mkl'
If you are providing a package for multiple people, then you should be producing a wheel and publishing that wheel for them to install
You have the wrong idea. This is a high performance computing system. I want multiple users on the same machine to use this from a single location, and I don't want that location to be my home directory where I build stuff. I certainly don't want them to install wheels individually. I've built hundreds of packages on *nix systems, and most allow specification of a "prefix" in which to install binaries, libraries, headers, documentation, etc. I should be able to do this for python packages as well, or there's something broken about how python thinks packages should be managed.
And yet, at runtime it seems to be using
openblas
and not MKL.
I don't think that is possible. Once there is a found: YES
in the build log, that is what is used - no other BLAS libraries are even searched for.
This part of the show_runtime
output:
'filepath': '/home/sbasu1/packages/lib/python3.11/site-packages/numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so',
shows that the numpy
you are importing is coming from a PyPI wheel, which has openblas64_...
vendored. So the package you built against MKL probably works, but isn't installed or is shadowed by another numpy
installed into python3.11/site-packages
.
Run-time dependency mkl-dynamic-lp64-iomp found: YES 2022.1
It probably doesn't matter here, but if you still have problems after removing the other numpy
, then this MKL version may be too old. MKL is a complete mess in how it's distributed, and broken in so many ways when trying to use anything but MKL SDL in older versions (SDL was the only thing that the numpy.distutils
supported, via direct linking to mkl_rt.so
) - so I suggest using at least the version we use in CI (2023.2, see https://github.com/numpy/numpy/blob/fba4fc16a26e77ec4ce2b1900577f088b1d5a7a6/.github/workflows/linux_blas.yml#L278).
I certainly don't want them to install wheels individually. I've built hundreds of packages on *nix systems, and most allow specification of a "prefix" in which to install binaries, libraries, headers, documentation, etc. I should be able to do this for python packages as well, or there's something broken about how python thinks packages should be managed.
This is perfectly fine, that is a supported and important use case. I think though that it has to be set up in a way that users cannot install other Python packages themselves with something like the --user
flag of pip
, because that has the problem that you can get two numpy
versions in the same environment. Something like that could be happening here.
shows that the
numpy
you are importing is coming from a PyPI wheel,
Ah you figured this out already. Then to go on to the actual use case:
spin
, it is only useful when trying to contribute to numpy
. It should never be used for the kind of HPC system install you're trying to do.numpy
itself.For MKL: do you want to link to a central MKL install, either in say /usr/lib
or in a custom location like /opt/intel/
? If the latter, then please ensure that opt/intel/
is on the library search path.
If you want the numpy
install to be self-contained and vendor MKL, then you have to run auditwheel
to actually do that vendoring.
@rgommers Thanks for commenting and explaining, and I apologize for my meandering bugfixing method. As you correctly surmised, I have since figured out that my initial complain of not finding MKL at runtime was because I was loading the wrong numpy, i.e., not the numpy that had successfully compiled against MKL. This in turn was because I was expecting spin build
to install into python's "global" prefix, i.e., the folder tree where the python binary was located, and not to build-install
inside the numpy source tree. Why do/did I use spin build
? Because I find meson build frustrating when debugging build errors; it rarely tells me what actually went wrong, and it always points to a non-existent log file, like so:
A full log can be found at /tmp/.../build/meson-logs/meson-log.txt
I cannot tell you how many hours I've spent hunting down these supposedly helpful meson log files that don't exist. I've also found that pip install
takes me down that same logfile-hunting journey when a build fails, probably because it uses meson build
. That's why I use spin build
, at least I can figure out from the output what went wrong. This is not my complain alone, a Google search of "missing meson log file" yields several discussion posts such as this.
Anyway, my current situation is
/opt/intel/...
/opt/local/.../python/3.11.6
which I want multiple users to be able to use. This python is built with Intel compilers.PYTHONPATH
and loading (say) the wrong numpy
and scipy
, but it's understood that if they do that they're on their own. My responsibility is to provide a python+numpy+scipy that works system-wide and uses the MKL that's installed in /opt/intel
.export PKG_CONFIG_PATH=/opt/intel/oneapi/mkl/2022.1.0/lib/pkgconfig:$PKG_CONFIG_PATH
CC=icx CXX=icpx spin build -- -Dallow-noblas=false -Dblas-order=mkl -Dlapack-order=mkl -Dblas=mkl-static-lp64-iomp -Dlapack=mkl-static-lp64-iomp
numpy
build available to all users by putting it in /opt/local/.../python/3.11.6
, because the above step only installs into build-install
. Right now I rsync
from build-install
into the location I want. That can't be the recommended way./opt/local
, I will follow that. If it didn't include hours of frustration with meson build
(see above), that would be an added bonus.I cannot tell you how many hours I've spent hunting down these supposedly helpful meson log files that don't exist. [...] This is not my complain alone, a Google search of "missing meson log file" yields several discussion posts such as this.
This is an annoying problem indeed, but it's easy to resolve: add -Cbuild-dir=build
to your pip
invocation, and you'll find the logs under ./build/
. Because this is indeed a recurring problem for users, I just wrote more extensive docs for that: https://github.com/mesonbuild/meson-python/pull/616
Right now I
rsync
frombuild-install
into the location I want. That can't be the recommended way.
Indeed, it is not. I suggest to change your build command to (first line is unchanged):
export PKG_CONFIG_PATH=/opt/intel/oneapi/mkl/2022.1.0/lib/pkgconfig:$PKG_CONFIG_PATH
CC=icx CXX=icpx python -m pip -Cbuild-dir=build -Csetup-args=-Dallow-noblas=false -Cetup-args=-Dblas-order=mkl -Cetup-args=-Dlapack-order=mkl -Cetup-args=-Dblas=mkl-static-lp64-iomp -Csetup-args=-Dlapack=mkl-static-lp64-iomp
(I don't think the *-order
args should be needed, but I kept it unchanged from your working variant to be sure)
That should be all. In case it builds fine but doesn't work at runtime because an MKL .so
goes missing (I can't remember what the OneAPI activation scripts do exactly), then do export LDFLAGS=-Wl,-rpath=/opt/intel/path/to/libdir
.
@rgommers I tried your python -m pip
command (after correcting the obvious -Cetup-args
typos to -Csetup-args
), but now I see something confusing.
CC=icx CXX=icpx python -m pip install -v . -Cbuild-dir=build -Csetup-args=-Dallow-noblas=false -Csetup-args=-Dblas-order=mkl -Csetup-args=-Dlapack-order=mkl -Csetup-args=-Dblas=mkl-dynamic-lp64-iomp -Csetup-args=-Dlapack=mkl-dynamic-lp64-iomp
This failed with the error
Installing build dependencies ... done
Running command Getting requirements to build wheel
Getting requirements to build wheel ... done
Running command pip subprocess to install backend dependencies
Collecting ninja>=1.8.2
Using cached ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl.metadata (5.3 kB)
Collecting patchelf>=0.11.0
Using cached patchelf-0.17.2.1-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.musllinux_1_1_x86_64.whl.metadata (3.3 kB)
Using cached ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl (307 kB)
Using cached patchelf-0.17.2.1-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.musllinux_1_1_x86_64.whl (425 kB)
Installing collected packages: patchelf, ninja
Successfully installed ninja-1.11.1.1 patchelf-0.17.2.1
Installing backend dependencies ... done
Running command Preparing metadata (pyproject.toml)
('\x1b[31m',)meson-python: error: Could not find ninja version 1.8.2 or newer.
error: subprocess-exited-with-error
× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
full command: /home/sbasu1/packages/bin/python /home/sbasu1/packages/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpmlzt6l82
cwd: /home/sbasu1/Downloads/sources/numpy-1.26.4
Preparing metadata (pyproject.toml) ... error
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
However, the error about ninja
version cannot be true, because
$ $ ninja --version
1.11.1.git.kitware.jobserver-1
On the other hand, spin build
from the same source tree builds fine.
(I don't think the
*-order
args should be needed, but I kept it unchanged from your working variant to be sure)
They're probably not needed, but I want things to fail if MKL is not found instead of silently using openblas.
However, the error about
ninja
version cannot be true, because
That is very strange. I don't know what is going on there, I've never seen that before. The build log also shows ninja
being downloaded right above where it fails. Some kind of pip
bug maybe (not likely though). If spin build
works, then disabling build isolation should definitely work as well - can you add --no-build-isolation
to the pip
invocation?
They're probably not needed, but I want things to fail if MKL is not found instead of silently using openblas.
It will. Using -Dblas
says "use this exact blas" and will disable auto-detection (if not, that is a bug). I verified with:
$ pip install . -Csetup-args=-Dblas=nonsense -Csetup-args=-Dallow-noblas=false --no-build-isolation
...
Run-time dependency nonsense found: NO (tried pkgconfig, framework and cmake)
../numpy/meson.build:185:4: ERROR: Problem encountered: No BLAS library detected! Install one, or use the `allow-noblas` build option (note, this may be up to 100x slower for some linear algebra operations).
So, the --no-build-isolation
flag seemed to do the trick. After ensuring that mkl-static-lp64-iomp.pc
was in my PKG_CONFIG_PATH
, I issued the following:
CC=icx CXX=icpx python -m pip install --no-build-isolation -v . -Cbuild-dir=build -Csetup-args=-Dallow-noblas=false -Csetup-args=-Dblas-order=mkl -Csetup-args=-Dlapack-order=mkl -Csetup-args=-Dblas=mkl-static-lp64-iomp -Csetup-args=-Dlapack=mkl-static-lp64-iomp
which successfully compiled numpy and installed it in the same folder tree/prefix when my python binary is located. Yay, thanks @rgommers! I'm OK closing this ticket unless you want otherwise.
Great, thanks @AgilentGCMS. Yes, let's close it, since things work for you now.
If you have problems again with BLAS/MKL/Meson, please feel free to ping me directly on a new issue.
Describe the issue:
I recently installed
numpy 1.26.4
from source on a machine that has intel oneapi compilers (icx/icpx) and MKL libraries. I installed withand during compilation, it told me that it had found the correct shared library,
And yet, at runtime it seems to be using
openblas
and not MKL.Reproduce the code example:
Error message:
No response
Python and NumPy Versions:
Runtime Environment:
[{'numpy_version': '1.26.4', 'python': '3.11.9 (main, Apr 4 2024, 11:00:55) [Clang 14.0.0 (icx ' '2022.1.0.20220316)]', 'uname': uname_result(system='Linux', node='gs6101-trident.ndc.nasa.gov', release='4.18.0-513.24.1.el8_9.x86_64', version='#1 SMP Thu Mar 14 14:20:09 EDT 2024', machine='x86_64')}, {'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3'], 'found': ['SSSE3', 'SSE41', 'POPCNT', 'SSE42', 'AVX', 'F16C', 'FMA3', 'AVX2', 'AVX512F', 'AVX512CD', 'AVX512_SKX'], 'not_found': ['AVX512_KNL', 'AVX512_KNM', 'AVX512_CLX', 'AVX512_CNL', 'AVX512_ICL']}}, {'architecture': 'SkylakeX', 'filepath': '/home/sbasu1/packages/lib/python3.11/site-packages/numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so', 'internal_api': 'openblas', 'num_threads': 8, 'prefix': 'libopenblas', 'threading_layer': 'pthreads', 'user_api': 'blas', 'version': '0.3.23.dev'}]
Context for the issue:
No response