Open sogoogos opened 4 years ago
stuartarchibald
commented
4 years ago Thanks for the report. I don't think there's a way to obtain this via pip, will add as a feature request. The package is available for download here: https://anaconda.org/numba/icc_rt
stuartarchibald
commented
4 years ago Apologies, I've interpreted this incorrectly! There's obviously icc_rt from Intel Corp https://pypi.org/project/icc-rt/ accessible via pip. I cannot reproduce the above:
$ conda create -n _tmp_issue4713 python=3.7
Collecting package metadata (current_repodata.json): done
Solving environment: done
==> WARNING: A newer version of conda exists. <==
current version: 4.7.10
latest version: 4.7.12
Please update conda by running
$ conda update -n base -c defaults conda
## Package Plan ##
environment location: <anaconda3path>/envs/_tmp_issue4713
added / updated specs:
- python=3.7
The following NEW packages will be INSTALLED:
_libgcc_mutex pkgs/main/linux-64::_libgcc_mutex-0.1-main
ca-certificates pkgs/main/linux-64::ca-certificates-2019.8.28-0
certifi pkgs/main/linux-64::certifi-2019.9.11-py37_0
libedit pkgs/main/linux-64::libedit-3.1.20181209-hc058e9b_0
libffi pkgs/main/linux-64::libffi-3.2.1-hd88cf55_4
libgcc-ng pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0
libstdcxx-ng pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0
ncurses pkgs/main/linux-64::ncurses-6.1-he6710b0_1
openssl pkgs/main/linux-64::openssl-1.1.1d-h7b6447c_2
pip pkgs/main/linux-64::pip-19.2.3-py37_0
python pkgs/main/linux-64::python-3.7.4-h265db76_1
readline pkgs/main/linux-64::readline-7.0-h7b6447c_5
setuptools pkgs/main/linux-64::setuptools-41.4.0-py37_0
sqlite pkgs/main/linux-64::sqlite-3.30.0-h7b6447c_0
tk pkgs/main/linux-64::tk-8.6.8-hbc83047_0
wheel pkgs/main/linux-64::wheel-0.33.6-py37_0
xz pkgs/main/linux-64::xz-5.2.4-h14c3975_4
zlib pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3
Proceed ([y]/n)? y
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
# $ conda activate _tmp_issue4713
#
# To deactivate an active environment, use
#
# $ conda deactivate
$ conda activate _tmp_issue4713
$ pip install numba icc_rt
Collecting numba
Using cached https://files.pythonhosted.org/packages/57/66/7ebc88e87b4ddf9b1c204e24d467cb0d13a7a890ecb4f83d20949f768929/numba-0.46.0-cp37-cp37m-manylinux1_x86_64.whl
Collecting icc_rt
Using cached https://files.pythonhosted.org/packages/1e/29/0ee89498c2b3e96620add5f4730d4e55e685b292a47aa67f52975fed9f71/icc_rt-2019.0-py2.py3-none-manylinux1_x86_64.whl
Collecting numpy (from numba)
Using cached https://files.pythonhosted.org/packages/ba/e0/46e2f0540370f2661b044647fa447fef2ecbcc8f7cdb4329ca2feb03fb23/numpy-1.17.2-cp37-cp37m-manylinux1_x86_64.whl
Collecting llvmlite>=0.30.0dev0 (from numba)
Using cached https://files.pythonhosted.org/packages/1f/3e/642ffb29ed35ca5e93f171ba327452bdee81ec76f2d711ef0f15b411928a/llvmlite-0.30.0-cp37-cp37m-manylinux1_x86_64.whl
Collecting intel-openmp (from icc_rt)
Using cached https://files.pythonhosted.org/packages/ea/f0/b1bc380d90241ab37149ed526a2ae585f6780ba616bd670b57f0559be37b/intel_openmp-2019.0-py2.py3-none-manylinux1_x86_64.whl
Installing collected packages: numpy, llvmlite, numba, intel-openmp, icc-rt
Successfully installed icc-rt-2019.0 intel-openmp-2019.0 llvmlite-0.30.0 numba-0.46.0 numpy-1.17.2
$ numba -s
System info:
--------------------------------------------------------------------------------
__Time Stamp__
2019-10-16 08:26:57.263046
<snip>
__SVML Information__
SVML state, config.USING_SVML : True
SVML library found and loaded : True
llvmlite using SVML patched LLVM : True
SVML operational : True
<snip>
__Current Conda Env__
_libgcc_mutex 0.1 main
ca-certificates 2019.8.28 0
certifi 2019.9.11 py37_0
icc-rt 2019.0 pypi_0 pypi
intel-openmp 2019.0 pypi_0 pypi
libedit 3.1.20181209 hc058e9b_0
libffi 3.2.1 hd88cf55_4
libgcc-ng 9.1.0 hdf63c60_0
libstdcxx-ng 9.1.0 hdf63c60_0
llvmlite 0.30.0 pypi_0 pypi
ncurses 6.1 he6710b0_1
numba 0.46.0 pypi_0 pypi
numpy 1.17.2 pypi_0 pypi
openssl 1.1.1d h7b6447c_2
pip 19.2.3 py37_0
python 3.7.4 h265db76_1
readline 7.0 h7b6447c_5
setuptools 41.4.0 py37_0
sqlite 3.30.0 h7b6447c_0
tk 8.6.8 hbc83047_0
wheel 0.33.6 py37_0
xz 5.2.4 h14c3975_4
zlib 1.2.11 h7b6447c_3
--------------------------------------------------------------------------------@sogoogos did you manage to find a way to reproduce this as I'm afraid I cannot? Thanks
sogoogos
commented
4 years ago Sorry about late reply! I actually still cannot configure it properly.
$ pip install numba icc_rt
Requirement already satisfied: numba in /home/sogo/work/python-virtualenvs/rmine-py3/lib/python3.6/site-packages
Requirement already satisfied: icc_rt in /home/sogo/work/python-virtualenvs/rmine-py3/lib/python3.6/site-packages
Requirement already satisfied: llvmlite>=0.30.0dev0 in /home/sogo/work/python-virtualenvs/rmine-py3/lib/python3.6/site-packages (from numba)
Requirement already satisfied: numpy in /home/sogo/work/python-virtualenvs/rmine-py3/lib/python3.6/site-packages (from numba)
Requirement already satisfied: intel-openmp in /home/sogo/work/python-virtualenvs/rmine-py3/lib/python3.6/site-packages (from icc_rt)
$ numba -s
/home/sogo/work/python-virtualenvs/rmine-py3/lib/python3.6/site-packages/numba/errors.py:137: UserWarning: Insufficiently recent colorama version found. Numba requires colorama >= 0.3.9
warnings.warn(msg)
System info:
--------------------------------------------------------------------------------
__Time Stamp__
2019-10-22 15:06:01.286165
__Hardware Information__
Machine : x86_64
CPU Name : skylake
Number of accessible CPU cores : 12
Listed accessible CPUs cores : 0-11
CFS restrictions : None
CPU Features :
64bit adx aes avx avx2 bmi bmi2 clflushopt cmov cx16 f16c fma fsgsbase invpcid
lzcnt mmx movbe pclmul popcnt prfchw rdrnd rdseed sahf sgx sse sse2 sse3 sse4.1
sse4.2 ssse3 xsave xsavec xsaveopt xsaves
__OS Information__
Platform : Linux-5.0.0-31-generic-x86_64-with-Ubuntu-18.04-bionic
Release : 5.0.0-31-generic
System Name : Linux
Version : #33~18.04.1-Ubuntu SMP Tue Oct 1 10:20:39 UTC 2019
OS specific info : Ubuntu18.04bionic
glibc info : glibc 2.25
__Python Information__
Python Compiler : GCC 8.3.0
Python Implementation : CPython
Python Version : 3.6.8
Python Locale : en_US UTF-8
__LLVM information__
LLVM version : 8.0.0
__CUDA Information__
Found 1 CUDA devices
id 0 b'GeForce GTX 1050 Ti with Max-Q Design' [SUPPORTED]
compute capability: 6.1
pci device id: 0
pci bus id: 1
Summary:
1/1 devices are supported
CUDA driver version : 10010
CUDA libraries:
Finding cublas from System
ERROR: can't locate lib
Finding cusparse from System
named libcusparse.so.10.3.0.243
Finding cufft from System
named libcufft.so.10.1.1.243
Finding curand from System
named libcurand.so.10.1.1.243
Finding nvvm from System
named libnvvm.so.3.3.0
Finding libdevice from System
searching for compute_20... ok
searching for compute_30... ok
searching for compute_35... ok
searching for compute_50... ok
__ROC Information__
ROC available : False
Error initialising ROC due to : No ROC toolchains found.
No HSA Agents found, encountered exception when searching:
Error at driver init:
NUMBA_HSA_DRIVER /opt/rocm/lib/libhsa-runtime64.so is not a valid file path. Note it must be a filepath of the .so/.dll/.dylib or the driver:
__SVML Information__
SVML state, config.USING_SVML : False
SVML library found and loaded : False
llvmlite using SVML patched LLVM : True
SVML operational : False
__Threading Layer Information__
TBB Threading layer available : True
OpenMP Threading layer available : False
+--> Disabled due to : Unknown import problem.
Workqueue Threading layer available : True
__Numba Environment Variable Information__
None set.
__Conda Information__
ERROR: The install method you used for conda--probably either `pip install conda`
or `easy_install conda`--is not compatible with using conda as an application.
If your intention is to install conda as a standalone application, currently
supported install methods include the Anaconda installer and the miniconda
installer. You can download the miniconda installer from
https://conda.io/miniconda.html.
Conda not present/not working.
Error was Command '['conda', 'info', '--json']' returned non-zero exit status 1.
--------------------------------------------------------------------------------
If requested, please copy and paste the information between
the dashed (----) lines, or from a given specific section as
appropriate.
=============================================================
IMPORTANT: Please ensure that you are happy with sharing the
contents of the information present, any information that you
wish to keep private you should remove before sharing.
=============================================================
seibert
commented
4 years ago The one difference I can see here is that while @stuartarchibald is installing icc_rt and numba with pip, he's still doing it inside a conda environment, instead of virtualenv. (They aren't that different in disk layout, but set different environment variables.) Maybe our SVML environment detection code is accidentally making a conda-specific assumptions?
stuartarchibald
commented
4 years ago This indeed might be a path thing. If you do (on the command line, in your virtual env with numba+icc_rt in it):
python -c 'from llvmlite import binding; binding.load_library_permanently("libsvml.so")'does it raise an exception?
sogoogos
commented
4 years ago Yes, it gave the following.
$ python -c 'from llvmlite import binding; binding.load_library_permanently("libsvml.so")'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/sogo/work/python-virtualenvs/rmine-py3/lib/python3.6/site-packages/llvmlite/binding/dylib.py", line 30, in load_library_permanently
raise RuntimeError(str(outerr))
RuntimeError: libsvml.so: cannot open shared object file: No such file or directory@sogoogos thanks, I guess the next question is... where does pip put that file and where is it in relation to something known, like python or site-packages?
sogoogos
commented
4 years ago Looks like libsvml.so is in lib
find . -name 'libsvml*'
./work/python-virtualenvs/rmine-py3/lib/libsvml.so
jampekka
commented
4 years ago I got the same problem on Debian Sid, and it turned out the problem was that the pip doesn't run ldconfig. After installing icc_rt globally (sudo pip3 install icc_rt), loading the libsvml.so fails with the always confusing "No such file or directory", but this is because the linker can't find the libintlc.so.5 required by libsvml.so even though it is in /usr/local/lib
TD:DR: sudo pip3 install icc_rt; sudo ldconfig
Don't know how this applies to virtualenvs or per-user installations.
esc
commented
4 years ago @jampekka thanks for posting this workaround, from what you are suggesting, this might be an issue with the icc_rt wheel.
MyraBaba
commented
4 years ago Hi , @stuartarchibald is there any solution for this issue ? I saw the mentioned .so in the .venv lib dir , but still not loading both binding or numba
esc
commented
4 years ago @MyraBaba it's a long shot, but as a temporary workaround, you could maybe use LD_LIBRARY_PATH and point it at the directory that contains the .so file.
MyraBaba
commented
4 years ago i install library without venv and now my venv numba says there is svml.
sudo pip3 install icc_rt; sudo ldconfig
but :
parallel=True , nopython=True , fastmath=True
is slover then the :
parallel=True , nopython=True
almost %25.
np.dot . and np.sum is the math func in the numba.!
esc
commented
4 years ago @MyraBaba I think the problem you are experiencing is not related to being able to install icc_rt libraries. I would suggest opening a new issue detailing the code you are running, what you expect to happen and what happens instead.
JSKenyon
commented
4 years ago Having run into the same problem, I can confirm that @esc's suggestion involving LD_LIBRARY_PATH works. I did:
export LD_LIBRARY_PATH=/path/to/venv/lib
slayoo
commented
4 years ago FWIW, here's a Travis-CI job with Linux, OSX and Windows runs of the following script:
jobs:
include:
- os: linux
language: python
python: "3.7"
script: python$PY_SFX -c 'from llvmlite import binding; binding.load_library_permanently("libsvml.so")'
- os: osx
language: shell
before_install: export PY_SFX=3
script: python$PY_SFX -c 'from llvmlite import binding; binding.load_library_permanently("libsvml.dylib")'
- os: windows
language: shell
before_install: choco install python --version 3.7
env: PATH=/c/Python37:/c/Python37/Scripts:$PATH
script: python$PY_SFX -c 'from llvmlite import binding; binding.load_library_permanently("libsvml.dll")'
install:
- pip$PY_SFX install numba
- numba -s
- pip$PY_SFX install icc-rt
- numba -shttps://travis-ci.org/github/slayoo/numba_issue_4713/builds/691936123
It shows that only on OSX the pip install icc-rt changes SVML operational from False to True. On Linux and Windows it remains False and the binding.load_library_permanently... fails.
Endor-8
commented
4 years ago I've been able to work around this issue in my Windows installation by manually patching numba's __init__.py like this:
# ...
def _try_enable_svml():
"""..."""
if not config.DISABLE_INTEL_SVML:
try:
if sys.platform.startswith('linux'):
llvmlite.binding.load_library_permanently("libsvml.so")
elif sys.platform.startswith('darwin'):
llvmlite.binding.load_library_permanently("libsvml.dylib")
elif sys.platform.startswith('win'):
# llvmlite.binding.load_library_permanently("svml_dispmd")
# Patch for the line above --------------------------------
import os.path
llvmlite.binding.load_library_permanently(os.path.join(
sys.prefix, "Library", "bin", "svml_dispmd"))
# ---------------------------------------------------------
else:
return False
# ...It's probably not the best workaround, but I hope it can help others with the same issue or give some light into the source of the problem. I can't test the same kind of patch on Linux, but I think it should work (using the correct path).
After applying this change, python -m numba -s | findstr SVML outputs:
__SVML Information__
SVML State, config.USING_SVML : True
SVML Library Loaded : True
llvmlite Using SVML Patched LLVM : True
SVML Operational : TrueAnd I have confirmed there is a significant speed improvement on compiled functions.
@esc's suggestion involving LD_LIBRARY_PATH seems to work only on Linux
esc
commented
4 years ago @sogoogos yes, probably something else is needed on Windows
esc
commented
4 years ago The Numba version 0.51.2 has been released recently and I was wondering if this may have fixed this issue? Is anybody still experiencing this today?
arthurprat
commented
4 years ago @esc I had the same issue, but on OS X, and with Numba 0.51.2. I made an ugly fixed inspired by @Endor-8 's idea: I changed Numba's __init__.py like this:
def _try_enable_svml():
"""
Tries to enable SVML if configuration permits use and the library is found.
"""
if not config.DISABLE_INTEL_SVML:
try:
if sys.platform.startswith('linux'):
llvmlite.binding.load_library_permanently("libsvml.so")
elif sys.platform.startswith('darwin'):
### CHANGED THIS ###
#llvmlite.binding.load_library_permanently("libsvml.dylib")
llvmlite.binding.load_library_permanently("/path/to/env3/lib/libintlc.dylib")
llvmlite.binding.load_library_permanently("/path/to/env3/lib/libsvml.dylib")
elif sys.platform.startswith('win'): # etcNote that I had to do it for both libintlc and libsvml.
In addition, while trying to fix things I ran sudo update_dyld_shared_cache, which I think is supposed to be the equivalent of ldconfig, but it's unclear whether this had any impact.
surak
commented
4 years ago I tried patching init.py as both @esc and @arthurprat, but still no candy, as it goes to this part:
try:
if not getattr(llvmlite.binding.targets, "has_svml")():
# has detection function, but no svml compiled in, therefore
# disable SVML
print("No svml compiled in")
return FalseIf anyone got this far down, turns out that loading the so libraries (and get to where I got) is not enough. You need to have a patched version of llvm to begin with, with a SVML patch, as shown here:
https://llvmlite.readthedocs.io/en/latest/admin-guide/install.html#building-manually
Endor-8
commented
3 years ago @esc I've upgraded to the latest version [0.51.2] and the issue persists. Fortunately, my patch still works for my case.
I think the llvmlite.binding.load_library_permanently function is looking for binaries in the sys.prefix folder instead of the sys.prefix + r"\Library\bin" one, since I changed my patch to use the relative path r"Library\bin\svml_dispmd" and it worked as well, implying that it was originally searching in the sys.prefix folder.
At least in Windows, this seems to be the only problem.
esc
commented
3 years ago @Endor-8 thanks for following up on this.
lines-of-codes
commented
3 years ago From the @Endor-8 fixing, I tried that and everything turns True.
I'm using Windows 10 (Specifically: Microsoft Windows 10 Home Single Language)
System Info (after being fixed, Note that I'm using python -m numba -s instead of numba -s because It doesn't work...(Maybe because it's not in the System Environment Path)):
System info:
Time Stamp Report started (local time) : 2021-04-04 16:45:53.606467 UTC start time : 2021-04-04 09:45:53.606467 Running time (s) : 7.498291
Hardware Information Machine : AMD64 CPU Name : skylake CPU Count : 4 Number of accessible CPUs : ? List of accessible CPUs cores : ? CFS Restrictions (CPUs worth of runtime) : None
CPU Features : 64bit adx aes avx avx2 bmi bmi2 clflushopt cmov cx16 cx8 f16c fma fsgsbase fxsr invpcid lzcnt mmx movbe pclmul popcnt prfchw rdrnd rdseed sahf sgx sse sse2 sse3 sse4.1 sse4.2 ssse3 xsave xsavec xsaveopt xsaves
Memory Total (MB) : ? Memory Available (MB) : ?
OS Information Platform Name : Windows-10-10.0.18362-SP0
Platform Release : 10 OS Name : Windows OS Version : 10.0.18362 OS Specific Version : 10 10.0.18362 SP0 Multiprocessor Free Libc Version : ?
Python Information Python Compiler : MSC v.1927 64 bit (AMD64) Python Implementation : CPython Python Version : 3.9.1 Python Locale : th_TH.cp874
LLVM Information LLVM Version : 10.0.1
CUDA Information CUDA Device Initialized : True CUDA Driver Version : 11020 CUDA Detect Output: Found 1 CUDA devices id 0 b'GeForce MX130' [SUPPORTED] compute capability: 5.0 pci device id: 0 pci bus id: 1 Summary: 1/1 devices are supported
CUDA Libraries Test Output: Finding cublas from
named cublas.dll trying to open library... ERROR: failed to open cublas: Could not find module 'cublas.dll' (or one of its dependencies). Try using the full path with constructor syntax. Finding cusparse from named cusparse.dll trying to open library... ERROR: failed to open cusparse: Could not find module 'cusparse.dll' (or one of its dependencies). Try using the full path with constructor syntax. Finding cufft from named cufft.dll trying to open library... ERROR: failed to open cufft: Could not find module 'cufft.dll' (or one of its dependencies). Try using the full path with constructor syntax. Finding curand from named curand.dll trying to open library... ERROR: failed to open curand: Could not find module 'curand.dll' (or one of its dependencies). Try using the full path with constructor syntax. Finding nvvm from named nvvm.dll trying to open library... ERROR: failed to open nvvm: Could not find module 'nvvm.dll' (or one of its dependencies). Try using the full path with constructor syntax. Finding cudart from named cudart.dll trying to open library... ERROR: failed to open cudart: Could not find module 'cudart.dll' (or one of its dependencies). Try using the full path with constructor syntax. Finding cudadevrt from named cudadevrt.lib ERROR: failed to find cudadevrt: cudadevrt.lib not found Finding libdevice from searching for compute_20... ERROR: can't open libdevice for compute_20 searching for compute_30... ERROR: can't open libdevice for compute_30 searching for compute_35... ERROR: can't open libdevice for compute_35 searching for compute_50... ERROR: can't open libdevice for compute_50 ROC information ROC Available : False ROC Toolchains : None HSA Agents Count : 0 HSA Agents: None HSA Discrete GPUs Count : 0 HSA Discrete GPUs : None
SVML Information SVML State, config.USING_SVML : True SVML Library Loaded : True llvmlite Using SVML Patched LLVM : True SVML Operational : True
Threading Layer Information TBB Threading Layer Available : False +--> Disabled due to Unknown import problem. OpenMP Threading Layer Available : True +-->Vendor: MS Workqueue Threading Layer Available : True +-->Workqueue imported successfully.
Numba Environment Variable Information None found.
Conda Information Conda not available.
Installed Packages Package Version
alabaster 0.7.12 altgraph 0.17 arcade 2.4.3 argcomplete 1.12.2 astroid 2.5.1 attrs 20.3.0 Babel 2.9.0 certifi 2020.12.5 cffi 1.14.5 chardet 4.0.0 click 7.1.2 colorama 0.4.4 cryptography 3.4.6 Cython 0.29.22 dearpygui 0.6.294 docutils 0.16 easygui 0.98.1 future 0.18.2 icc-rt 2019.0 idna 2.10 imagesize 1.2.0 intel-openmp 2021.2.0 isort 5.7.0 Jinja2 2.11.3 Kivy 2.0.0 kivy-deps.angle 0.3.0 kivy-deps.glew 0.3.0 kivy-deps.gstreamer 0.3.1 kivy-deps.sdl2 0.3.1 Kivy-Garden 0.1.4 lazy-object-proxy 1.5.2 llvmlite 0.36.0 MarkupSafe 1.1.1 mccabe 0.6.1 numba 0.53.1 numpy 1.19.5 orjson 3.5.1 packaging 20.8 panda 0.3.1 panda3d 1.10.8 pandas 1.2.1 pefile 2019.4.18 Pillow 8.1.0 pip 21.0.1 pipx 0.16.0.0 pybind11 2.6.2 pycparser 2.20 pygame 2.0.1 pyglet 1.5.15 Pygments 2.8.0 pyinstaller 4.1 pyinstaller-hooks-contrib 2020.11 pylint 2.7.2 pymunk 5.7.0 pyparsing 2.4.7 pyperclip 1.8.1 pypiwin32 223 PyQt5 5.15.2 pyqt5-plugins 5.15.2.2.0.1 PyQt5-sip 12.8.1 pyqt5-tools 5.15.2.3.0.2 python-dateutil 2.8.1 python-dotenv 0.15.0 pytiled-parser 0.9.4a3 pytz 2020.5 pywin32 300 pywin32-ctypes 0.2.0 PyYAML 5.4.1 qt5-applications 5.15.2.2.1 qt5-tools 5.15.2.1.0.1 requests 2.25.1 screeninfo 0.6.7 selenium 3.141.0 setuptools 54.1.2 six 1.15.0 snowballstemmer 2.1.0 Sphinx 3.5.1 sphinxcontrib-applehelp 1.0.2 sphinxcontrib-devhelp 1.0.2 sphinxcontrib-htmlhelp 1.0.3 sphinxcontrib-jsmath 1.0.1 sphinxcontrib-qthelp 1.0.3 sphinxcontrib-serializinghtml 1.1.4 toml 0.10.2 tqdm 4.56.0 unitypackage-extractor 1.0.0 urllib3 1.26.4 ursina 3.4.0 userpath 1.4.2 wheel 0.36.2 wrapt 1.12.1 wxPython 4.1.1
Error log Error (subprocess): [WinError 2] The system cannot find the file specified Error (subprocess): [WinError 2] The system cannot find the file specified
Warning log Warning (roc): Error initialising ROC: No ROC toolchains found. Warning (roc): No HSA Agents found, encountered exception when searching: Error at driver init:
HSA is not currently supported on this platform (win32). : Warning: Conda not available. Error was [WinError 2] The system cannot find the file specified
Warning (psutil): psutil cannot be imported. For more accuracy, consider installing it.
If requested, please copy and paste the information between the dashed (----) lines, or from a given specific section as appropriate.
============================================================= IMPORTANT: Please ensure that you are happy with sharing the contents of the information present, any information that you wish to keep private you should remove before sharing.
github-actions[bot]
commented
3 years ago This issue is marked as stale as it has had no activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with any updates and confirm that this issue still needs to be addressed.
gmarkall
commented
3 years ago @esc @stuartarchibald It seems like a summary of this issue is:
LD_LIBRARY_PATH.Would it make sense to create / accept a patch to Numba that also searches for the pip-installed libsvml location? If so, I wonder if this might be a good first or second issue?
stuartarchibald
commented
3 years ago @esc @stuartarchibald It seems like a summary of this issue is:
* The pip-installed wheel puts libsvml somewhere that Numba doesn't expect it * The workarounds include patching Numba to look in the pip-installed location as well, or running ldconfig / setting `LD_LIBRARY_PATH`.Would it make sense to create / accept a patch to Numba that also searches for the pip-installed libsvml location? If so, I wonder if this might be a good first or second issue?
I think when @sklam and I looked at this a while ago in relation to libtbb (which has the same problem), there wasn't a consistent place that pip installed libraries. Perhaps gathering and collating all the information on this would probably be the first task so as to work out if it's feasible/reasonable to attempt?
github-actions[bot]
commented
3 years ago This issue is marked as stale as it has had no activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with any updates and confirm that this issue still needs to be addressed.
lfaucheux
commented
3 years ago [unstaling]. That's what I've come to.
if 32 << bool(sys.maxsize >> 32) == 64:
# `NB_SVML_LIBS_DIR` below is arbitrary: it may have been set manually beforehand.
# ... Otherwise, the default values are pretty robust for what I can tell.
_nb_svml_dir = os.environ.get('NB_SVML_LIBS_DIR') or os.path.join(
sys.prefix, *(["Library", "bin"] if ON_WINDOWS else ["lib"])
)
from llvmlite.binding import load_library_permanently
_nb_loader = lambda so: load_library_permanently(
os.path.join(_nb_svml_dir, so)
)
if ON_LINUX:
_nb_loader("libintlc.so.5")
_nb_loader("libsvml.so")
elif ON_DARWIN:
_nb_loader("libintlc.dylib")
_nb_loader("libsvml.dylib")
elif ON_WINDOWS:
_nb_loader("svml_dispmd")
import numba as nb
from numba.misc.numba_sysinfo import display_sysinfo
display_sysinfo()where ON_LINUX, ON_DARWIN and ON_WINDOWS are defined as you do in __init__.py. Also I do not remember why I restricted the approach to 64bit systems (?)
jorenham
commented
11 months ago This is also relevant to intel-cmplr-lib-rt on numba==0.58.1.
The only way I can get SVML to work (with python-poetry), is like this:
export LD_LIBRARY_PATH=/path/to/my/project/.venv/lib
$ poetry run python
Python 3.11.6 (main, Oct 23 2023, 22:48:54) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numba
>>> numba.config.USING_SVML
TrueUnfortunately, dynamically setting the env var does not work:
>>> import os, sys
>>> os.environ["LD_LIBRARY_PATH"] = f"{sys.prefix}/lib"
>>> import numba
>>> numba.config.USING_SVML
False
>>> numba._try_enable_svml()
FalseFor any other poetry users that stumble across this issue: You can work around this issue by installing poetry-dotenv-plugin, so that you can create an .env file in your project root with e.g. LD_LIBRARY_PATH=${PWD}/.venv/lib.
pwuertz
commented
1 month ago Since Python >= 3.10 we can make use of importlib.metadata for discovering Python packages and bundled files.
Here is a function for locating and loading SVML at runtime, which should work with both intel-cmplr-lib-rt and icc_rt packages and without manual preparations like LD_LIBRARY_PATH or ldconfig:
import typing
import pathlib
def load_svml_from_distribution() -> bool:
"""
Attempt to load SVML from known distributions like "intel-cmplr-lib-rt" or "icc_rt".
Returns True if the library was loaded successfully.
"""
import importlib.metadata # <- Requires Python >= 3.10, but Numba still supports 3.9
def iter_distributions(names: typing.Iterable[str]) -> typing.Iterator[importlib.metadata.Distribution]:
for name in names:
try:
yield importlib.metadata.distribution(name)
except importlib.metadata.PackageNotFoundError:
continue
def locate_file(filename: str, dist: importlib.metadata.Distribution) -> pathlib.Path | None:
return next((dist.locate_file(f).resolve() for f in dist.files if f.name == filename), None)
# TODO: Add library names for other platforms
required_libraries = ["libintlc.so.5", "libsvml.so"]
package_candidates = ["intel-cmplr-lib-rt", "icc-rt"]
for dist in iter_distributions(package_candidates):
library_paths = [locate_file(lib, dist) for lib in required_libraries]
if all(library_paths):
from llvmlite import binding
for library_path in library_paths:
binding.load_library_permanently(str(library_path))
return True
return False@stuartarchibald This is probably what you had in mind in https://github.com/numba/numba/issues/4713#issuecomment-855834064?
Hello, I am trying to take advantage of Intel SVML. However, installing icc_rt by using pip (through PyCharm) does not make numba utilizes it since
numba -ssays SVML is False as below. This instruction only gives a way to install with conda. Is there any way to do it without conda?