search

snorkel-team / snorkel

A system for quickly generating training data with weak supervision
https://snorkel.org
Apache License 2.0
5.81k stars 857 forks source link

Intel MKL from PyPi does not suffice #1506

Closed chaturv3di closed 4 years ago

chaturv3di commented 4 years ago

Issue description

I've built my virtual environment solely from PyPi (and it's a requirement). Everything installs well, but I'm facing run time error: E ImportError: libmkl_rt.so: cannot open shared object file: No such file or directory. This is used by scipy for multi-threading. This error can be avoided when installing nomkl package, which is available only from anaconda.

Since I can install only using pip, I can install intel-scipy, intel-numpy, etc. that come with multi-threading support. However, these packages only provide scipy-1.1.0 and numpy-1.15.1, which do not meet Snorkel's requirements.

Has anyone tried a pip-only installation and gotten things to work?

Here's my complete error log.

    from snorkel.labeling import LFAnalysis
../../build/myProject/venv/lib/python3.6/site-packages/snorkel/labeling/__init__.py:3: in <module>
    from .analysis import LFAnalysis  # noqa: F401
../../build/myProject/venv/lib/python3.6/site-packages/snorkel/labeling/analysis.py:8: in <module>
    from sklearn.metrics import confusion_matrix
../../build/myProject/venv/lib/python3.6/site-packages/sklearn/__init__.py:76: in <module>
    from .base import clone
../../build/myProject/venv/lib/python3.6/site-packages/sklearn/base.py:16: in <module>
    from .utils import _IS_32BIT
../../build/myProject/venv/lib/python3.6/site-packages/sklearn/utils/__init__.py:20: in <module>
    from .validation import (as_float_array,
../../build/myProject/venv/lib/python3.6/site-packages/sklearn/utils/validation.py:21: in <module>
    from .fixes import _object_dtype_isnan
../../build/myProject/venv/lib/python3.6/site-packages/sklearn/utils/fixes.py:18: in <module>
    from scipy.sparse.linalg import lsqr as sparse_lsqr  # noqa
../../build/myProject/venv/lib/python3.6/site-packages/scipy/sparse/linalg/__init__.py:113: in <module>
    from .isolve import *
../../build/myProject/venv/lib/python3.6/site-packages/scipy/sparse/linalg/isolve/__init__.py:6: in <module>
    from .iterative import *
../../build/myProject/venv/lib/python3.6/site-packages/scipy/sparse/linalg/isolve/iterative.py:10: in <module>
    from . import _iterative
E   ImportError: libmkl_rt.so: cannot open shared object file: No such file or directory

System info

Here's the list of relevant packages in my current environment (using standard numpy and scipy, and not Intel's).

intel-openmp             2019.0         
mkl                      2019.0         
mkl-include              2019.0         
numpy                    1.16.4         
packaging                18.0           
pandas                   0.25.1         
scikit-learn             0.21.2         
scipy                    1.3.1          
snorkel                  0.9.2
chaturv3di commented 4 years ago

This turned out to be relatively straightforward. I just had to update my environment variable.

export LD_LIBRARY_PATH=/path/to/build/myProject/venv/lib/:$LD_LIBRARY_PATH