Annoying install issue between scipy and scikit-learn

[jacquerie]

When running

(beard) $  pip install --no-cache-dir git+https://github.com/inveniosoftware/beard@master#egg=beard

in a fresh virtualenv we get:

Collecting beard from git+https://github.com/inveniosoftware/beard@master#egg=beard
  Cloning https://github.com/inveniosoftware/beard (to master) to /tmp/pip-build-FTsam1/beard
Collecting jellyfish (from beard)
  Downloading jellyfish-0.5.1.tar.gz
Collecting numpy>=1.9 (from beard)
  Downloading numpy-1.10.1.tar.gz (4.0MB)
    100% |████████████████████████████████| 4.1MB 55.5MB/s 
Collecting scikit-learn>=0.15.2 (from beard)
  Downloading scikit-learn-0.17.tar.gz (7.8MB)
    100% |████████████████████████████████| 7.8MB 29.8MB/s 
Collecting scipy>=0.14 (from beard)
  Downloading scipy-0.16.1.tar.gz (12.2MB)
    100% |████████████████████████████████| 12.2MB 57.4MB/s 
Collecting six (from beard)
  Downloading six-1.10.0-py2.py3-none-any.whl
Collecting unidecode (from beard)
  Downloading Unidecode-0.04.18.tar.gz (206kB)
    100% |████████████████████████████████| 208kB 55.8MB/s 
Collecting fuzzy (from beard)
  Downloading Fuzzy-1.1.tar.gz
Installing collected packages: jellyfish, numpy, scikit-learn, scipy, six, unidecode, fuzzy, beard
  Running setup.py install for jellyfish
  Running setup.py install for numpy
  Running setup.py install for scikit-learn
    Complete output from command /home/jnotarst/.virtualenvs/beard/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-FTsam1/scikit-learn/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-4HmVs6-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/jnotarst/.virtualenvs/beard/include/site/python2.7/scikit-learn:
    Partial import of sklearn during the build process.
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-FTsam1/scikit-learn/setup.py", line 247, in <module>
        setup_package()
      File "/tmp/pip-build-FTsam1/scikit-learn/setup.py", line 237, in setup_package
        .format(scipy_req_str, instructions))
    ImportError: Scientific Python (SciPy) is not installed.
    scikit-learn requires SciPy >= 0.9.
    Installation instructions are available on the scikit-learn website: http://scikit-learn.org/stable/install.html

    ----------------------------------------
Command "/home/jnotarst/.virtualenvs/beard/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-FTsam1/scikit-learn/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-4HmVs6-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/jnotarst/.virtualenvs/beard/include/site/python2.7/scikit-learn" failed with error code 1 in /tmp/pip-build-FTsam1/scikit-learn

that is, pip tries to install scikit-learn first, which checks for scipy, which isn't yet installed, so it fails.

Of course, if we install scipy first and then we run again that command, everything works:

(beard) $ pip install scipy>=0.14
(beard) $ pip install --no-cache-dir git+https://github.com/inveniosoftware/beard@master#egg=beard
Collecting beard from git+https://github.com/inveniosoftware/beard@master#egg=beard
  Cloning https://github.com/inveniosoftware/beard (to master) to /tmp/pip-build-VkyqKb/beard
Requirement already satisfied (use --upgrade to upgrade): jellyfish in ./lib/python2.7/site-packages (from beard)
Requirement already satisfied (use --upgrade to upgrade): numpy>=1.9 in ./lib/python2.7/site-packages (from beard)
Collecting scikit-learn>=0.15.2 (from beard)
  Downloading scikit-learn-0.17.tar.gz (7.8MB)
    100% |████████████████████████████████| 7.8MB 53.3MB/s 
Requirement already satisfied (use --upgrade to upgrade): scipy>=0.14 in ./lib/python2.7/site-packages (from beard)
Collecting six (from beard)
  Downloading six-1.10.0-py2.py3-none-any.whl
Collecting unidecode (from beard)
  Downloading Unidecode-0.04.18.tar.gz (206kB)
    100% |████████████████████████████████| 208kB 29.7MB/s 
Collecting fuzzy (from beard)
  Downloading Fuzzy-1.1.tar.gz
Installing collected packages: scikit-learn, six, unidecode, fuzzy, beard
  Running setup.py install for scikit-learn
  Running setup.py install for unidecode
  Running setup.py install for fuzzy
  Running setup.py install for beard
Successfully installed beard-0.0 fuzzy-1.1 scikit-learn-0.17 six-1.10.0 unidecode-0.4.18

[MSusik]

So, does changing the order of dependencies in beard/setupy.py work?

I ordered them alphabetically without thinking about such consequences :stuck_out_tongue:

[jacquerie]

...yes, it does!

(beard) $ pip install --no-cache-dir git+https://github.com/jacquerie/beard@import-order#egg=beard
Collecting beard from git+https://github.com/jacquerie/beard@import-order#egg=beard
  Cloning https://github.com/jacquerie/beard (to import-order) to /tmp/pip-build-d4GoeS/beard
Collecting jellyfish (from beard)
  Downloading jellyfish-0.5.1.tar.gz
Collecting numpy>=1.9 (from beard)
  Downloading numpy-1.10.1.tar.gz (4.0MB)
    100% |████████████████████████████████| 4.1MB 42.7MB/s 
Collecting scipy>=0.14 (from beard)
  Downloading scipy-0.16.1.tar.gz (12.2MB)
    100% |████████████████████████████████| 12.2MB 52.3MB/s 
Collecting scikit-learn>=0.15.2 (from beard)
  Downloading scikit-learn-0.17.tar.gz (7.8MB)
    100% |████████████████████████████████| 7.8MB 50.9MB/s 
Collecting six (from beard)
  Downloading six-1.10.0-py2.py3-none-any.whl
Collecting unidecode (from beard)
  Downloading Unidecode-0.04.18.tar.gz (206kB)
    100% |████████████████████████████████| 208kB 24.9MB/s 
Collecting fuzzy (from beard)
  Downloading Fuzzy-1.1.tar.gz
Installing collected packages: jellyfish, numpy, scipy, scikit-learn, six, unidecode, fuzzy, beard
  Running setup.py install for jellyfish
  Running setup.py install for numpy
  Running setup.py install for scipy
  Running setup.py install for scikit-learn
  Running setup.py install for unidecode
  Running setup.py install for fuzzy
  Running setup.py install for beard
Successfully installed beard-0.0 fuzzy-1.1 jellyfish-0.5.1 numpy-1.10.1 scikit-learn-0.17 scipy-0.16.1 six-1.10.0 unidecode-0.4.18

Submitting a PR now.

[kaplun]

So it's a bug in scikit-learn?

[jacquerie]

So it's a bug in scikit-learn?

Arguably, no, because scikit-learn's documentation tells you to install scipy and numpy first, so the problem only happens if you try to install them at the same time. On the other hand, it's always annoying when code that looks declarative is in fact procedural, so we could argue that it is a bug in pip...

[glouppe]

We explicitly dont pip install scipy and numpy in scikit-learn as we believe installing these two packages properly is very installation dependent and difficult to make right automatically. In particular, this stems eg from the fact these libraries depend on algebra libraries (blas, lapack, etc) for which there exists several implementations and for which it is crucially important to let the user decide what is best for his needs.

[kaplun]

In particular, this stems eg from the fact these libraries depend on algebra libraries (blas, lapack, etc) for which there exists several implementations and for which it is crucially important to let the user decide what is best for his needs.

But these libraries are system dependent anyway, therefore they wouldn't be installed by pip, only assumed to exist for the corresponding pythonic wrappers. In principle pip-installing scipy/numpy is going to use whatever library it will find available. (unless of course you can install multiple versions of blas/lapack and therefore one has to make an informed choice when building scipy/numpy...)

[glouppe]

Yes, but this will lead to a very suboptimal installation and very lengthy building process. In summary, the recommended way is to use binaries that come properly configured for your installation (e.g., the one of the conda packages if you are using anaconda, as we do to install the system here for Travis).

[dubey5]

I have python 3.6 and have installed numpy but not able to install scipy and sklearn.I Have tried almost every way from every source but still not working.These modules are not compatible with 3.6 so should i downgrade my python version? Plz help

[arsalan0312]

how can i install scipy? please guide me

[MSusik]

@dubey5 @arsalan0312 scipy and sklearn are available for Python 3.6. You can use pip to install them, but an easier way would be to try an anaconda environment.

[arsalan0312]

@MSusik thank you but when i install scipy using pip then the following error occure.. " config = setup_module.configuration(*args) File "scipy\linalg\setup.py", line 20, in configuration raise NotFoundError('no lapack/blas resources found') numpy.distutils.system_info.NotFoundError: no lapack/blas resources found"

[MSusik]

@arsalan0312 This is not related to the beard library, you do not have lapack installed. It's a dependency of scipy.

[arsalan0312]

@MSusik do you any right link of lapack ?, so i can installed it .

[glouppe]

@arsalan0312 You should try installing your Python environment with the Anaconda distribution. https://www.continuum.io/downloads

All of those installations and dependencies issues are taken care of for you in this way.

On 15 June 2017 at 23:08, Muhammad Arsalan notifications@github.com wrote:

@MSusik https://github.com/msusik do you any right link of lapack ?, so i can installed it .

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/inspirehep/beard/issues/80#issuecomment-308866263, or mute the thread https://github.com/notifications/unsubscribe-auth/AAdKSzdbBCn1eL6ZNFD5_dRBRlAio3u8ks5sEZ09gaJpZM4GegAK .

[andikas]

Hi all,

I have installed scipy to my system,

But when I try to install scikit-learn, its throw error that scipy not installed yet.

Do you guys have any idea about this? Really need some advice here.

Thanks, Cheers

[MSusik]

The error says that Numpy is missing the link to the Math Kernel Library (MKL).

As @glouppe said, I'd suggest using Anaconda. It is easier.

[andikas]

Hi @MSusik ,

Thanks for your help. The problem solved by installing Numpy-MKL.

Thanks, Cheers