pip install issue: Mac OS X and AWS Linux

wyattmcdonnell commented 4 years ago

Hey Shobi!

I get the following when trying to pip install parc in a fresh Anaconda environment with Python 3.7 on both Mac OS X 10.14.6 (Mojave) and on AWS' flavor of Linux:

Collecting parc
  Downloading https://files.pythonhosted.org/packages/2d/82/7dcddc30ba3cd6cec30a1f939d40bb21882bef791d87a1ddab174e825b5e/parc-0.18-py3-none-any.whl
Collecting python-igraph (from parc)
  Using cached https://files.pythonhosted.org/packages/0f/a0/4e7134f803737aa6eebb4e5250565ace0e2599659e22be7f7eba520ff017/python-igraph-0.7.1.post6.tar.gz
Collecting leidenalg (from parc)
Collecting pandas (from parc)
  Using cached https://files.pythonhosted.org/packages/39/73/99aa822ee88cef5829607217c11bf24ecc1171ae5d49d5f780085f5da518/pandas-0.25.1-cp37-cp37m-macosx_10_9_x86_64.macosx_10_10_x86_64.whl
Collecting hnswlib (from parc)
  Using cached https://files.pythonhosted.org/packages/51/ee/850ac2cdc9483a5a26fd4173be486f48db0bdb9e2b200dfc3149a572a907/hnswlib-0.3.2.0.tar.gz
Collecting pybind11 (from parc)
  Downloading https://files.pythonhosted.org/packages/4b/4d/ae1c4d8e8b139afa9682054dd42df3b0e3b5c1731287933021b9fd7e9cc4/pybind11-2.4.3-py2.py3-none-any.whl (150kB)
     |████████████████████████████████| 153kB 3.6MB/s 
Collecting scipy (from parc)
  Using cached https://files.pythonhosted.org/packages/d5/06/1a696649f4b2e706c509cb9333fdc6331fbe71251cede945f9e1fa13ea34/scipy-1.3.1-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Collecting numpy (from parc)
  Using cached https://files.pythonhosted.org/packages/b4/e8/5ececadd9cc220bb783b4ce6ffaa9266925d37ed41237bc23bc530ab4f3d/numpy-1.17.2-cp37-cp37m-macosx_10_6_intel.whl
Collecting pytz>=2017.2 (from pandas->parc)
  Downloading https://files.pythonhosted.org/packages/e7/f9/f0b53f88060247251bf481fa6ea62cd0d25bf1b11a87888e53ce5b7c8ad2/pytz-2019.3-py2.py3-none-any.whl (509kB)
     |████████████████████████████████| 512kB 16.1MB/s 
Collecting python-dateutil>=2.6.1 (from pandas->parc)
  Using cached https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl
Collecting six>=1.5 (from python-dateutil>=2.6.1->pandas->parc)
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Building wheels for collected packages: python-igraph, hnswlib
  Building wheel for python-igraph (setup.py) ... done
  Created wheel for python-igraph: filename=python_igraph-0.7.1.post6-cp37-cp37m-macosx_10_9_x86_64.whl size=1903873 sha256=0c6639b7e49b66a679add893935b4cdabec863b711406adf644e1a59b6726a56
  Stored in directory: /Users/wyatt.mcdonnell/Library/Caches/pip/wheels/41/d6/02/34eebae97e25f5b87d60f4c0687e00523e3f244fa41bc3f4a7
  Building wheel for hnswlib (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: //anaconda3/envs/parc/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/bq/7jsphcrn5jj6rt7jrll1x5y80000gp/T/pip-install-yfhtmyc0/hnswlib/setup.py'"'"'; __file__='"'"'/private/var/folders/bq/7jsphcrn5jj6rt7jrll1x5y80000gp/T/pip-install-yfhtmyc0/hnswlib/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/bq/7jsphcrn5jj6rt7jrll1x5y80000gp/T/pip-wheel-wlmtzn3j --python-tag cp37
       cwd: /private/var/folders/bq/7jsphcrn5jj6rt7jrll1x5y80000gp/T/pip-install-yfhtmyc0/hnswlib/
  Complete output (41 lines):
  running bdist_wheel
  running build
  running build_ext
  creating var
  creating var/folders
  creating var/folders/bq
  creating var/folders/bq/7jsphcrn5jj6rt7jrll1x5y80000gp
  creating var/folders/bq/7jsphcrn5jj6rt7jrll1x5y80000gp/T
  gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I//anaconda3/envs/parc/include -arch x86_64 -I//anaconda3/envs/parc/include -arch x86_64 -I//anaconda3/envs/parc/include/python3.7m -c /var/folders/bq/7jsphcrn5jj6rt7jrll1x5y80000gp/T/tmpq3tagf4m.cpp -o var/folders/bq/7jsphcrn5jj6rt7jrll1x5y80000gp/T/tmpq3tagf4m.o -std=c++14
  gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I//anaconda3/envs/parc/include -arch x86_64 -I//anaconda3/envs/parc/include -arch x86_64 -I//anaconda3/envs/parc/include/python3.7m -c /var/folders/bq/7jsphcrn5jj6rt7jrll1x5y80000gp/T/tmpw16kygtu.cpp -o var/folders/bq/7jsphcrn5jj6rt7jrll1x5y80000gp/T/tmpw16kygtu.o -fvisibility=hidden
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/private/var/folders/bq/7jsphcrn5jj6rt7jrll1x5y80000gp/T/pip-install-yfhtmyc0/hnswlib/setup.py", line 116, in <module>
      zip_safe=False,
    File "//anaconda3/envs/parc/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
      return distutils.core.setup(**attrs)
    File "//anaconda3/envs/parc/lib/python3.7/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "//anaconda3/envs/parc/lib/python3.7/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "//anaconda3/envs/parc/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "//anaconda3/envs/parc/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 192, in run
      self.run_command('build')
    File "//anaconda3/envs/parc/lib/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "//anaconda3/envs/parc/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "//anaconda3/envs/parc/lib/python3.7/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "//anaconda3/envs/parc/lib/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "//anaconda3/envs/parc/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "//anaconda3/envs/parc/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 84, in run
      _build_ext.run(self)
    File "//anaconda3/envs/parc/lib/python3.7/distutils/command/build_ext.py", line 340, in run
      self.build_extensions()
    File "/private/var/folders/bq/7jsphcrn5jj6rt7jrll1x5y80000gp/T/pip-install-yfhtmyc0/hnswlib/setup.py", line 88, in build_extensions
      import pybind11
  ModuleNotFoundError: No module named 'pybind11'
  ----------------------------------------
  ERROR: Failed building wheel for hnswlib
  Running setup.py clean for hnswlib
Successfully built python-igraph
Failed to build hnswlib
Installing collected packages: python-igraph, leidenalg, numpy, pytz, six, python-dateutil, pandas, pybind11, hnswlib, scipy, parc
  Running setup.py install for hnswlib ... done
Successfully installed hnswlib-0.3.2.0 leidenalg-0.7.0 numpy-1.17.2 pandas-0.25.1 parc-0.18 pybind11-2.4.3 python-dateutil-2.8.0 python-igraph-0.7.1.post6 pytz-2019.3 scipy-1.3.1 six-1.12.0

For what it's worth, if I manually install everything first with the following bash script and then clone PARC.git and python3 setup.py install everything works!

!/usr/bin/env bash
conda create --name parc_20191014 python=3 --yes |& tee -a parc.log
conda activate parc_20191014 |& tee -a parc.log
conda install -c anaconda numpy pandas scipy setuptools scikit-learn seaborn --yes |& tee -a parc.log
conda install -c conda-forge pybind11 python-igraph umap-learn --yes |& tee -a parc.log
pip install leidenalg |& tee -a parc.log
pip install ipykernel |& tee -a parc.log
pip3 install jupyter |& tee -a parc.log
python -m ipykernel install --user --name parc --display-name "Python (parc)" |& tee -a parc.log
git clone https://github.com/nmslib/hnswlib |& tee -a parc.log
cd hnswlib |& tee -a parc.log
cd python_bindings |& tee -a parc.log
python3 setup.py install |& tee -a parc.log
cd ../../ |& tee -a parc.log
git clone https://github.com/ShobiStassen/PARC.git |& tee -a parc.log
cd PARC |& tee -a parc.log
# 2019-10-14, most recently tested with: git checkout 46a36996539bf57bb0d544898d900a63fdaa5b90 |& tee -a parc.log
python3 setup.py install

Cheers, Wyatt

ShobiStassen commented 4 years ago

hi Wyatt! I think this has to do with setup.py not installing pybind11 before it tries to install hnswlib. Does the pip install work if you first install pybind11 followed by pip install parc pip install pybind11 then pip install parc

wyattmcdonnell commented 4 years ago

Hi Shobi!

If I pip install pybind11 and then pip install parc, I get effectively the same error, but with numpy, as follows:

File "/mnt/home/wyatt.mcdonnell/anaconda3/envs/parc_pip/lib/python3.7/distut
ils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/mnt/home/wyatt.mcdonnell/anaconda3/envs/parc_pip/lib/python3.7/distut
ils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/mnt/home/wyatt.mcdonnell/anaconda3/envs/parc_pip/lib/python3.7/site-p
ackages/setuptools/command/build_ext.py", line 84, in run
      _build_ext.run(self)
    File "/mnt/home/wyatt.mcdonnell/anaconda3/envs/parc_pip/lib/python3.7/distut
ils/command/build_ext.py", line 340, in run
      self.build_extensions()
    File "/tmp/pip-install-_xfidwoz/hnswlib/setup.py", line 89, in build_extensi
ons
      import numpy as np
  ModuleNotFoundError: No module named 'numpy'
  ----------------------------------------
  ERROR: Failed building wheel for hnswlib
  Running setup.py clean for hnswlib

esimonds commented 4 years ago

Congrats to Shobi and team for the official publication in Bioinformatics yesterday!

@wyattmcdonnell I can't comment on how this would work in an AWS Linux instance, but in OS X Catalina, the steps below worked to install PARC inside Anaconda:

Download and install Anaconda with graphical GUI (Anaconda Navigator) and Python 3.7
In Anaconda Navigator, go to "Environments" tab
Create an environment called "ParcEnv" with Python 3.7
In Anaconda navigator, go to "Home" tab
Change channel to "ParcEnv"
Launch Qt console
Install as many packages as possible with conda: conda install pybind11 numpy scipy pandas matplotlib sklearn
Install remaining packages with pip: pip install hnswlib python-igraph leidenalg

You should get this output:

Successfully built python-igraph leidenalg hnswlib
Installing collected packages: python-igraph, leidenalg, hnswlib, parc
Successfully installed hnswlib-0.3.4 leidenalg-0.7.0 parc-0.19 python-igraph-0.7.1.post6
Note: you may need to restart the kernel to use updated packages.

install sklearn to run the demo dataset: pip install sklearn

You should get this output:

Successfully installed joblib-0.14.1 scikit-learn-0.22.1 sklearn-0.0

From the PARC GitHub home page, run the code in "Example Usage 1. (small test sets) - IRIS and Digits dataset from sklearn". Just be sure to convert all of the "//" into "#" if running as a script (in the Spyder script editor, for example).

Output looks like this:

input data has shape 150 (samples) x 4 (features)
Starting local pruning based on Minkowski metric at 2 s.dev above mean
Starting global pruning
Starting community detection
0.01191401481628418
list of cluster labels and populations 6 [(0, 36), (1, 34), (2, 29), (3, 21), (4, 19), (5, 11)]
time elapsed 0.1 seconds
target is 0
cluster 2  has majority 0 with population 29
cluster 3  has majority 0 with population 21
target 0 has f1-score of 100.00
...
target is 9
cluster 1  has majority 9 with population 147
target 9 has f1-score of 88.69
f1-score (unweighted) mean 95.28 %
f1-score weighted (by population) 95.30 %

...and some plots, too.

I realize your comment is a few months old, but I hope that helps someone get started on OS X!

ES

ShobiStassen / PARC

pip install issue: Mac OS X and AWS Linux #3