Installing via pip does not resolve some dependencies

[tpanza]

When I run this on Linux command line from an activated conda environment:

pip install parameter-sherpa

I get this error:

Looking in indexes: https://[PROXY-TO-PYPI]/pypi/pypi-releases/simple
Collecting parameter-sherpa
  Downloading https://[PROXY-TO-PYPI]/pypi/pypi-releases/packages/e4/6a/96dac197a9c8f3aa82a83a2805bc40ba615613e35a715e70912385670fb7/parameter-sherpa-1.0.5.tar.gz (513kB)
     |████████████████████████████████| 522kB 600kB/s
    ERROR: Command errored out with exit status 1:
     command: /app/local/anaconda3/envs/my-conda-env/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-spl_3eo8/parameter-sherpa/setup.py'"'"'; __file__='"'"'/tmp/pip-install-spl_3eo8/parameter-sherpa/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info
         cwd: /tmp/pip-install-spl_3eo8/parameter-sherpa/
    Complete output (11 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-spl_3eo8/parameter-sherpa/setup.py", line 13, in <module>
        import sherpa
      File "/tmp/pip-install-spl_3eo8/parameter-sherpa/sherpa/__init__.py", line 21, in <module>
        from .core import *
      File "/tmp/pip-install-spl_3eo8/parameter-sherpa/sherpa/core.py", line 32, in <module>
        from .database import _Database
      File "/tmp/pip-install-spl_3eo8/parameter-sherpa/sherpa/database.py", line 22, in <module>
        import pymongo
    ModuleNotFoundError: No module named 'pymongo'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

So I then manually installed pymongo and tried again. I then received this error:

Looking in indexes: https://[PROXY-TO-PYPI]/pypi/pypi-releases/simple
Collecting parameter-sherpa
  Downloading https://[PROXY-TO-PYPI]/pypi/pypi-releases/packages/e4/6a/96dac197a9c8f3aa82a83a2805bc40ba615613e35a715e70912385670fb7/parameter-sherpa-1.0.5.tar.gz (513kB)
     |████████████████████████████████| 522kB 1.4MB/s
    ERROR: Command errored out with exit status 1:
     command: /app/local/anaconda3/envs/my-conda-env/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-19323g_f/parameter-sherpa/setup.py'"'"'; __file__='"'"'/tmp/pip-install-19323g_f/parameter-sherpa/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info
         cwd: /tmp/pip-install-19323g_f/parameter-sherpa/
    Complete output (11 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-19323g_f/parameter-sherpa/setup.py", line 13, in <module>
        import sherpa
      File "/tmp/pip-install-19323g_f/parameter-sherpa/sherpa/__init__.py", line 24, in <module>
        from . import algorithms
      File "/tmp/pip-install-19323g_f/parameter-sherpa/sherpa/algorithms/__init__.py", line 20, in <module>
        from .core import *
      File "/tmp/pip-install-19323g_f/parameter-sherpa/sherpa/algorithms/core.py", line 28, in <module>
        import sklearn.gaussian_process
    ModuleNotFoundError: No module named 'sklearn'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Further, after I manually installed the scikit-learn package, I got another similar error, this time for GPyOpt.

Looking in indexes: https://[PROXY-TO-PYPI]/pypi/pypi-releases/simple
Collecting parameter-sherpa
  Downloading https://[PROXY-TO-PYPI]/pypi/pypi-releases/packages/e4/6a/96dac197a9c8f3aa82a83a2805bc40ba615613e35a715e70912385670fb7/parameter-sherpa-1.0.5.tar.gz (513kB)
     |████████████████████████████████| 522kB 1.5MB/s
    ERROR: Command errored out with exit status 1:
     command: /app/local/anaconda3/envs/my-conda-env/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-pfmvbs6a/parameter-sherpa/setup.py'"'"'; __file__='"'"'/tmp/pip-install-pfmvbs6a/parameter-sherpa/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info
         cwd: /tmp/pip-install-pfmvbs6a/parameter-sherpa/
    Complete output (11 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-pfmvbs6a/parameter-sherpa/setup.py", line 13, in <module>
        import sherpa
      File "/tmp/pip-install-pfmvbs6a/parameter-sherpa/sherpa/__init__.py", line 24, in <module>
        from . import algorithms
      File "/tmp/pip-install-pfmvbs6a/parameter-sherpa/sherpa/algorithms/__init__.py", line 22, in <module>
        from .bayesian_optimization import GPyOpt
      File "/tmp/pip-install-pfmvbs6a/parameter-sherpa/sherpa/algorithms/bayesian_optimization.py", line 8, in <module>
        import GPyOpt as gpyopt_package
    ModuleNotFoundError: No module named 'GPyOpt'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

So I then had to manually run pip install GPyOpt.

After that, I was then able to successfully run pip install parameter-sherpa. In this case, some other dependencies actually were successfully resolved:

Looking in indexes: https://[PROXY-TO-PYPI]/pypi/pypi-releases/simple
Collecting parameter-sherpa
  Downloading https://[PROXY-TO-PYPI]/pypi/pypi-releases/packages/e4/6a/96dac197a9c8f3aa82a83a2805bc40ba615613e35a715e70912385670fb7/parameter-sherpa-1.0.5.tar.gz (513kB)
     |████████████████████████████████| 522kB 751kB/s
Requirement already satisfied: pandas>=0.20.3 in /localdata/local/anaconda3/envs/my-conda-env/lib/python3.6/site-packages (from parameter-sherpa) (0.25.1)
Requirement already satisfied: pymongo>=3.5.1 in /localdata/local/anaconda3/envs/my-conda-env/lib/python3.6/site-packages (from parameter-sherpa) (3.9.0)
Requirement already satisfied: numpy>=1.8.2 in /localdata/local/anaconda3/envs/my-conda-env/lib/python3.6/site-packages (from parameter-sherpa) (1.16.5)
Requirement already satisfied: scipy>=1.0.0 in /localdata/local/anaconda3/envs/my-conda-env/lib/python3.6/site-packages (from parameter-sherpa) (1.3.1)
Requirement already satisfied: scikit-learn>=0.19.1 in /localdata/local/anaconda3/envs/my-conda-env/lib/python3.6/site-packages (from parameter-sherpa) (0.21.3)
Collecting flask>=0.12.2 (from parameter-sherpa)
  Downloading https://[PROXY-TO-PYPI]/pypi/pypi-releases/packages/9b/93/628509b8d5dc749656a9641f4caf13540e2cdec85276964ff8f43bbb1d3b/Flask-1.1.1-py2.py3-none-any.whl (94kB)
     |████████████████████████████████| 102kB 989kB/s
Collecting drmaa>=0.7.7 (from parameter-sherpa)
  Downloading https://[PROXY-TO-PYPI]/pypi/pypi-releases/packages/de/3f/3f74887d9ba45c34a9b12c754d064ba2a8f9458d6ab3967915d4337bc6df/drmaa-0.7.9.tar.gz
Requirement already satisfied: python-dateutil>=2.6.1 in /localdata/local/anaconda3/envs/my-conda-env/lib/python3.6/site-packages (from pandas>=0.20.3->parameter-sherpa) (2.8.0)
Requirement already satisfied: pytz>=2017.2 in /localdata/local/anaconda3/envs/my-conda-env/lib/python3.6/site-packages (from pandas>=0.20.3->parameter-sherpa) (2019.2)
Requirement already satisfied: joblib>=0.11 in /localdata/local/anaconda3/envs/my-conda-env/lib/python3.6/site-packages (from scikit-learn>=0.19.1->parameter-sherpa) (0.14.0)
Collecting click>=5.1 (from flask>=0.12.2->parameter-sherpa)
  Downloading https://[PROXY-TO-PYPI]/pypi/pypi-releases/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl (81kB)
     |████████████████████████████████| 81kB 790kB/s
Collecting itsdangerous>=0.24 (from flask>=0.12.2->parameter-sherpa)
  Downloading https://[PROXY-TO-PYPI]/pypi/pypi-releases/packages/76/ae/44b03b253d6fade317f32c24d100b3b35c2239807046a4c953c7b89fa49e/itsdangerous-1.1.0-py2.py3-none-any.whl
Requirement already satisfied: Jinja2>=2.10.1 in /localdata/local/anaconda3/envs/my-conda-env/lib/python3.6/site-packages (from flask>=0.12.2->parameter-sherpa) (2.10.1)
Requirement already satisfied: Werkzeug>=0.15 in /localdata/local/anaconda3/envs/my-conda-env/lib/python3.6/site-packages (from flask>=0.12.2->parameter-sherpa) (0.16.0)
Requirement already satisfied: six>=1.5 in /localdata/local/anaconda3/envs/my-conda-env/lib/python3.6/site-packages (from python-dateutil>=2.6.1->pandas>=0.20.3->parameter-sherpa) (1.12.0)
Requirement already satisfied: MarkupSafe>=0.23 in /localdata/local/anaconda3/envs/my-conda-env/lib/python3.6/site-packages (from Jinja2>=2.10.1->flask>=0.12.2->parameter-sherpa) (1.1.1)
Building wheels for collected packages: parameter-sherpa, drmaa
  Building wheel for parameter-sherpa (setup.py) ... done
  Created wheel for parameter-sherpa: filename=parameter_sherpa-1.0.5-py2.py3-none-any.whl size=542235 sha256=05441754f58915f1c4f3ed058c4e5604a42995d4867a35163e2e9cfcb71042a8
  Stored in directory: /net/ent-home/home/s/sv182c/.cache/pip/wheels/6e/37/de/811d40a6b17c2241f25b5cbfe55cf966ff3fe0b6c5154e3445
  Building wheel for drmaa (setup.py) ... done
  Created wheel for drmaa: filename=drmaa-0.7.9-py2.py3-none-any.whl size=19751 sha256=3db54bb8c6b45aa2060eaf1527350fb50258b21e98c17a7db5b1008105c1e1e2
  Stored in directory: /net/ent-home/home/s/sv182c/.cache/pip/wheels/2e/fb/8c/e4209fe365b3e0ba15f30ebe1aadb7a2a8ad9720a678d1539e
Successfully built parameter-sherpa drmaa
Installing collected packages: click, itsdangerous, flask, drmaa, parameter-sherpa
Successfully installed click-7.0 drmaa-0.7.9 flask-1.1.1 itsdangerous-1.1.0 parameter-sherpa-1.0.5

So it looks like these dependencies were not able to be resolved when installing via pip:

• pymongo
• scikit-learn
• GPyOpt

But these dependencies could be resolved when installing via pip:

• flask (and its sub-dependencies were also recursively resolved)
• drmaa

Lastly, these dependencies are "unknown" as to whether they will be resolved, since my environment happened to already have them:

• numpy
• scipy
• pandas

I realize that the pip installation docs say:

since the source is regularly updated we recommend to clone from GitHub as described below

However, I would like to quickly and easily deploy this to an easily replicatable environment via the pip package manager, and installing via GitHub is not practical.

Is there any way that you could please look into this? (and, of course, the fix would need to be published back to the PyPI)

[LarsHH]

Hi @tpanza ,

Sorry about the complications there. I'm looking into this. For now, this is how Sherpa is being installed on the Travis virtual machine which should work.

git clone --depth=50 --branch=master https://github.com/sherpa-ai/sherpa.git sherpa-ai/sherpa
cd sherpa-ai/sherpa
pip install -r requirements.txt
pip install .

I know this is using GitHub, but it does work in a fully automatic manner.

[tpanza]

I am far from a guru when it comes to Python packaging, but I think there are two issues:

1. Need to add GPyOpt to the REQUIRED list.
2. On line 13 of setup.py, there is an import sherpa. This is used to get the ground truth of what the __version__ is. However, the __init__.py is in turn attempting to import packages that may not yet be installed (because they are dependencies that setup.py may need to install). Some alternative approaches are listed here. The problem we are having here is the warning under number 6 on that list:

Although this technique is common, beware that it will fail if sample/__init__.py imports packages from install_requires dependencies, which will very likely not be installed yet when setup.py is run.

[LarsHH]

Hehe it seems like you're more of a guru than me. What you're saying totally makes sense.

I had not added GPyOpt to the required list at some point because I thought to keep it as an optional dependency but at this point it is probably most sensible to include it..