search

pypa / pip

The Python package installer
https://pip.pypa.io/
MIT License
9.47k stars 3.01k forks source link

``pip install -r '' gives a very slow speed. #8021

Closed hongyi-zhao closed 4 years ago

hongyi-zhao commented 4 years ago

Hi,

I'm from China, and I use https://pypi.tuna.tsinghua.edu.cn as the pypi mirror to duplicate a environment using the requirments file.

But I see the very slow speed here:

$ pip install -r aiida-quantum-mobile 
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting aiida-bands-inspect==0.3.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/42/10/41ab7a9ad7c4c46a823815e75251cbdeaccd1e8dbdb69d9259ca1166ce02/aiida-bands-inspect-0.3.0.tar.gz (9.7 kB)
Collecting aiida-bigdft==0.1.0a0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/1b/8d/4b1954ac615b24afee5354e5622f8d2b4e4df0f70cae00b33801ed202c82/aiida_bigdft-0.1.0a0-py3-none-any.whl (192 kB)
     |████████████████████████████████| 192 kB 2.0 MB/s 
Collecting aiida-castep==1.0.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a6/aa/933551918904f342e11967b59f9ffe0d0c23f481fd79fd228bf883ea65d6/aiida-castep-1.0.0.tar.gz (253 kB)
     |████████████████████████████████| 253 kB 14.4 MB/s 
Collecting aiida-codtools==2.0.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/87/60/4315851c8dd6e983cc3d5a30f4e731543249c982ac29ab78930d90d172a9/aiida_codtools-2.0.0-py3-none-any.whl (38 kB)
Obtaining aiida_core from git+https://github.com/aiidateam/aiida-core.git@20fa6c252912d8af04d9926f93a9197d3cc2b651#egg=aiida_core (from -r aiida-quantum-mobile (line 5))
  Cloning https://github.com/aiidateam/aiida-core.git (to revision 20fa6c252912d8af04d9926f93a9197d3cc2b651) to ./.pyenv/versions/3.6.9/envs/aiida-quantum-mobile/src/aiida-core
  Running command git clone -q https://github.com/aiidateam/aiida-core.git /home/werner/.pyenv/versions/3.6.9/envs/aiida-quantum-mobile/src/aiida-core
  Running command git checkout -q 20fa6c252912d8af04d9926f93a9197d3cc2b651
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting aiida-cp2k==1.0.0b6
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/f6/d9/29410785d307cf4027c50bda95a5daa50a6651f24cc2954e7807d30a9db4/aiida_cp2k-1.0.0b6.tar.gz (33 kB)
Collecting aiida-ddec==1.0.0a1
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/07/c0/c8ea9ef2fe71fa22b864a5453c27bacf60d8711c090d6ac8648e1d196c40/aiida-ddec-1.0.0a1.tar.gz (8.2 kB)

Any hints for this problem?

Regards

McSinyx commented 4 years ago

Hi, could you please elaborate on the issue? Is it your experience that it is slow but pip displays otherwise (2 and 14.4 MBps seems, ehm, pretty desirable where I'm from)?

hongyi-zhao commented 4 years ago

These files are all small files, so the displayed speed cannot be used to judge the real speed. Only for large file this speed can be used as the standards. I put the above logs due to the pip freeze there.

It spends about 5 mins, but only freeze there.

So, I said that.

Regards

McSinyx commented 4 years ago

Got it! May we have the content of aiida-quantum-mobile to reproduce? Or alternatively the verbose log of pip (by adding -v)? My guess would be purely because of connection problem but that doesn't seem sensible, given the connection is not even international.

hongyi-zhao commented 4 years ago

After a second try, I find that the ``pip install -r '' will spend much time for analysis the dependencies and I guess maybe this is the trouble maker.

The content of the aiida-quantum-mobile is rather long, you could please give a try on it:

aiida-bands-inspect==0.3.0
aiida-bigdft==0.1.0a0
aiida-castep==1.0.0
aiida-codtools==2.0.0
-e git+https://github.com/aiidateam/aiida-core.git@20fa6c252912d8af04d9926f93a9197d3cc2b651#egg=aiida_core
aiida-cp2k==1.0.0b6
aiida-ddec==1.0.0a1
aiida-diff==1.0.0
aiida-export-migration-tests==0.8.0
aiida-fleur==1.1.0
aiida-optimize==0.3.1
aiida-qeq==1.0.0a1
aiida-quantumespresso==3.0.0a6
aiida-raspa==1.0.0
aiida-siesta==1.0.1
aiida-tbmodels==0.2.0
aiida-tools==0.3.2
aiida-vasp==0.2.4
aiida-wannier90==2.0.1
aiida-wannier90-workflows==1.0.1
aiida-yambo==1.0.0
aiida-zeopp==1.1.1
aiidalab==20.2.0b2
aiidalab-widgets-base==1.0.0b2
alabaster==0.7.12
aldjemy==0.9.1
alembic==1.4.0
aniso8601==8.0.0
appmode-aiidalab==0.5.0.1
arrow==0.15.5
ase==3.19.0
async-generator==1.10
attrs==19.3.0
Babel==2.8.0
backcall==0.1.0
bcrypt==3.1.7
binaryornot==0.4.4
bleach==3.1.1
bokeh==1.4.0
bqplot==0.12.3
bump2version==1.0.0
castepinput==0.1.4
certifi==2019.11.28
cffi==1.14.0
cftime==1.1.0
chainmap==1.0.3
chardet==3.0.4
circus==0.16.1
Click==7.0
click-completion==0.5.2
click-config-file==0.5.0
click-spinner==0.1.8
configobj==5.0.6
cookiecutter==1.6.0
coverage==5.0.3
cryptography==2.8
cycler==0.10.0
dataclasses==0.7
decorator==4.4.1
deepdiff==3.3.0
defusedxml==0.6.0
Django==2.2.10
docutils==0.15.2
dulwich==0.19.15
elementpath==1.1.8
entrypoints==0.3
ete3==3.1.1
fileupload==0.1.5
Flask==1.1.1
Flask-Cors==3.0.8
Flask-RESTful==0.3.8
frozendict==1.2
fsc.export==1.2.0
furl==2.1.0
future==0.18.2
graphviz==0.13.2
h5py==2.10.0
idna==2.9
imagesize==1.2.0
importlib-metadata==1.5.0
ipydatawidgets==4.0.1
ipykernel==5.1.4
ipython==7.12.0
ipython-genutils==0.2.0
ipywidgets==7.5.1
itsdangerous==1.1.0
jedi==0.16.0
Jinja2==2.11.1
jinja2-time==0.2.0
jsonpickle==1.3
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client==6.0.0
jupyter-console==6.1.0
jupyter-core==4.6.3
jupyter-server==0.1.1
jupyterlab-pygments==0.1.0
kiwipy==0.5.3
kiwisolver==1.1.0
lxml==4.5.0
Mako==1.1.1
Markdown==3.1.1
MarkupSafe==1.1.1
masci-tools==0.3.11
matplotlib==3.2.0rc3
mistune==0.8.4
monty==3.0.2
more-itertools==8.2.0
mpmath==1.1.0
nbconvert==5.6.1
nbformat==5.0.4
netCDF4==1.5.3
networkx==2.4
nglview==2.7.1
nose==1.3.7
notebook==5.7.8
numpy==1.17.4
orderedmultidict==1.0.1
packaging==20.1
palettable==3.3.0
pandas==1.0.1
pandocfilters==1.4.2
paramiko==2.7.1
parsevasp==0.4.3
parso==0.6.1
pbr==5.4.4
pexpect==4.8.0
pg8000==1.13.2
pgtest==1.3.2
pickleshare==0.7.5
pika==1.1.0
Pillow==7.0.0
pipdeptree==0.13.2
pluggy==0.13.1
plumpy==0.14.5
poyo==0.5.0
prometheus-client==0.7.1
prompt-toolkit==3.0.3
psutil==5.7.0
psycopg2-binary==2.8.4
ptyprocess==0.6.0
py==1.8.1
PyCifRW==4.4.1
pycparser==2.19
PyDispatcher==2.0.5
Pygments==2.5.2
pymatgen==2020.1.28
PyMySQL==0.9.3
PyNaCl==1.3.0
pyparsing==2.4.6
pyrsistent==0.15.7
pysmiles==1.0.0
pytest==5.3.5
pytest-cov==2.8.1
pytest-mpl==0.11
pytest-timeout==1.3.4
python-dateutil==2.8.1
python-editor==1.0.4
python-memcached==1.59
pythreejs==2.1.1
pytz==2019.3
PyYAML==5.1.2
pyzmq==18.1.1
qe-tools==1.1.3
qtconsole==4.7.1
QtPy==1.9.0
reentry==1.3.1
requests==2.23.0
requests-cache==0.5.2
ruamel.yaml==0.16.10
ruamel.yaml.clib==0.2.0
scipy==1.4.1
scramp==1.1.0
seekpath==1.9.4
Send2Trash==1.5.0
shellingham==1.3.2
shortuuid==0.5.0
simplejson==3.17.0
six==1.14.0
snowballstemmer==2.0.0
spglib==1.14.1.post0
Sphinx==2.4.4
sphinx-rtd-theme==0.4.3
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-contentui==0.2.4
sphinxcontrib-details-directive==0.1.0
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==1.0.3
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.4
SQLAlchemy==1.3.13
sqlalchemy-diff==0.1.3
SQLAlchemy-Utils==0.34.2
sqlparse==0.3.0
subprocess32==3.5.4
sympy==1.5.1
tabulate==0.8.6
terminado==0.8.3
testpath==0.4.4
topika==0.2.1
tornado==4.5.3
traitlets==4.3.3
traittypes==0.2.1
tzlocal==2.1b1
upf-to-json==0.9.2
urllib3==1.25.8
voila==0.1.20
voluptuous==0.11.7
wcwidth==0.1.8
webencodings==0.5.1
Werkzeug==1.0.0
whichcraft==0.6.1
widgetsnbextension==3.5.1
wrapt==1.11.2
xmlschema==1.0.13
zipp==3.0.0

The python version for this environment is 3.6.9. This environment is make by this project:

https://github.com/marvel-nccr/quantum-mobile/releases/

But I failed to re-create this environment.

Regards

uranusjr commented 4 years ago

If dependency resolution time is an issue, try --no-deps. The requirements.txt file you provide looks like output from pip freeze; dependency calculation would be unnecessary if that’s the case.

Unfortunately pip currently cannot avoid dependency resolution automatically since it has no way to tell from the requirements file.

hongyi-zhao commented 4 years ago

Tried with the following: $ pip install --no-deps -v -r aiida-quantum-mobile

But it seems the install process will stuck on some small packages for some time, while for large packages, the speed is good but finally failed, see the following:

[...]

Collecting aiida-qeq==1.0.0a1 (from -r aiida-quantum-mobile (line 12))
  1 location(s) to search for versions of aiida-qeq:
  * https://pypi.tuna.tsinghua.edu.cn/simple/aiida-qeq/
  Getting page https://pypi.tuna.tsinghua.edu.cn/simple/aiida-qeq/
  https://pypi.tuna.tsinghua.edu.cn:443 "GET /simple/aiida-qeq/ HTTP/1.1" 200 606
  Analyzing links from page https://pypi.tuna.tsinghua.edu.cn/simple/aiida-qeq/
    Found link https://pypi.tuna.tsinghua.edu.cn/packages/ed/1f/8457e68ecee5faae1b0f4d1cfae68e09dc7743d05d647dd88187c41e659e/aiida-qeq-0.1.0.tar.gz#sha256=8e9b3703122cf77f3df62e14f30eeb625d642f31fff228173ef936cdf55fad6d (from https://pypi.tuna.tsinghua.edu.cn/simple/aiida-qeq/), version: 0.1.0
    Skipping link https://pypi.tuna.tsinghua.edu.cn/packages/e9/18/45d5b42dafab82eabb7c9f922d395f13337fffe0f836c0e2c937230662bb/aiida_qeq-0.1.0-py2-none-any.whl#sha256=f47d1ec4e7caed049b81269473da2a945d71ada7c72920e78b8a37fe1822f73b (from https://pypi.tuna.tsinghua.edu.cn/simple/aiida-qeq/); it is not compatible with this Python
  Could not find a version that satisfies the requirement aiida-qeq==1.0.0a1 (from -r aiida-quantum-mobile (line 12)) (from versions: 0.1.0)
Cleaning up...
  Removing source in /tmp/pip-install-c4ejud5t/aiida-bands-inspect
  Removing source in /tmp/pip-install-c4ejud5t/aiida-castep
  Removing source in /tmp/pip-install-c4ejud5t/aiida-cp2k
  Removing source in /tmp/pip-install-c4ejud5t/aiida-ddec
  Removing source in /tmp/pip-install-c4ejud5t/aiida-optimize
Removed build tracker '/tmp/pip-req-tracker-hsqoboib'
No matching distribution found for aiida-qeq==1.0.0a1 (from -r aiida-quantum-mobile (line 12))
Exception information:
Traceback (most recent call last):
  File "/home/werner/.pyenv/versions/3.6.9/envs/test/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 143, in main
    status = self.run(options, args)
  File "/home/werner/.pyenv/versions/3.6.9/envs/test/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 318, in run
    resolver.resolve(requirement_set)
  File "/home/werner/.pyenv/versions/3.6.9/envs/test/lib/python3.6/site-packages/pip/_internal/resolve.py", line 102, in resolve
    self._resolve_one(requirement_set, req)
  File "/home/werner/.pyenv/versions/3.6.9/envs/test/lib/python3.6/site-packages/pip/_internal/resolve.py", line 256, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/home/werner/.pyenv/versions/3.6.9/envs/test/lib/python3.6/site-packages/pip/_internal/resolve.py", line 209, in _get_abstract_dist_for
    self.require_hashes
  File "/home/werner/.pyenv/versions/3.6.9/envs/test/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 218, in prepare_linked_requirement
    req.populate_link(finder, upgrade_allowed, require_hashes)
  File "/home/werner/.pyenv/versions/3.6.9/envs/test/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 164, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/home/werner/.pyenv/versions/3.6.9/envs/test/lib/python3.6/site-packages/pip/_internal/index.py", line 621, in find_requirement
    'No matching distribution found for %s' % req
pip._internal.exceptions.DistributionNotFound: No matching distribution found for aiida-qeq==1.0.0a1 (from -r aiida-quantum-mobile (line 12))
You are using pip version 18.1, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
hongyi-zhao commented 4 years ago

Based on the above info, I tried with the following steps:


$ pip install --upgrade pip
$ pip install --no-deps -v -r aiida-quantum-mobile

It seems for large packages, the process is very slow, see the following for more info:

  Added pymatgen==2020.1.28 from https://pypi.tuna.tsinghua.edu.cn/packages/cf/50/bcec98192a865cf9b56a746f3f383ba6be2b7c275f0037226fccc44d7b00/pymatgen-2020.1.28.tar.gz#sha256=886eb915f3efd60ab0295202cdbe0802f1632187b60517d466917211e02d5d7d (from -r aiida-quantum-mobile (line 147)) to build tracker '/tmp/pip-req-tracker-2e4wcokr'
    Running setup.py (path:/tmp/pip-install-6bbv0fxb/pymatgen/setup.py) egg_info for package pymatgen
    Running command python setup.py egg_info

But finally, this method let me successfully done the trick.

Regards

uranusjr commented 4 years ago

In the cases you point out, pip is slow because it is trying to build the package from source. There is no way to avoid the build. If you want installation to be faster, pre-build the packages into wheels (.whl files) with pip wheel, put the wheel files in a local directory, and use --find-links to tell pip where the directory is.

hongyi-zhao commented 4 years ago

Thanks for your explanations.

Regards