Closed hongyi-zhao closed 4 years ago
Hi, could you please elaborate on the issue? Is it your experience that it is slow but pip displays otherwise (2 and 14.4 MBps seems, ehm, pretty desirable where I'm from)?
These files are all small files, so the displayed speed cannot be used to judge the real speed. Only for large file this speed can be used as the standards. I put the above logs due to the pip freeze there.
It spends about 5 mins, but only freeze there.
So, I said that.
Regards
Got it! May we have the content of aiida-quantum-mobile
to reproduce? Or alternatively the verbose log of pip
(by adding -v
)? My guess would be purely because of connection problem but that doesn't seem sensible, given the connection is not even international.
After a second try, I find that the ``pip install -r '' will spend much time for analysis the dependencies and I guess maybe this is the trouble maker.
The content of the aiida-quantum-mobile is rather long, you could please give a try on it:
aiida-bands-inspect==0.3.0
aiida-bigdft==0.1.0a0
aiida-castep==1.0.0
aiida-codtools==2.0.0
-e git+https://github.com/aiidateam/aiida-core.git@20fa6c252912d8af04d9926f93a9197d3cc2b651#egg=aiida_core
aiida-cp2k==1.0.0b6
aiida-ddec==1.0.0a1
aiida-diff==1.0.0
aiida-export-migration-tests==0.8.0
aiida-fleur==1.1.0
aiida-optimize==0.3.1
aiida-qeq==1.0.0a1
aiida-quantumespresso==3.0.0a6
aiida-raspa==1.0.0
aiida-siesta==1.0.1
aiida-tbmodels==0.2.0
aiida-tools==0.3.2
aiida-vasp==0.2.4
aiida-wannier90==2.0.1
aiida-wannier90-workflows==1.0.1
aiida-yambo==1.0.0
aiida-zeopp==1.1.1
aiidalab==20.2.0b2
aiidalab-widgets-base==1.0.0b2
alabaster==0.7.12
aldjemy==0.9.1
alembic==1.4.0
aniso8601==8.0.0
appmode-aiidalab==0.5.0.1
arrow==0.15.5
ase==3.19.0
async-generator==1.10
attrs==19.3.0
Babel==2.8.0
backcall==0.1.0
bcrypt==3.1.7
binaryornot==0.4.4
bleach==3.1.1
bokeh==1.4.0
bqplot==0.12.3
bump2version==1.0.0
castepinput==0.1.4
certifi==2019.11.28
cffi==1.14.0
cftime==1.1.0
chainmap==1.0.3
chardet==3.0.4
circus==0.16.1
Click==7.0
click-completion==0.5.2
click-config-file==0.5.0
click-spinner==0.1.8
configobj==5.0.6
cookiecutter==1.6.0
coverage==5.0.3
cryptography==2.8
cycler==0.10.0
dataclasses==0.7
decorator==4.4.1
deepdiff==3.3.0
defusedxml==0.6.0
Django==2.2.10
docutils==0.15.2
dulwich==0.19.15
elementpath==1.1.8
entrypoints==0.3
ete3==3.1.1
fileupload==0.1.5
Flask==1.1.1
Flask-Cors==3.0.8
Flask-RESTful==0.3.8
frozendict==1.2
fsc.export==1.2.0
furl==2.1.0
future==0.18.2
graphviz==0.13.2
h5py==2.10.0
idna==2.9
imagesize==1.2.0
importlib-metadata==1.5.0
ipydatawidgets==4.0.1
ipykernel==5.1.4
ipython==7.12.0
ipython-genutils==0.2.0
ipywidgets==7.5.1
itsdangerous==1.1.0
jedi==0.16.0
Jinja2==2.11.1
jinja2-time==0.2.0
jsonpickle==1.3
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client==6.0.0
jupyter-console==6.1.0
jupyter-core==4.6.3
jupyter-server==0.1.1
jupyterlab-pygments==0.1.0
kiwipy==0.5.3
kiwisolver==1.1.0
lxml==4.5.0
Mako==1.1.1
Markdown==3.1.1
MarkupSafe==1.1.1
masci-tools==0.3.11
matplotlib==3.2.0rc3
mistune==0.8.4
monty==3.0.2
more-itertools==8.2.0
mpmath==1.1.0
nbconvert==5.6.1
nbformat==5.0.4
netCDF4==1.5.3
networkx==2.4
nglview==2.7.1
nose==1.3.7
notebook==5.7.8
numpy==1.17.4
orderedmultidict==1.0.1
packaging==20.1
palettable==3.3.0
pandas==1.0.1
pandocfilters==1.4.2
paramiko==2.7.1
parsevasp==0.4.3
parso==0.6.1
pbr==5.4.4
pexpect==4.8.0
pg8000==1.13.2
pgtest==1.3.2
pickleshare==0.7.5
pika==1.1.0
Pillow==7.0.0
pipdeptree==0.13.2
pluggy==0.13.1
plumpy==0.14.5
poyo==0.5.0
prometheus-client==0.7.1
prompt-toolkit==3.0.3
psutil==5.7.0
psycopg2-binary==2.8.4
ptyprocess==0.6.0
py==1.8.1
PyCifRW==4.4.1
pycparser==2.19
PyDispatcher==2.0.5
Pygments==2.5.2
pymatgen==2020.1.28
PyMySQL==0.9.3
PyNaCl==1.3.0
pyparsing==2.4.6
pyrsistent==0.15.7
pysmiles==1.0.0
pytest==5.3.5
pytest-cov==2.8.1
pytest-mpl==0.11
pytest-timeout==1.3.4
python-dateutil==2.8.1
python-editor==1.0.4
python-memcached==1.59
pythreejs==2.1.1
pytz==2019.3
PyYAML==5.1.2
pyzmq==18.1.1
qe-tools==1.1.3
qtconsole==4.7.1
QtPy==1.9.0
reentry==1.3.1
requests==2.23.0
requests-cache==0.5.2
ruamel.yaml==0.16.10
ruamel.yaml.clib==0.2.0
scipy==1.4.1
scramp==1.1.0
seekpath==1.9.4
Send2Trash==1.5.0
shellingham==1.3.2
shortuuid==0.5.0
simplejson==3.17.0
six==1.14.0
snowballstemmer==2.0.0
spglib==1.14.1.post0
Sphinx==2.4.4
sphinx-rtd-theme==0.4.3
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-contentui==0.2.4
sphinxcontrib-details-directive==0.1.0
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==1.0.3
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.4
SQLAlchemy==1.3.13
sqlalchemy-diff==0.1.3
SQLAlchemy-Utils==0.34.2
sqlparse==0.3.0
subprocess32==3.5.4
sympy==1.5.1
tabulate==0.8.6
terminado==0.8.3
testpath==0.4.4
topika==0.2.1
tornado==4.5.3
traitlets==4.3.3
traittypes==0.2.1
tzlocal==2.1b1
upf-to-json==0.9.2
urllib3==1.25.8
voila==0.1.20
voluptuous==0.11.7
wcwidth==0.1.8
webencodings==0.5.1
Werkzeug==1.0.0
whichcraft==0.6.1
widgetsnbextension==3.5.1
wrapt==1.11.2
xmlschema==1.0.13
zipp==3.0.0
The python version for this environment is 3.6.9. This environment is make by this project:
https://github.com/marvel-nccr/quantum-mobile/releases/
But I failed to re-create this environment.
Regards
If dependency resolution time is an issue, try --no-deps
. The requirements.txt file you provide looks like output from pip freeze
; dependency calculation would be unnecessary if that’s the case.
Unfortunately pip currently cannot avoid dependency resolution automatically since it has no way to tell from the requirements file.
Tried with the following:
$ pip install --no-deps -v -r aiida-quantum-mobile
But it seems the install process will stuck on some small packages for some time, while for large packages, the speed is good but finally failed, see the following:
[...]
Collecting aiida-qeq==1.0.0a1 (from -r aiida-quantum-mobile (line 12))
1 location(s) to search for versions of aiida-qeq:
* https://pypi.tuna.tsinghua.edu.cn/simple/aiida-qeq/
Getting page https://pypi.tuna.tsinghua.edu.cn/simple/aiida-qeq/
https://pypi.tuna.tsinghua.edu.cn:443 "GET /simple/aiida-qeq/ HTTP/1.1" 200 606
Analyzing links from page https://pypi.tuna.tsinghua.edu.cn/simple/aiida-qeq/
Found link https://pypi.tuna.tsinghua.edu.cn/packages/ed/1f/8457e68ecee5faae1b0f4d1cfae68e09dc7743d05d647dd88187c41e659e/aiida-qeq-0.1.0.tar.gz#sha256=8e9b3703122cf77f3df62e14f30eeb625d642f31fff228173ef936cdf55fad6d (from https://pypi.tuna.tsinghua.edu.cn/simple/aiida-qeq/), version: 0.1.0
Skipping link https://pypi.tuna.tsinghua.edu.cn/packages/e9/18/45d5b42dafab82eabb7c9f922d395f13337fffe0f836c0e2c937230662bb/aiida_qeq-0.1.0-py2-none-any.whl#sha256=f47d1ec4e7caed049b81269473da2a945d71ada7c72920e78b8a37fe1822f73b (from https://pypi.tuna.tsinghua.edu.cn/simple/aiida-qeq/); it is not compatible with this Python
Could not find a version that satisfies the requirement aiida-qeq==1.0.0a1 (from -r aiida-quantum-mobile (line 12)) (from versions: 0.1.0)
Cleaning up...
Removing source in /tmp/pip-install-c4ejud5t/aiida-bands-inspect
Removing source in /tmp/pip-install-c4ejud5t/aiida-castep
Removing source in /tmp/pip-install-c4ejud5t/aiida-cp2k
Removing source in /tmp/pip-install-c4ejud5t/aiida-ddec
Removing source in /tmp/pip-install-c4ejud5t/aiida-optimize
Removed build tracker '/tmp/pip-req-tracker-hsqoboib'
No matching distribution found for aiida-qeq==1.0.0a1 (from -r aiida-quantum-mobile (line 12))
Exception information:
Traceback (most recent call last):
File "/home/werner/.pyenv/versions/3.6.9/envs/test/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 143, in main
status = self.run(options, args)
File "/home/werner/.pyenv/versions/3.6.9/envs/test/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 318, in run
resolver.resolve(requirement_set)
File "/home/werner/.pyenv/versions/3.6.9/envs/test/lib/python3.6/site-packages/pip/_internal/resolve.py", line 102, in resolve
self._resolve_one(requirement_set, req)
File "/home/werner/.pyenv/versions/3.6.9/envs/test/lib/python3.6/site-packages/pip/_internal/resolve.py", line 256, in _resolve_one
abstract_dist = self._get_abstract_dist_for(req_to_install)
File "/home/werner/.pyenv/versions/3.6.9/envs/test/lib/python3.6/site-packages/pip/_internal/resolve.py", line 209, in _get_abstract_dist_for
self.require_hashes
File "/home/werner/.pyenv/versions/3.6.9/envs/test/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 218, in prepare_linked_requirement
req.populate_link(finder, upgrade_allowed, require_hashes)
File "/home/werner/.pyenv/versions/3.6.9/envs/test/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 164, in populate_link
self.link = finder.find_requirement(self, upgrade)
File "/home/werner/.pyenv/versions/3.6.9/envs/test/lib/python3.6/site-packages/pip/_internal/index.py", line 621, in find_requirement
'No matching distribution found for %s' % req
pip._internal.exceptions.DistributionNotFound: No matching distribution found for aiida-qeq==1.0.0a1 (from -r aiida-quantum-mobile (line 12))
You are using pip version 18.1, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Based on the above info, I tried with the following steps:
$ pip install --upgrade pip
$ pip install --no-deps -v -r aiida-quantum-mobile
It seems for large packages, the process is very slow, see the following for more info:
Added pymatgen==2020.1.28 from https://pypi.tuna.tsinghua.edu.cn/packages/cf/50/bcec98192a865cf9b56a746f3f383ba6be2b7c275f0037226fccc44d7b00/pymatgen-2020.1.28.tar.gz#sha256=886eb915f3efd60ab0295202cdbe0802f1632187b60517d466917211e02d5d7d (from -r aiida-quantum-mobile (line 147)) to build tracker '/tmp/pip-req-tracker-2e4wcokr'
Running setup.py (path:/tmp/pip-install-6bbv0fxb/pymatgen/setup.py) egg_info for package pymatgen
Running command python setup.py egg_info
But finally, this method let me successfully done the trick.
Regards
In the cases you point out, pip is slow because it is trying to build the package from source. There is no way to avoid the build. If you want installation to be faster, pre-build the packages into wheels (.whl
files) with pip wheel
, put the wheel files in a local directory, and use --find-links
to tell pip where the directory is.
Thanks for your explanations.
Regards
Hi,
I'm from China, and I use https://pypi.tuna.tsinghua.edu.cn as the pypi mirror to duplicate a environment using the requirments file.
But I see the very slow speed here:
Any hints for this problem?
Regards