search

pypa / pipenv

Python Development Workflow for Humans.
https://pipenv.pypa.io
MIT License
24.87k stars 1.87k forks source link

pipenv install is very slow #356

Closed Diggsey closed 7 years ago

Diggsey commented 7 years ago

Running pipenv install after changing one dependency takes about ~5 minutes for me, on a windows 10 machine with an SSD.

The vast majority of that time is spent inside Locking [packages] dependencies...

It seems like there might be some quadratic-or-worse complexity in this step?

I've included most of our pipfile below, but I had to remove some of our private repo dependencies:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[packages]
alembic = "==0.8.4"
amqp = "==1.4.7"
analytics-python = "==1.2.5"
anyjson = "==0.3.3"
billiard = "==3.3.0.20"
braintree = "==3.20.0"
celery = "==3.1.18"
coverage = "==4.0.3"
docopt = "==0.4.0"
eventlet = "==0.19.0"
flake8 = "==3.0.4"
Flask-Cors = "==2.1.2"
Flask-Login = "==0.3.2"
Flask = "==0.12.1"
funcsigs = "==0.4"
fuzzywuzzy = "==0.12.0"
gcloud = "==0.14.0"
html2text = "==2016.9.19"
itsdangerous = "==0.24"
Jinja2 = "==2.8"
jsonpatch = "==1.15"
jsonschema = "==2.5.1"
PyJWT = "==1.4.2"
kombu = "==3.0.30"
LayerClient = "==0.1.9"
MarkupSafe = "==0.23"
mixpanel = "==4.3.0"
mock = "==1.3.0"
nose-exclude = "==0.4.1"
nose = "==1.3.7"
numpy = "==1.12.1"
pdfrw = "==0.3"
Pillow = "==4.1.0"
pusher = "==1.6"
pycountry = "==1.20"
pycryptodome = "==3.4.5"
pymongo = "==3.2"
PyMySQL = "==0.7.4"
python-dateutil = "<=2.5.1"
python-Levenshtein = "==0.12.0"
python-magic = "==0.4.6"
python-coveralls = "==2.9.0"
pytz = "==2015.6"
raygun4py = "==3.1.2"
"repoze.retry" = "==1.3"
requests = "==2.8.1"
sendgrid = "==2.2.1"
slacker = "==0.7.3"
SQLAlchemy-Enum34 = "==1.0.1"
SQLAlchemy-Utils = "==0.31.6"
SQLAlchemy = "==1.1.9"
typing = "==3.5.2.2"
twilio = "==5.6.0"
Unidecode = "==0.4.19"
voluptuous = "==0.8.11"
Wand = "==0.4.4"
watchdog = "==0.8.3"
Werkzeug = "==0.12.1"
wheel = "==0.24.0"
WTForms = "==2.0.2"
xmltodict = "==0.9.2"
zeep = "==0.24.0"
nateprewitt commented 7 years ago

Hey again @Diggsey, this is due to the way we're writing changes right now. I have these changes ready to merge but they're breaking for the projects.py API so we're going to hold off until the next major release. Hopefully we'll have this up and ready in the next few weeks. I'll leave this open to track the issue for now.

kennethreitz commented 7 years ago

We sprinted on this together at PyCon. It'll be faster soon.

NicolasWebDev commented 7 years ago

Right now for me it is not slow, it is freezing...

A pipenv install my_package or a simple pipenv install does not give me any output, after 20 minutes.

EDIT: Confirmation, still nothing after a few hours. Is it the same problem? Usually it was slow, but it ended after 5 to 10 minutes.

nateprewitt commented 7 years ago

Hey @NicolasWebDev, what version of pipenv are you using? Also do you have delegator.py installed on your system separately? If so, what version is that at? This was an issue that should have been resolved in v3.6.0.

If everything above is up to date, could you please provide the contents of your Pipfile? Thanks!

NicolasWebDev commented 7 years ago

Hi @nateprewitt, you were right, I was on v3.5.x. Updating to 4.1.1 resolved the freezing issue. Now it is still taking 5 minutes, but at least it is usable!

Sorry for the noise, I always forget that packages installed through pip are not automatically updated. Thanks!

nateprewitt commented 7 years ago

Glad you got things resolve @NicolasWebDev! We're working on getting this sped up more, hopefully #373 will be a step closer in the next release.

nateprewitt commented 7 years ago

@Diggsey @NicolasWebDev, I've just released 4.1.2 which should have these speed improvements added. There's still some work to do here, but this will at least expedite the initial bootstrap time for pipenv.

Diggsey commented 7 years ago

@nateprewitt Thanks for the update, pipenv does seem faster for me now, but it still takes several minutes just to run pipenv lock, even when nothing has changed - it still waits in Locking [packages] dependencies... for the vast majority of that time.

nateprewitt commented 7 years ago

@Diggsey, a lot of that time is because you're downloading a huge number of files in that Pipfile. It seems like you're pinning all of your dependencies too. Are you directly importing all of these into your project or are some dependency requirements of the others?

Diggsey commented 7 years ago

@nateprewitt We could remove some of them, but the majority are direct dependencies - why does it need to download all of the dependencies every time it generates the lock file?

nateprewitt commented 7 years ago

We need to determine everything that it installs as dependencies. In order to get that, we download each package and determine what an installation looks like at lock time. This allows us to appropriately pin everything in the Pipfile.lock. Without downloading, there isn't a reliable way of checking sub-dependencies and resolving range dependency declarations.

Diggsey commented 7 years ago

Given that most of the packages are going to stay the same over time, would it be possible to cache the downloaded packages?

kennethreitz commented 7 years ago

@Diggsey want to look into that for us?

uranusjr commented 7 years ago

This might be a silly question, but doesn’t Pip already do package caching?

kennethreitz commented 7 years ago

I'm under the impression that it does.

marctc commented 7 years ago

Can pipenv use the pip cache system or it has to be implemented from the scratch?

kennethreitz commented 7 years ago

Pipenv just runs pip, so the cache should automatically be used.

kennethreitz commented 7 years ago

fixed! lock is wicked fast now.

tilgovi commented 7 years ago

Oh, thank you. I think that was the only thing holding my back from pushing everyone over to pipenv at work.

Diggsey commented 7 years ago

Wow, nice, that was literally more than a 100x speedup for me, and it also caught a dependency conflict that the previous version didn't catch!

What would be useful is a verbose flag for pipenv lock - I was only able to diagnose the dependency conflict by editing piptools/logging.py to enable verbose logging, but once I did that it gave a very clear indication of what was going on.

tricoder42 commented 7 years ago

I'm probably missing something :) Where is it fixed? Latest release is 4 days ago, so I installed latest version from master. However, pipenv install is still slow.

I tried:

Using latest stable version (5.3.5.), it takes 3:40 to install one package:

∙ time pipenv install --dev raven
Installing raven...
Collecting raven
  Using cached raven-6.1.0-py2.py3-none-any.whl
Collecting contextlib2 (from raven)
  Using cached contextlib2-0.5.5-py2.py3-none-any.whl
Installing collected packages: contextlib2, raven
Successfully installed contextlib2-0.5.5 raven-6.1.0

Adding raven to Pipfile's [dev-packages]...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Updated Pipfile.lock!
pipenv install --dev raven  10,11s user 2,77s system 5% cpu 3:40,04 total

Using version from master, it still hangs (one package, +10 minutes)

EDIT: It has just finished:

pipenv install graphene_django  8,03s user 1,28s system 1% cpu 11:23,11 total

Any ideas? Thanks a lot!

kennethreitz commented 7 years ago

Sometimes dependencies take a while to install, especially if they have c compilations. Want to share your Pipfile?

tricoder42 commented 7 years ago

I understand sometimes it takes a while, but it was slow from the beginning. Just curious if it's a problem at my side.

Here's my Pipfile:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[dev-packages]
pytest = "*"
pytest-django = "*"
pytest-testmon = "*"
pytest-watch = "*"
django-debug-toolbar = "*"
raven = "*"

[packages]
dj-database-url = "*"
Django = "*"
djangorestframework = "*"
gunicorn = "*"
newrelic = "*"
psycopg2 = "*"
requests = "*"
whitenoise = "*"
graphene-django = "*"
kennethreitz commented 7 years ago

psycopg2 might be taking a while, as it may be compiling from source. Everything else should be nice and fast. Try removing it and see how much your speed increases.

kennethreitz commented 7 years ago

$ pipenv install raven just took like 1s for me.

tricoder42 commented 7 years ago

$ pipenv install raven just took like 1s for me.

That's what I would expect! Can I turn on verbose output somehow?

I tried remove psycopg2, but it doesn't affect much. Running pipenv install raven hangs for a while.

I have:

kennethreitz commented 7 years ago

i see no reason why raven shouldn't be instantaneous.

do $ pip install raven inside of $ pipenv shell and tell me if it's slow there too.

kennethreitz commented 7 years ago

All pipenv is doing is running pip, so that's effectively the "verbose mode"

tricoder42 commented 7 years ago

That's instant:

∙ time pip install raven                                                                                                                                 19:38  tricoder@issac
Requirement already satisfied: raven in /Users/tricoder/.envs/lingui-api-VE1OToiy/lib/python3.6/site-packages
Requirement already satisfied: contextlib2 in /Users/tricoder/.envs/lingui-api-VE1OToiy/lib/python3.6/site-packages (from raven)
noglob pip install raven  0,54s user 0,15s system 76% cpu 0,900 total

Running pipenv hangs about 2-3 minutes (inside/outside pipenv shell).

∙ time pipenv install raven                                                                                                                              19:39  tricoder@issac
Installing raven...
Requirement already satisfied: raven in /Users/tricoder/.envs/lingui-api-VE1OToiy/lib/python3.6/site-packages
Requirement already satisfied: contextlib2 in /Users/tricoder/.envs/lingui-api-VE1OToiy/lib/python3.6/site-packages (from raven)

Adding raven to Pipfile's [packages]...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Updated Pipfile.lock!
pipenv install raven  4,49s user 0,46s system 2% cpu 3:21,17 total
kennethreitz commented 7 years ago

@Diggsey can you open a new issue about --verbose and provide an example of what you'd like it to look like?

kennethreitz commented 7 years ago

@tricoder42 is the slow part the locking step or the install step? Locking was vastly improved in the latest releases.

kennethreitz commented 7 years ago 

$ time pipenv install raven
Installing raven...
Collecting raven
  Using cached raven-6.1.0-py2.py3-none-any.whl
Requirement already satisfied: contextlib2 in /Users/kennethreitz/.local/share/virtualenvs/pipenv-u9yqWeFK/lib/python3.6/site-packages (from raven)
Installing collected packages: raven
Successfully installed raven-6.1.0

Adding raven to Pipfile's [packages]...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Updated Pipfile.lock!
        9.30 real         5.49 user         0.42 sys
tricoder42 commented 7 years ago

It's like 50:50 installing:locking

kennethreitz commented 7 years ago

@tricoder42 and you're using latest?

kennethreitz commented 7 years ago

Let me replicate with your exact pipfile.

tricoder42 commented 7 years ago

I guess so:

∙ pipenv --version                                                                                                                                       19:42  tricoder@issac
pipenv, version 5.3.5
∙ pipsi upgrade git+https://github.com/kennethreitz/pipenv.git#egg=pipenv                                                                                19:45  tricoder@issac
Collecting pipenv from git+https://github.com/kennethreitz/pipenv.git#egg=pipenv
  Cloning https://github.com/kennethreitz/pipenv.git to /private/var/folders/g9/1wbckv154mbby3tm411z_m340000gn/T/pip-build-se4ao5/pipenv
...
Installing collected packages: pipenv
  Found existing installation: pipenv 5.3.5
    Uninstalling pipenv-5.3.5:
      Successfully uninstalled pipenv-5.3.5
  Running setup.py install for pipenv ... done
Successfully installed pipenv-5.3.5
kennethreitz commented 7 years ago 

$ time pipenv install
No package provided, installing all dependencies.
Pipfile found at /Users/kennethreitz/pipenv/testapp/Pipfile. Considering this to be the project home.
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Updated Pipfile.lock!
Installing dependencies from Pipfile.lock...
[================================] 22/22 - 00:00:37
       58.94 real        40.51 user         8.62 sys
tricoder42 commented 7 years ago

It does even when I'm installing first package into fresh new pipenv. I'll try to create pipenv --three instead of pipenv --python python3.6

kennethreitz commented 7 years ago

@tricoder42 want to hop on a google hangouts in like an hour?

kennethreitz commented 7 years ago

Or we can use Apple screen sharing if you use Messages.app.

Add me! I'm me@kennethreitz.org.

kennethreitz commented 7 years ago

I'm about to join a meeting, but I'll be available after that.

tricoder42 commented 7 years ago

Cool! I'll try clean & reinstall everything from the scratch and we'll see. I'm available in one hour

kennethreitz commented 7 years ago

Awesome — we'll figure this out then. Add me on messages.app :)

ryantuck commented 6 years ago

If anyone is experiencing extremely slow Locking behavior with v11.9.0, i found that downgrading to v9.0.0 takes a 5m30s install down to 1m36s.

techalchemy commented 6 years ago

@ryantuck I'd recommend if you're going to pin an old version to pin 9.0.3 but you lose a significant amount of added security for the speed, you might as well just use --skip-lock at that point

ryantuck commented 6 years ago

--skip-lock definitely sped things up. I went down this path because pipenv install --system --python=3.6 wasn't actually installing to the correct system python, and i was under the assumption that i needed to generate the Pipfile.lock before attempting the system install. It still didn't work, so I went back to using regular old pip, but will comment back on this thread if I ever make any progress along these lines.

techalchemy commented 6 years ago

—system and —python are mutually exclusive — the latter option always needs a virtualenv

kavdev commented 6 years ago

yep, locking takes a while for me too. v11.10.0. Ubuntu on WSL.

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
babel = "==2.5.3"
"boto3" = "==1.7.3"
colorama = "==0.3.9"
coreapi = "==2.3.3"
dj-database-url = "==0.5.0"
djangorestframework = "==3.7.7"
django-axes = "==4.0.2"
django-clever-selects = "==0.8.2"
django-crispy-forms = "==1.7.2"
django-choices = "==1.6.0"
django-extra-views = "==0.10.0"
django-filter = "==1.1.0"
django-hijack = "==2.1.7"
django-hijack-admin = "==2.1.7"
django-js-reverse = "==0.8.1"
django-model-utils = "==3.1.1"
django-phonenumber-field = "==2.0.0"
django-polymorphic = "==2.0.2"
django-redis-cache = "==1.7.1"
django-role-permissions = "==2.2.0"
"django-s3direct" = "==1.0.4"
django-static-precompiler = {extras = ["libsass"], version = "==1.8.2"}
django-storages = "==1.6.6"
"django-tables2" = "==1.21.2"
django-webpack-loader = "==0.6.0"
django-widget-tweaks = "==1.4.2"
facebookads = "==2.11.4"
googleads = "==11.0.1"
markdown = "==2.6.11"
phonenumbers = "==8.9.3"
pillow = "==5.1.0"
"psycopg2-binary" = "==2.7.4"
pygments = "==2.2.0"
pyssim = "==0.4"
python-dotenv = "==0.8.2"
pytz = "==2018.4"
raven = "==6.6.0"
sendgrid-django = "==4.2.0"
slacker = "==0.9.65"
termcolor = "==1.1.0"
tqdm = "==4.21.0"
twitter-ads = "==3.0.0"
brotlipy = "==0.7.0"
waitress = "==1.1.0"
whitenoise = "==3.3.1"
Django = "==2.0.4"

[dev-packages]
coverage = "==4.5.1"
selenium = "==3.11.0"
tblib = "==1.3.2"
"flake8" = "==3.5.0"
django-debug-toolbar = "==1.9.1"
django-extensions = "==2.0.6"

[requires]
python_version = "3.6"
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (7a535c)!
Installing dependencies from Pipfile.lock (7a535c)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 92/92 — 00:01:01

real    8m1.993s
user    5m32.406s
sys     7m15.203s
jtratner commented 6 years ago

It should be a little bit faster the second or third time because of caching tho. Do you see an improvement? On Thu, Apr 12, 2018 at 10:23 AM Alexander Kavanaugh < notifications@github.com> wrote:

yep, locking takes a while for me too. v11.10.0. Ubuntu on WSL.

[[source]]url = "https://pypi.python.org/simple"verify_ssl = truename = "pypi" [packages]babel = "==2.5.3""boto3" = "==1.7.3"colorama = "==0.3.9"coreapi = "==2.3.3"dj-database-url = "==0.5.0"djangorestframework = "==3.7.7"django-axes = "==4.0.2"django-clever-selects = "==0.8.2"django-crispy-forms = "==1.7.2"django-choices = "==1.6.0"django-extra-views = "==0.10.0"django-filter = "==1.1.0"django-hijack = "==2.1.7"django-hijack-admin = "==2.1.7"django-js-reverse = "==0.8.1"django-model-utils = "==3.1.1"django-phonenumber-field = "==2.0.0"django-polymorphic = "==2.0.2"django-redis-cache = "==1.7.1"django-role-permissions = "==2.2.0""django-s3direct" = "==1.0.4"django-static-precompiler = {extras = ["libsass"], version = "==1.8.2"}django-storages = "==1.6.6""django-tables2" = "==1.21.2"django-webpack-loader = "==0.6.0"django-widget-tweaks = "==1.4.2"facebookads = "==2.11.4"googleads = "==11.0.1"markdown = "==2.6.11"phonenumbers = "==8.9.3"pillow = "==5.1.0""psycopg2-binary" = "==2.7.4"pygments = "==2.2.0"pyssim = "==0.4"python-dotenv = "==0.8.2"pytz = "==2018.4"raven = "==6.6.0"sendgrid-django = "==4.2.0"slacker = "==0.9.65"termcolor = "==1.1.0"tqdm = "==4.21.0"twitter-ads = "==3.0.0"brotlipy = "==0.7.0"waitress = "==1.1.0"whitenoise = "==3.3.1"Django = "==2.0.4" [dev-packages]coverage = "==4.5.1"selenium = "==3.11.0"tblib = "==1.3.2""flake8" = "==3.5.0"django-debug-toolbar = "==1.9.1"django-extensions = "==2.0.6" [requires]python_version = "3.6"

Pipfile.lock not found, creating… Locking [dev-packages] dependencies… Locking [packages] dependencies… Updated Pipfile.lock (7a535c)! Installing dependencies from Pipfile.lock (7a535c)… 🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 92/92 — 00:01:01

real 8m1.993s user 5m32.406s sys 7m15.203s

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/pypa/pipenv/issues/356#issuecomment-380882203, or mute the thread https://github.com/notifications/unsubscribe-auth/ABhjqwIPyHtX0NTVoV1UPYR7HcwYm-2kks5tn42SgaJpZM4NbeoN .

kavdev commented 6 years ago

I had already installed the packages before that; I just removed the lockfile and ran install again. I'll do it again to get another data point