Lock updating is very slow

[wichert]

Updating a lockfile can be very slow. A standard pipenv lock easily takes well over a minute for me:

$ time pipenv lock 
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (abef76)!

real    1m56.988s
user    0m21.805s
sys 0m2.417s

This means every time I need to install, uninstall or upgrade a package I need to take a 2-minute break to wait for pipenv to finish updating its lockfile. I'm not sure why that is; yarn and npm seem to perform a similar task but only take seconds, even for projects that have many many more dependencies.

[techalchemy]

We are aware and have many issues tracking this topic. See #1785 #1886 #1891 and PR #1896

npm and yarn have the advantage of not having to fully download and execute each prospective package to determine their dependency graph because the dependencies are specified in plaintext. Python dependencies require us to fully download and execute the setup files of each package to resolve and compute. That's just the reality, it's a bit slow. If you can't wait 2 minutes or you feel it's not worth the tradeoff, you can always pass --skip-lock.

Closing to track in the other issues.

[djangorobert]

Same here im trying it out right now and installing django is already at 10 minutes and going

[djangorobert]

seems that if you try and install a couple at a time its slow but if you do one at a time works a bit faster

[techalchemy]

If you can provide a Pipfile that reproduces the slow behavior it would be helpful -- thanks

[dideler]

Python dependencies require us to fully download and execute the setup files of each package to resolve and compute

Is this still the case when the package has its own Pipfile.lock, or will pipenv use that when possible to determine dependencies?

[uranusjr]

will pipenv use that when possible to determine dependencies?

No. Pipenv is an application depedency resolution tool. When a dependency is used as a Python package by another application, it is no longer an application. Its dependencies are determined by setup.py (or setup.cfg or whatever you use to upload to PyPI). Loading dependencies from a lock file is a sure route to dependency hell, we will likely never ever do that.

[iddan]

It's still super slow

[uranusjr]

@iddan Thanks for the reminder, Captain Obvious!

[iddan]

Sorry, as an OSS maintainer I know how sometimes issues can get dismissed because of age. Just wanted to state that even the issue received discussion it’s still relevant

[dactylroot]

@techalchemy note of --skip-lock above is wonderful. This should be a more accessible or publicized option. Can we set it as a default somewhere?

[CauchyPolymer]

@techalchemy what if it takes 20 minutes with my brand new mac pro?

[theY4Kman]

@techalchemy note of --skip-lock above is wonderful. This should be a more accessible or publicized option. Can we set it as a default somewhere?

As far as I gather, the overwhelming benefit of pipenv is the assurance dependencies will play nicely together — not just for you, but for anyone later dealing in your code. The product of that assurance, the lock file, absolutely takes more time than anyone expects or desires, including the devs — see #2200.

However, I think you can also understand the opportunity pipenv has to shepherd well-meaning devs in the Python community at large toward a workflow imposing less head-scratching on future contributors — who might've only been visitors had they given up during the "figure out how to setup the dev environment" stage; and less hands thrown up by future maintainers — who might've only been drive-by PR authors had they given up during the "seriously screwing around with deep project internals" stage.

If --skip-lock were to become a permanent flag in a Pipfile or a setting in a pipenv config, pipenv's perception would slowly slide toward "better pip", and just another stepping stone fading into the horizon as the community eventually landed on a less compromising spiritual successor.

Better to leave it available only as an env var, or some other method whose application rests squarely in the "your user-specific local config, your fault" territory, allowing pipenv to overcome the passing phase of lockfile generation slowness without giving up the truly beneficial cultural shift toward explicitness over implicitness in package management.

Python's incredibly vast standard library is an enormous asset, whose history has undergone many eras of imposing consistency. That most standard packages play nicely together is an enormous feat involving consideration over many years by many people. One day, that play-nice-ability will extend to most Python projects encountered on the web — far, far from the stdlib, and with far, far fewer PEPs required (and far, far fewer BDFLs vacating in frustration). The impact of such a unilaterally buttery experience is hard to measure, but some current languages did refuse to compromise conceptual integrity for immediate convenience... and oh, the places they'll Go.

So yes, generating the lockfile is slow, and yes, it's frustrating when you only wanted pip install --save. But it's only because we've been sweeping an elephant into the closet for years — believing we didn't have a tangled mess of expectations and intentions from external dependencies, because "it installed fine on my machine".

Lockfile generation is slow only because it's making explicit what we've all taken for granted. But because it hurts, we will adjust things so it doesn't. We broke our arm because we pushed ourselves doing things we believed in. Sure, we can avoid the pain by never using that arm again— or we can put it in a cast while it heals.

I'd be the last person to tell you not to make pipenv convenient for yourself today (otherwise I'd be a shitty developer — though, the jury's still out), but I implore you to see the frustration of lockfile generation as a growing pain while the Python community develops into a strong, healthy body with more fully-functioning limbs than one really expected when removing a cast. We made it to Python 3. Let's make it to dependency management in the stdlib.

[sjjpo2002]

This took 38 minutes on my machine to create the lock file. Running Windows 10 and using Python 3.7

[vanyakosmos]

Only numpy and pillow were already installed and it took <1 seconds to install numba and 25 minutes to lock it. Does pipenv forcibly compile every lib on lock or how does this work?

[techalchemy]

FYI persistent skip lock is just waiting on someone to flip the switch for auto_envvar_prefix which is a click setting. I've been 100% focused on core functionality so I haven't had a chance to look at this yet but I suspect it isn't that difficult

[zazazack]

TLDR; Typical pipenv install invocation: Time: 144.82 real 33.68 user 5.77 sys. With --skip-lock: Time: 4.54 real 5.33 user 0.87 sys.

Pandas-datareader install fails on first attempt, possibly cause of lock hanging. Is this an issue anyone else is seeing with other packages?

Using version 2018.11.26

$ pipenv --version
pipenv, version 2018.11.26

Contents of requirements.txt

sklearn-pandas
sklearn
pandas
numpy
matplotlib
statsmodels

Typical pipenv install invocation. Timed execution using time (BSD).

Results: 144.82 real 33.68 user 5.77 sys

$ time pipenv install -r requirements.txt
Requirements file provided! Importing into Pipfile…
Pipfile.lock (0fdb67) out of date, updating to (a65489)…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
✔ Success!
Updated Pipfile.lock (0fdb67)!
Installing dependencies from Pipfile.lock (0fdb67)…
  🎅   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 15/15 — 00:00:04
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
      144.82 real        33.68 user         5.77 sys

Invoking w\ --skip-lock

Results: 4.54 real 5.33 user 0.87 sys

$ time pipenv install -r requirements.txt --skip-lock
Requirements file provided! Importing into Pipfile…
Installing dependencies from Pipfile…
  🎅   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 6/6 — 00:00:01
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
        4.54 real         5.33 user         0.87 sys

[jufemaiz]

I think https://github.com/pandas-dev/pandas/ may be a problem? It's a common point with the time for me too.

Although pytest may also be an issue :\

[black-snow]

It won't even finish on my machine:

Installing pandas…
Adding pandas to Pipfile's [packages]…
Installation Succeeded
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Traceback (most recent call last):
  File "c:\python36\lib\site-packages\pipenv\vendor\pexpect\expect.py", line 109, in expect_loop
    return self.timeout()
  File "c:\python36\lib\site-packages\pipenv\vendor\pexpect\expect.py", line 82, in timeout
    raise TIMEOUT(msg)
pexpect.exceptions.TIMEOUT: <pexpect.popen_spawn.PopenSpawn object at 0x00000292ADCCCDD8>
searcher: searcher_re:
    0: re.compile('\n')

[theY4Kman]

@black-snow I'd recommend trying it in a different shell. Without diving too deep into things, it seems like pexpect (a library for programmatically interfacing with interactive CLI programs) is used to detect the shell pipenv is being executed from, and that might be stalling. pexpect kinda seems overkill for such a thing, but I'm not aware of the whole context.

[black-snow]

@theY4Kman thanks for the advice. pipenv is working fine on another pc with same ubuntu and bash version ...

[kstohr]

Read this while waiting for pipfile to lock...:) Would be great if there was a solution.

[uetchy]

Seems like we need some kind of API to parse and cache Python packages deps and distribute it in a machine-friendly format. So we will no longer need to download entire packages and parse them.

[jufemaiz]

I believe bundler and ruby gems maintains and uses somthing like that.

[danielcompton]

Java also has POM files (XML) which contain package dependencies and other information about the package. They are uploaded separately from the compiled JARs.

[black-snow]

Every newer package manager has some separate meta files (npm/yarn, composer, gradle/maven, cargo, ruby gems/bundler, ...).

[uetchy]

related issue: https://github.com/pypa/warehouse/issues/474

[brandonrobertz]

You can get dependency information from PyPi without downloading the entire bundle (see PEP 566, which superseded PEP 426).

package_name = 'Django'
PYPI_API_URL = 'https://pypi.python.org/pypi/{package_name}/json'
package_details_url = PYPI_API_URL.format(package_name=package_name)
response = requests.get(package_details_url)
data = json.loads(response.content)
data['info'].get('requires_dist')
data['info'].get('requires')
data['info'].get('setup_requires')
data['info'].get('test_requires')
data['info'].get('install_requires')

[jourdanrodrigues]

@techalchemy did you see the comment above?

[massenz]

This is pretty consistently happening, essentially pipenv going away around town while "locking" something: why is this issue closed?

Understand the --skip-lock is the way to go, but it's not clear at all why "installing" takes a few seconds, and "locking" takes forever: it would be great if at least the lock command would emit some progress/update logs: as things stand, it's not even clear if it just got stuck in some sort of while True forever...

I'd at least like to know if it's me doing something wrong, or just a "feature" of pipenv.

[upwell]

In our project pipenv lock is so so so slow. It definitely impacted our normal use. Adding a new package becomes a real pain for us now. Is there anyway we can debug this behavior?

[awhillas]

I'm trying to install PyTorch and it took 20 minutes to lock and then it pulls the following error

Installing initially failed dependencies…
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.6/dist-packages/pipenv/core.py", line 1992, in do_install
[pipenv.exceptions.InstallError]:       skip_lock=skip_lock,
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.6/dist-packages/pipenv/core.py", line 1253, in do_init
[pipenv.exceptions.InstallError]:       pypi_mirror=pypi_mirror,
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.6/dist-packages/pipenv/core.py", line 859, in do_install_dependencies
[pipenv.exceptions.InstallError]:       retry_list, procs, failed_deps_queue, requirements_dir, **install_kwargs
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.6/dist-packages/pipenv/core.py", line 763, in batch_install
[pipenv.exceptions.InstallError]:       _cleanup_procs(procs, not blocking, failed_deps_queue, retry=retry)
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.6/dist-packages/pipenv/core.py", line 681, in _cleanup_procs
[pipenv.exceptions.InstallError]:       raise exceptions.InstallError(c.dep.name, extra=err_lines)
[pipenv.exceptions.InstallError]: ['Collecting pytorch==1.0.2 (from -r /tmp/pipenv-pb00kf8t-requirements/pipenv-vae35p2d-requirement.txt (line 1))', '  Using cached https://files.pythonhosted.org/packages/ee/67/f403d4ae6e9cd74b546ee88cccdb29b8415a9c1b3d80aebeb20c9ea91d96/pytorch-1.0.2.tar.gz', 'Building wheels for collected packages: pytorch', '  Building wheel for pytorch (setup.py): started', "  Building wheel for pytorch (setup.py): finished with status 'error'", '  Running setup.py clean for pytorch', 'Failed to build pytorch', 'Installing collected packages: pytorch', '  Running setup.py install for pytorch: started', "    Running setup.py install for pytorch: finished with status 'error'"]
[pipenv.exceptions.InstallError]: ['ERROR: Complete output from command /home/alex/.local/share/virtualenvs/pytorch-umelu-tG/bin/python3 -u -c \'import setuptools, tokenize;__file__=\'"\'"\'/tmp/pip-install-hix3yz6v/pytorch/setup.py\'"\'"\';f=getattr(tokenize, \'"\'"\'open\'"\'"\', open)(__file__);code=f.read().replace(\'"\'"\'\\r\\n\'"\'"\', \'"\'"\'\\n\'"\'"\');f.close();exec(compile(code, __file__, \'"\'"\'exec\'"\'"\'))\' bdist_wheel -d /tmp/pip-wheel-f_h8w1pz --python-tag cp36:', '  ERROR: Traceback (most recent call last):', '    File "<string>", line 1, in <module>', '    File "/tmp/pip-install-hix3yz6v/pytorch/setup.py", line 15, in <module>', '      raise Exception(message)', '  Exception: You tried to install "pytorch". The package named for PyTorch is "torch"', '  ----------------------------------------', '  ERROR: Failed building wheel for pytorch', '    ERROR: Complete output from command /home/alex/.local/share/virtualenvs/pytorch-umelu-tG/bin/python3 -u -c \'import setuptools, tokenize;__file__=\'"\'"\'/tmp/pip-install-hix3yz6v/pytorch/setup.py\'"\'"\';f=getattr(tokenize, \'"\'"\'open\'"\'"\', open)(__file__);code=f.read().replace(\'"\'"\'\\r\\n\'"\'"\', \'"\'"\'\\n\'"\'"\');f.close();exec(compile(code, __file__, \'"\'"\'exec\'"\'"\'))\' install --record /tmp/pip-record-xr7o93_5/install-record.txt --single-version-externally-managed --compile --install-headers /home/alex/.local/share/virtualenvs/pytorch-umelu-tG/include/site/python3.6/pytorch:', '    ERROR: Traceback (most recent call last):', '      File "<string>", line 1, in <module>', '      File "/tmp/pip-install-hix3yz6v/pytorch/setup.py", line 11, in <module>', '        raise Exception(message)', '    Exception: You tried to install "pytorch". The package named for PyTorch is "torch"', '    ----------------------------------------', 'ERROR: Command "/home/alex/.local/share/virtualenvs/pytorch-umelu-tG/bin/python3 -u -c \'import setuptools, tokenize;__file__=\'"\'"\'/tmp/pip-install-hix3yz6v/pytorch/setup.py\'"\'"\';f=getattr(tokenize, \'"\'"\'open\'"\'"\', open)(__file__);code=f.read().replace(\'"\'"\'\\r\\n\'"\'"\', \'"\'"\'\\n\'"\'"\');f.close();exec(compile(code, __file__, \'"\'"\'exec\'"\'"\'))\' install --record /tmp/pip-record-xr7o93_5/install-record.txt --single-version-externally-managed --compile --install-headers /home/alex/.local/share/virtualenvs/pytorch-umelu-tG/include/site/python3.6/pytorch" failed with error code 1 in /tmp/pip-install-hix3yz6v/pytorch/']
ERROR: ERROR: Package installation failed...

The error is unreadle, no idea what went wrong. Installing with pip in the environment work fine! This is really a show stopper. Going back to requirements.txt...

[shtratos]

This is the workaround I use for now:

export PIPENV_SKIP_LOCK=true

Then pipenv install foo won't be locking and you can lock manually when you have time by running pipenv lock.

[BramVanroy]

@awhillas Pretty sure the last line says all you need:

You tried to install "pytorch". The package named for PyTorch is "torch"

[evictor]

Locking dependencies is important so I don't think "skip lock" is a lasting solution. At the same time, I simply don't buy that "locking dependencies" (whatever that could possibly entail under the hood) is maximally optimized as it is now and functionally requires many minutes or hours to complete. Indeed, my pipenv lock ran for several minutes on a Pipfile that has a laughable 5 dependencies before failing—stack attached at bottom—and during that time used only 10–15% of available CPU and a little sip of memory.

Can we at least put forth some group effort to profile and determine the bottlenecks? I have a feeling there are just some silly low hanging fruit in there just waiting to take this process into reasonable runtime.

pipenv version 2018.11.26

For Pipfile:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]

[dev-packages]
keras = "*"
tensorflow = "~=1.13"
setuptools = "*"
wheel = "*"
twine = "*"

[requires]
python_version = ">=3.6"

Pipfile.lock (63b275) out of date, updating to (5e165c)…
Locking [dev-packages] dependencies…
Traceback (most recent call last):
  File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/vendor/pexpect/expect.py", line 109, in expect_loop
    return self.timeout()
  File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/vendor/pexpect/expect.py", line 82, in timeout
    raise TIMEOUT(msg)
pexpect.exceptions.TIMEOUT: <pexpect.popen_spawn.PopenSpawn object at 0x105a17210>
searcher: searcher_re:
    0: re.compile('\n')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/bin/pipenv", line 11, in <module>
    load_entry_point('pipenv==2018.11.26', 'console_scripts', 'pipenv')()
  File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/vendor/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/vendor/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/cli/command.py", line 254, in install
    editable_packages=state.installstate.editables,
  File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/core.py", line 1992, in do_install
    skip_lock=skip_lock,
  File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/core.py", line 1221, in do_init
    pypi_mirror=pypi_mirror,
  File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/core.py", line 1068, in do_lock
    lockfile=lockfile
  File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/utils.py", line 649, in venv_resolve_deps
    c = resolve(cmd, sp)
  File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/utils.py", line 517, in resolve
    result = c.expect(u"\n", timeout=environments.PIPENV_INSTALL_TIMEOUT)
  File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/vendor/delegator.py", line 215, in expect
    self.subprocess.expect(pattern=pattern, timeout=timeout)
  File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/vendor/pexpect/spawnbase.py", line 341, in expect
    timeout, searchwindowsize, async_)
  File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/vendor/pexpect/spawnbase.py", line 369, in expect_list
    return exp.expect_loop(timeout)
  File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/vendor/pexpect/expect.py", line 119, in expect_loop
    return self.timeout(e)
  File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/vendor/pexpect/expect.py", line 82, in timeout
    raise TIMEOUT(msg)
pexpect.exceptions.TIMEOUT: <pexpect.popen_spawn.PopenSpawn object at 0x105a17210>
searcher: searcher_re:
    0: re.compile('\n')

[evictor]

As an addendum to previous comment, running pip lock separately took a reasonable amount of time, ~15 seconds, after first running pip install --skip-lock. So perhaps the post-install invocation of lock is out of date or otherwise faulty. :)

[evictor]

FYI I found tensorflow is the culprit of the slow/timing out lock, if that helps profile pipenv! (Still consider this a pipenv problem...)

[brandonrobertz]

Tensorflow is one of many packages that can cause pipenv to basically become a useless tool. I like the suggestion of group profiling, though. I think that's an excellent idea to start tacking this problem. Just to re-iterate, PEP 566 enabled enumerating dependence (via pypi) without loading the entire source, which may be helpful: https://github.com/pypa/pipenv/issues/1914#issuecomment-457965038

[shtratos]

@brandonrobertz from what I see, downloading all packages of the dependencies is where most time is spent. This also has been confirmed before.

How to verify this:

• Try creating a new Pipfile and installing scipy (for example) in it with locking enabled
• Wait until locking completes. (Takes about 5 minutes on my machine)
• remove Pipfile.lock
• run pipenv lock - locking now will take very little time (6 seconds on my machine), because all packages are already downloaded and kept in Pipenv cache which is normally in ~/.cache/pipenv

Here's the Dockerfile I used to test this:

FROM python:3.6
ENV WORKDIR=/work/
WORKDIR /work/
RUN python3 -m pip install --upgrade pip
RUN python3 -m pip install pipenv
RUN PIPENV_SKIP_LOCK=true pipenv install scipy
RUN date
RUN pipenv lock
RUN date
RUN rm Pipfile.lock
RUN pipenv lock
RUN date

[brandonrobertz]

@shtratos Yeah, that makes sense and others have suggested that in this issue thread. Downloading and parsing dependencies is expensive. Some of those steps seem like they can be eliminated by pulling from the pypi dependency API.

For some libraries this will probably not work, due to poor quality and bad practices (not using setup.py or requirements.txt). But since some of the major offenders seem to be very popular libraries (tensorflow, numpy), implementing this with a fallback to the super slow process might be a good path forward.

[evictor]

Can you point me in the direction of where to find that code? I could take a stab at parallelizing it in a fork.

[yassineameur]

I think https://github.com/pandas-dev/pandas/ may be a problem? It's a common point with the time for me too.

Although pytest may also be an issue :\

I doi not think so, it works fine on my machine, the problem seems to be more general than that

[djbrown]

In my case the problem seems to be pylint. It always hangs on locking when simply running pipenv install pylint see https://github.com/pypa/pipenv/issues/2284#issuecomment-569457752

I have the same problem in all of my projects. The cause seems to be pylint. Pipenv (pip) can install it successfully, but locking takes forever! pipenv, version 2018.11.26

Minimal working example

djbrown@DESKTOP-65P6D75:~$ mkdir test
djbrown@DESKTOP-65P6D75:~$ cd test
djbrown@DESKTOP-65P6D75:~/test$ pipenv install --dev pylint --verbose
Creating a virtualenv for this project…
Pipfile: /home/djbrown/test/Pipfile
Using /usr/bin/python3 (3.6.9) to create virtualenv…
⠸ Creating virtual environment...Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /home/djbrown/.local/share/virtualenvs/test-PW-auWy_/bin/python3
Also creating executable in /home/djbrown/.local/share/virtualenvs/test-PW-auWy_/bin/python
Installing setuptools, pip, wheel...done.

✔ Successfully created virtual environment!
Virtualenv location: /home/djbrown/.local/share/virtualenvs/test-PW-auWy_
Creating a Pipfile for this project…
Installing pylint…
⠋ Installing...Installing 'pylint'
$ ['/home/djbrown/.local/share/virtualenvs/test-PW-auWy_/bin/pip', 'install', '--verbose', '--upgrade', 'pylint', '-i', 'https://pypi.org/simple']
Adding pylint to Pipfile's [dev-packages]…
✔ Installation Succeeded
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
⠇ Locking...

[vlsd]

We are aware and have many issues tracking this topic. See #1785 #1886 #1891 and PR #1896

npm and yarn have the advantage of not having to fully download and execute each prospective package to determine their dependency graph because the dependencies are specified in plaintext. Python dependencies require us to fully download and execute the setup files of each package to resolve and compute. That's just the reality, it's a bit slow. If you can't wait 2 minutes or you feel it's not worth the tradeoff, you can always pass --skip-lock.

Closing to track in the other issues.

3 out of the 4 other issues are now closed and the remaining one hasn't seen any activity since 2018. This issue still persists, so maybe re-opening it would be a good idea?

[wichert]

Python dependencies require us to fully download and execute the setup files of each package to resolve and compute

I don't think that is still true for wheels, which should be the majority of packages now?

[evictor]

I know I at least have to build the wheel for dlib every time, which is horrendous.

The process of resolving dependencies should be cached somewhere on a per-package-version basis, even if it requires another remote lookup on the client each time to get the tree (it shouldn't, you could just cache it locally too after the fact). For example, package repos could easily store resolved dep trees. The additional network hit for the cached resolved dep tree would always be faster than downloading an entire package and computing.

[massenz]

FWIW I've removed pipenv from all my projects (this being the one main reason, there are others).

virtualenv + pip (with requirements.txt) now work pretty well, even for Prod deployments; and in any event, one deploys a fully-formed container these days; after getting really into pipenv, I no longer see the point of it.

[dpinol]

Please reopen this issue. Otherwise pipenv will never be the reference packaging tool

[kaszperro]

It's still super slow. I recommend going with Poetry.

[lyle-nel]

Could we signal that this is still a problem by re-opening this issue?

[matteius]

@lyle-nel I don't think re-opening this specific issue would help since the content of it is dated, for example,

Python dependencies require us to fully download and execute the setup files of each package to resolve and compute. That's just the reality, it's a bit slow.

This is no longer true about pipenv as it gets the package hashes from the pypi API. There is a merged PR connected here that relates back to the issue at hand being resolved. If the community still feels that locking is very slow (too slow) and that there might be some optimization that can be done that wouldn't compromise the integrity of locking (I am unsure) then at this stage, the path forward is to open a new issue with as much detail as possible, including method for reproducing the issue using the latest version of pipenv, and any suggestions on how to improve it would make the ticket even better.