Pipenv lock still takes way too long to complete

[mjpieters]

pipenv lock is still taking up to 30 minutes to complete for my medium-sized project.

While #4403 was closed as fixed and indeed no full tree copies are made any more, pipenv lock is still unworkably slow. What used to take 50 seconds under 2018.11.26, now takes 30 minutes to figure out that nothing changed:

$ export PIPENV_INSTALL_TIMEOUT=10000
$ cp Pipfile.lock Pipfile.lock.before
$ time pipenv lock
Locking [dev-packages] dependencies…
Building requirements...
Resolving dependencies...
✔ Success!
Locking [packages] dependencies…
Building requirements...
Resolving dependencies...
✔ Success!
Updated Pipfile.lock (0e1198)!

real    29m19.701s
user    28m39.300s
sys 4m8.222s
$ diff -su Pipfile.lock.old Pipfile.lock
Files Pipfile.lock.old and Pipfile.lock are identical

This is making it untenable to keep using Pipenv.

Full pipenv lock -v output in a gist.

Pipfile:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
flake8 = "*"
flake8-bugbear = "*"
black = "==19.10b0"
pre-commit = "*"
pytest = "*"
pytest-cov = "*"

[packages]
project-name = {editable = true,path = "."}
"zc.buildout" = "*"

[requires]
python_version = "3.6"

where project-name (redacted name) lists the following install-requires lines:

install_requires =
    apache-airflow[aws,celery,postgres,redis] >= 1.10.11
    airflow_multi_dagrun
    airflow-prometheus-exporter

---

$ pipenv --support

Pipenv version: `'2020.8.13'` Pipenv location: `'/usr/local/Cellar/pipenv/2020.8.13/libexec/lib/python3.8/site-packages/pipenv'` Python location: `'/usr/local/Cellar/pipenv/2020.8.13/libexec/bin/python3.8'` Python installations found: - `3.9.0`: `/Users/mj/bin/python3.9` - `3.8.5`: `/usr/local/bin/python3` - `3.8.5`: `/usr/local/bin/python3.8` - `3.8.3`: `/Users/mj/bin/python3.8` - `3.7.7`: `/Users/mj/bin/python3.7` - `3.7.6`: `/opt/miniconda3/bin/python3` - `3.7.6`: `/opt/miniconda3/bin/python3.7m` - `3.7.6`: `/opt/miniconda3/bin/python3.7` - `3.7.3`: `/Users/mj/.pyenv/versions/3.7.3/bin/python3` - `3.6.10`: `/Users/mj/bin/python3.6` - `3.6.1`: `/Users/mj/bin/pypy3` - `3.5.9`: `/Users/mj/bin/python3.5` - `3.4.10`: `/Users/mj/bin/python3.4` - `3.3.7`: `/Users/mj/bin/python3.3` - `3.2.6`: `/Users/mj/bin/python3.2` - `2.7.18`: `/Users/mj/bin/python2.7` - `2.7.16`: `/usr/bin/python2.7` - `2.6.9`: `/Users/mj/bin/python2.6` - `2.5.6`: `/Users/mj/bin/python2.5` - `2.4.6`: `/Users/mj/bin/python2.4` PEP 508 Information: ``` {'implementation_name': 'cpython', 'implementation_version': '3.8.5', 'os_name': 'posix', 'platform_machine': 'x86_64', 'platform_python_implementation': 'CPython', 'platform_release': '17.7.0', 'platform_system': 'Darwin', 'platform_version': 'Darwin Kernel Version 17.7.0: Wed May 27 17:00:02 PDT ' '2020; root:xnu-4570.71.80.1~1/RELEASE_X86_64', 'python_full_version': '3.8.5', 'python_version': '3.8', 'sys_platform': 'darwin'} ``` System environment variables: - `PATH` - `AUTOJUMP_ERROR_PATH` - `NVM_INC` - `TERM_PROGRAM` - `GIT_PROMPT_END` - `rvm_bin_path` - `AUTOJUMP_SOURCED` - `GEM_HOME` - `NVM_CD_FLAGS` - `SHELL` - `TERM` - `HISTSIZE` - `TMPDIR` - `COPYFILE_DISABLE` - `IRBRC` - `Apple_PubSub_Socket_Render` - `CONDA_SHLVL` - `PERL5LIB` - `TERM_PROGRAM_VERSION` - `CONDA_PROMPT_MODIFIER` - `TERM_SESSION_ID` - `MY_RUBY_HOME` - `PERL_MB_OPT` - `LC_ALL` - `USER` - `HISTFILESIZE` - `NVM_DIR` - `COMMAND_MODE` - `CONDA_EXE` - `rvm_path` - `SSH_AUTH_SOCK` - `__CF_USER_TEXT_ENCODING` - `PTPIMG_API_KEY` - `_CE_CONDA` - `rvm_prefix` - `_` - `CONDA_PREFIX` - `PWD` - `HOMEBREW_GITHUB_API_TOKEN` - `EDITOR` - `LANG` - `ITERM_PROFILE` - `PYTHONSTARTUP` - `XPC_FLAGS` - `ITERM_ORIG_PS1` - `XPC_SERVICE_NAME` - `HISTCONTROL` - `_CE_M` - `FIGNORE` - `rvm_version` - `COLORFGBG` - `HOME` - `SHLVL` - `LC_TERMINAL_VERSION` - `ITERM_PREV_PS1` - `ITERM_SESSION_ID` - `PERL_LOCAL_LIB_ROOT` - `LOGNAME` - `CONDA_PYTHON_EXE` - `LC_CTYPE` - `GEM_PATH` - `GIT_PROMPT_ONLY_IN_REPO` - `CONDA_DEFAULT_ENV` - `NVM_BIN` - `DISPLAY` - `LC_TERMINAL` - `HGSRC` - `SECURITYSESSIONID` - `RUBY_VERSION` - `PERL_MM_OPT` - `COLORTERM` - `PIP_DISABLE_PIP_VERSION_CHECK` - `PYTHONDONTWRITEBYTECODE` - `PIP_SHIMS_BASE_MODULE` - `PIP_PYTHON_PATH` - `PYTHONFINDER_IGNORE_UNSUPPORTED` Pipenv–specific environment variables: Debug–specific environment variables: - `PATH`: `/usr/local/Cellar/pipenv/2020.8.13/libexec/tools:/usr/local/google-sdks/google-cloud-sdk/bin:/Users/mj/perl5/bin:/Users/mj/.nvm/versions/node/v12.14.1/bin:/Users/mj/.rvm/gems/ruby-2.3.0/bin:/Users/mj/.rvm/gems/ruby-2.3.0@global/bin:/Users/mj/.rvm/rubies/ruby-2.3.0/bin:/opt/miniconda3/bin:/opt/miniconda3/condabin:/Users/mj/bin:/Users/mj/Development/Library/zopatista_setup/bin:/Users/mj/Development/Library/zopatista_setup/audio:/usr/local/Cellar/ruby/2.7.1_2/bin:/Users/mj/.cargo/bin:/Users/mj/.poetry/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/usr/local/MacGPG2/bin:/usr/local/sbin:/opt/X11/bin:/Applications/Postgres.app/Contents/Versions/latest/bin:/Users/mj/.local/bin:/Users/mj/.rvm/bin:/Applications/Visual Studio Code.app/Contents/Resources/app/bin` - `SHELL`: `/bin/bash` - `EDITOR`: `vim` - `LANG`: `en_GB.UTF-8` - `PWD`: `/Users/mj` ---------------------------

[tbrodbeck]

For some reason locking pandas takes soooo long in my project. I even activated verbose mode, but still it does not give me any information what it is doing for 10min now... (found another "closed" issue: https://github.com/pypa/pipenv/issues/1198)

Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
Reporter.starting()
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.starting()
Reporter.adding_requirement(SpecifierRequirement('rpi.gpio'), None)
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('rpi.gpio'), None)
Reporter.adding_requirement(SpecifierRequirement('pytest'), None)
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('pytest'), None)
Reporter.adding_requirement(SpecifierRequirement('pyyaml'), None)
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('pyyaml'), None)
Reporter.adding_requirement(SpecifierRequirement('pyinstaller'), None)
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('pyinstaller'), None)
Reporter.adding_requirement(SpecifierRequirement('pandas'), None)
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('pandas'), None)
⠼ Locking...

Is there a way how to solve this with pipenv?

[matteius]

@tbrodbeck What version are you on? Can you try the branch linked off this issue? https://github.com/pypa/pipenv/issues/4926

[CharString]

@tbrodbeck I thinks it's bad taste to advocate another project here. So try the different branch. But there are alternatives to poetry, like pdm.

[tbrodbeck]

@matteius I am on version 2022.1.8 and Python 3.9.2

I tried to switch to the last branch mentioned, but I cannot access it:

fatal: invalid reference: issues-4651-infiite-iteration-take4

[matteius]

@tbrodbeck I am sorry for I should have been more clear that its this passing branch with the pip 22.0.4 vendoring changes: https://github.com/pypa/pipenv/pull/4969

[tbrodbeck]

I think you would need to provide me some installation instructions. I was not able to install the branch with python3 setup.py install or sudo python3 setup.py install

[matteius]

@tbrodbeck Hmmm, I seem to recall having issues with the install command as well, so I have been installing it using python3 setup.py develop --user to install it at user level which is preferred, if you need it at a system level you can drop the --user flag.

[tbrodbeck]

Hmm I am still getting permission denied:

t@v2:~/pipenv $ python3 setup.py develop --user
running develop
/home/t/.local/lib/python3.9/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
/home/t/.local/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
running egg_info
error: [Errno 13] Permission denied

[matteius]

@tbrodbeck I'd rather not ping all these people with each update -- perhaps make a new issue report and we can discuss more. I suspect you need to pip uninstall pipenv first if you are getting a permission denied trying to install to --user location, like its already installed there.

[em1208]

Pipenv unfortunately it is terribly slow in my case. Locking is taking more than 30min and still has not completed.

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[requires]
python_version = "3.7"

[packages]
Babel = ">=2.5"
bleach = "<3.0"
"boto3" = "<=1.9.183"
Django = ">=1.11,<=2.2.28"
dj_database_url = ">=0.3.0"
dj_email_url = ">=0.0.4"
django-babel = ">=0.6.1"
django-bootstrap4 = ">=0.0.4"
django-cache-url = ">=1.0.0"
django-countries = ">=5.2"
django-filter = ">=2.0"
django-graphql-jwt = ">=0.1.13,<=0.2.1"
pyjwt= "==1.7.1"
django-impersonate = ">=1.3"
django-measurement = "==3.0.0"
django-mptt = ">=0.7.1"
django-phonenumber-field = ">=1.3.0,<3.0"
django-prices = ">=1.0.2, <2.0"
django-prices-openexchangerates = ">=1.0.0"
django-prices-vatlayer = ">=1.0.2"
django-redis = ">=4.0.0"
django-storages = {extras = ["google"],version = "*"}
django-templated-email = ">=2.2.0"
django-versatileimagefield = ">=1.10"
django-webpack-loader = ">=0.3.0,<=0.6.0"
django-elasticsearch-dsl = ">=0.5.0.<0.6.0"
django-recaptcha = "<2.0.0"
django-silk = "==2.0.0"
django-debug-toolbar = "*"
django-debug-toolbar-request-history = "*"
django-extensions = "*"
ipython = "*"
elasticsearch = "==6.3.1"
elasticsearch-dsl = ">=6.0.0,<6.1.0"
faker = ">=0.7.7"
freezegun = ">=0.3.9"
google-measurement-protocol = ">=1.0.0"
"google-i18n-address" = ">=2.3.0"
graphene-django = ">=2.2,<=2.3.2"
Markdown = ">=2.4,<=3.1.1"
maxminddb = "*"
"maxminddb-geolite2" = "*"
phonenumberslite = ">=8.10.10"
prices = ">=1.0.0"
"psycopg2-binary" = ">=2.7,<=2.8.6"
purl = ">=0.4.1"
raven = "==6.9.0"
requests = ">=1.2.0"
social-auth-app-django = ">=2.0.0"
text-unidecode = "*"
uwsgi = {version = ">=2.0.0",platform_system = "!= 'Windows'"}
celery = {version = ">=4.3",extras = ["redis"]}
redis = "<4.0"
kombu = "<5.0.0"
weasyprint = ">=0.42.2"
graphene-django-optimizer = ">=0.4.0,<=0.4.2"
braintree = "==3.49.0"
razorpay = ">=1.1.1"
stripe = "*"
"urllib3" = "*"
dataclasses = {version = "*",markers = "python_version < '3.7'"}
django-contact-form = "*"
whitenoise = {extras = ["brotli"],version = "*"}

[dev-packages]
pytest = "*"
pytest-django = "*"
pytest-django-queries = "*"
pytest-vcr = "*"
pytest-mock = "*"
pytest-xdist = "*"
pylint-django = "*"
coverage = "*"
pytest-cov = "*"
codecov = "*"
tox = "*"
pycodestyle = "*"
pydocstyle = "*"
pylint = "*"
pylint_celery = "*"
isort = "*"
yapf = "*"
pylint-plugin-utils = "*"
transifex-client = "*"
black = "==19.3b0"
pre-commit = "*"
flake8 = "*"

I'm using pipenv, version 2022.4.8. Any suggestion how to solve this issue would be much appreciated.

[leirons]

I have more than 120 packages and it took about 1 hour, i thought the problem in me, but i was wrong, it's awfull use it in this state

[matteius]

@leirons Are you locking against pypi or a private package server? Private servers do not provide a hash API that is used to optimize the lock generation process.

[matteius]

@em1208 I just tried your example from within my Ubuntu VirtualBox VM (which is slower than my native machine) and it took 4.5 minutes. pipenv, version 2022.8.5

✔ Success! 
Updated Pipfile.lock (555f96)!

real    4m34.679s
user    3m21.142s
sys 0m8.162s

[matteius]

I am locking this discussion of this old and closed issue, and I am once again asking that if you are having a new issue that you don't respond to an old closed issue report with minimal details -- please open a new issue report and follow the template.

It is highly likely that your comment could be missed responding to an old issue as well, because we actively prioritize and tag new issues as they come in.