`pipenv install XYZ` command sometimes hangs indefinitely at "Locking [packages] dependencies"

[MarenstainBear]

Similar to these issues:

• https://github.com/pypa/pipenv/issues/356
• https://github.com/pypa/pipenv/issues/278

Summary

When using pipenv install XYZ, the process hangs indefinitely at "Locking [packages] dependencies".

This doesn't happen every time, and may be related to which package is being installed. This morning it is happening every time I try to install tensorflow. Last night it was happening for numpy as well, but today numpy is fine.

The hanged response seems to last indefinitely (though I did manually terminate the process after an hour). The Pipfile.lock is not written, but the Pipfile is written and the expected packages are installed and listed when I run pipenv run pip freeze.

I was able to reproduce this using various permutations of:

• (L)ubuntu 16.04 and (X)ubuntu 17.10
• pipenv --three and pipenv --two
• pipenv installed through python 2.7 and pipenv installed through python 3.5
• ...and I downloaded the pipenv source and installed that in an isolated environment to see if the issue was specific to pipenv version 11.8.3. I experienced the same behavior in 11.9.0.

I am also able to reproduce it by simply running pipenv lock or pipenv update if I have tensorflow (or whatever package is currently causing me trouble) listed in my Pipfile.

$ python -m pipenv.help output

Pipenv version: `'11.8.3'` Pipenv location: `'/home/mary/.local/lib/python2.7/site-packages/pipenv'` Python location: `'/usr/bin/python'` Other Python installations in `PATH`: - `2.7`: `/usr/bin/python2.7` - `2.7`: `/usr/bin/python2.7` - `3.5`: `/usr/bin/python3.5m` - `3.5`: `/usr/bin/python3.5` - `2.7.12`: `/usr/bin/python` - `2.7.12`: `/usr/bin/python2` - `3.5.2`: `/usr/bin/python3` PEP 508 Information: ``` {'implementation_name': 'cpython', 'implementation_version': '0', 'os_name': 'posix', 'platform_machine': 'x86_64', 'platform_python_implementation': 'CPython', 'platform_release': '4.13.0-37-generic', 'platform_system': 'Linux', 'platform_version': '#42~16.04.1-Ubuntu SMP Wed Mar 7 16:03:28 UTC 2018', 'python_full_version': '2.7.12', 'python_version': '2.7', 'sys_platform': 'linux2'} ``` System environment variables: - `MANDATORY_PATH` - `_LXSESSION_PID` - `XDG_GREETER_DATA_DIR` - `PROJECT_HOME` - `LC_CTYPE` - `PYTHONDONTWRITEBYTECODE` - `XDG_CURRENT_DESKTOP` - `XDG_SESSION_TYPE` - `LOGNAME` - `XDG_SEAT` - `PATH` - `XDG_VTNR` - `QT_PLATFORM_PLUGIN` - `PYTHONUNBUFFERED` - `VIRTUALENVWRAPPER_SCRIPT` - `ZSH` - `DISPLAY` - `SSH_AGENT_PID` - `LANG` - `TERM` - `SHELL` - `XDG_SESSION_PATH` - `XAUTHORITY` - `LANGUAGE` - `SHLVL` - `QT_LINUX_ACCESSIBILITY_ALWAYS_ON` - `QT_QPA_PLATFORMTHEME` - `SESSION_FOLDER` - `QT_ACCESSIBILITY` - `WINDOWID` - `LIBVIRT_DEFAULT_URI` - `HOME` - `XDG_SESSION_DESKTOP` - `SAL_USE_VCLPLUGIN` - `XDG_RUNTIME_DIR` - `WORKON_HOME` - `SSH_AUTH_SOCK` - `VTE_VERSION` - `GDMSESSION` - `VIRTUALENVWRAPPER_WORKON_CD` - `XDG_SEAT_PATH` - `PIP_PYTHON_PATH` - `XDG_SESSION_ID` - `DBUS_SESSION_BUS_ADDRESS` - `_` - `VIRTUALENVWRAPPER_HOOK_DIR` - `VIRTUALENVWRAPPER_PROJECT_FILENAME` - `DESKTOP_SESSION` - `LSCOLORS` - `XDG_CONFIG_DIRS` - `DEFAULTS_PATH` - `XDG_CONFIG_HOME` - `OLDPWD` - `LS_COLORS` - `GDM_LANG` - `XDG_DATA_DIRS` - `PWD` - `XDG_MENU_PREFIX` - `LESS` - `PAGER` - `USER` Pipenv–specific environment variables: Debug–specific environment variables: - `PATH`: `/home/mary/bin:/home/mary/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games` - `SHELL`: `/usr/bin/zsh` - `LANG`: `en_US.UTF-8` - `PWD`: `/home/mary/Development/Projects/poor_mans_smart_camera` --------------------------- Contents of `Pipfile` ('/home/mary/Development/Projects/poor_mans_smart_camera/Pipfile'): ```toml [[source]] url = "https://pypi.python.org/simple" verify_ssl = true name = "pypi" [dev-packages] [packages] numpy = "*" tensorflow = "*" [requires] python_version = "3.5" ```

---

Expected result

I expect the step in which the Pipfile.lock is generated to take at most a minute.

I have good hardware with lots of free memory and processing power, so I don't think this is a resource constraint. My internet speed is 60 Mbps, so I doubt it is a networking issue when requesting packages.

Actual result

verbose_output.txt

While this screenshot doesn't show verbose output like the file above, it does visually demonstrate what I am experiencing.

Steps to replicate

1. Spin up new virtualenv using pipenv --three or pipenv --two.
2. Confirm things are working as expected with a package you know is installing fine such as pipenv install pylint.
3. Now cause it to hang by installing a package that is not working for whatever reason such as pipenv install tensorflow.

Pipfile

[ghost]

I copied the command that I now see includes a dash instead of two hyphens.

I've now run pipenv-resolver --debug pendulum which runs correctly and produces the following results: pipenv-resolver_pendulum.txt

[techalchemy]

Sorry about that I triage a lot of stuff on mobile and it auto converts to em-dashes. So this suggests an installation issue. What version winds up in your lockfile? Can you pip install that version of pendulum directly?

[ghost]

No worries, I see that you're putting a lot of effort to this project.

I managed to install pendulum with pipenv but skipping the generation of the lock file. After a day and with a fresh boot it correctly generated the lock file as well, and the pendulum version is "==1.5.1"

Bad thing is that so far there is no way to reproduce this issue, which makes it difficult to fix, but hopefully something will come up.

[techalchemy]

Next release should help with this a bit. Should have it in a few hours.

[ericflores]

I was experiencing the same problem when installing pandas. As I had faced problems with pandas and the latest version of pip, so I decided to downgrade pip to 9.0.3 and the issue with pipenv also got solved. Check this issue for your reference: https://github.com/pandas-dev/pandas/issues/20666

[speed-of-light]

pipenv install scrapy hangs over 10 mins

python --version 3.6.5 pip --version 10.0.1

cat Pipfile

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages] scrapy = "*"

[dev-packages]

[requires] python_version = "3.6"

[jakul]

I had the same issue. Here's where it was hanging for me:

$ pipenv lock -vv
Locking [dev-packages] dependencies…
Using pip: -i https://pypi.org/simple

                          ROUND 1
Current constraints:

Finding the best candidates:

Finding secondary dependencies:
------------------------------------------------------------
Result of round 1: stable, done

Locking [packages] dependencies…

$ pipenv-resolver --debug requests
DEBUG:pip9.vcs:Registered VCS backend: git
DEBUG:pip9.vcs:Registered VCS backend: hg
DEBUG:pip9.vcs:Registered VCS backend: svn
DEBUG:pip9.vcs:Registered VCS backend: bzr
DEBUG:notpip.index:2 location(s) to search for versions of requests:
DEBUG:notpip.index:* https://pypi.python.org/simple/requests/
DEBUG:notpip.index:* https://pypi.boughtbymany.com/5c0a44cd-ec93-412c-b3a9-c3dfb72c22ee/requests/
DEBUG:notpip.index:Getting page https://pypi.python.org/simple/requests/
DEBUG:pip9._vendor.cachecontrol.controller:Looking up "https://pypi.python.org/simple/requests/" in the cache
DEBUG:pip9._vendor.cachecontrol.controller:Returning cached "301 Moved Permanently" response (ignoring date and etag information)
DEBUG:pip9._vendor.cachecontrol.controller:Looking up "https://pypi.org/simple/requests/" in the cache
WARNING:pip9._vendor.cachecontrol.controller:Cache entry deserialization failed, entry ignored
DEBUG:pip9._vendor.urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org
DEBUG:pip9._vendor.urllib3.connectionpool:https://pypi.org:443 "GET /simple/requests/ HTTP/1.1" 200 16346
DEBUG:pip9._vendor.cachecontrol.controller:Updating cache with response from "https://pypi.org/simple/requests/"
DEBUG:pip9._vendor.cachecontrol.controller:Caching due to etag

The problem was something to do with pip trying to write something to it's cache. I was able to resolve it by clearing my pip cache - which for Mac users is rm -rf ~/Library/Caches/pip.

Thanks to all the contributers to this issue!

[woodrad]

Clearing pip’s cache worked for me too! Thanks so much to @MarenstainBear @uranusjr @techalchemy @jtratner for your work!

[renzon]

Same stuff here, pip env hanging on Mac OS. I've tried to create a simple pipenv install flask on a new folder only for the sake of testing. It was hanging, but I noted it's was not using pip's cache dir, but its own:

Building wheels for collected packages: itsdangerous, MarkupSafe
  Running setup.py bdist_wheel for itsdangerous: started
  Running setup.py bdist_wheel for itsdangerous: finished with status 'done'
  Stored in directory: /Users/renzo/Library/Caches/pipenv/wheels/2c/4a/61/5599631c1554768c6290b08c02c72d7317910374ca602ff1e5
  Running setup.py bdist_wheel for MarkupSafe: started
  Running setup.py bdist_wheel for MarkupSafe: finished with status 'done'
  Stored in directory: /Users/renzo/Library/Caches/pipenv/wheels/33/56/20/ebe49a5c612fffe1c5a632146b16596f9e64676768661e4e46

So what worked for me was

rm -rf ~/Library/Caches/pipenv

Really strange behaviour. Found this behaviour on after upgrading python from 3.6.3 to 3.7.0

[techalchemy]

pipenv --clear will handle this for you

[duke79]

Error: no such option: --clear

[techalchemy]

Upgrade to the latest version first.

edit unless this isn’t released and is only in master. In that case wait until I cut a release probably this week, or use the version available in master :D can’t remember if this actually made the cutoff...

[Mathspy]

@techalchemy It's definitely not released yet as I have latest version 2018.7.1 and I don't have --clear option too

[uranusjr]

For this particular problem, pipenv lock --clear should be enough.

[duke79]

pipenv lock --clear works. Thanks!

[crifan]

still NOT work for me for pipenv lock --clear

macOS High Sierra 10.13.6 Python: Python 3.6.4 pipenv: version 2018.7.1

Hope fix this Locking [packages] dependencies... hang forever ASAP

[jakul]

@crifan I've found that there's a bug stopping pipenv lock --clear from working (https://github.com/pypa/pipenv/issues/2628).

Try to manually clear your caches - I use rm -r ~/Library/Caches/pip*.

[braco]

Happening in this alpine linux based Dockerfile (child of this):

FROM frolvlad/alpine-python3

RUN pip install pipenv

COPY ./app /app
COPY Pipfile /app/
WORKDIR /app

RUN pipenv install

With this Pipfile:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[dev-packages]

[packages]
numpy = "*"
pandas = "*"
tensorflow = "*"
flask = "*"

[requires]
python_version = "3.6"

[memeplex]

~/devel/dopper:fixes: pipenv lock --clear --verbose
Locking [dev-packages] dependencies...
Locking [packages] dependencies...

hangs forever.

Pipfile:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
proto = {editable = true, path = "."}
halt = {editable = true, path = "/home/carlos/devel/halt"}
gcd = {editable = true, path = "/home/carlos/devel/gcd"}

[dev-packages]

[requires]
python_version = "3.6"

[techalchemy]

Lockfiles which only contain stuff that is on your local machine are just as unhelpful as not providing any information. Unless you are offering new debugging information or a new thing that we can take and reproduce, please be mindful about issue tracker noise.

[memeplex]

Those are simple dependency projects that I can easily install manually running pip install -e . on each one, but running pipenv install -e . on the main project hangs indefinitely, so I found out this issue and tried pipenv lock --clear which also hangs. I understand there is not much you can do without further information, but at least this is evidence that things that run smoothly the old way are hanging with pipenv and that the proposed solution is not working everywhere. Sorry I can't provide more information. Maybe if --verbose mode were indeed verbose here.

[memeplex]

Maybe this could shed some light:

~/devel/dopper:fixes: pipenv --rm
Removing virtualenv (/home/carlos/.local/share/virtualenvs/dopper-0v9QQl0s)...
~/devel/dopper:fixes: rm -rf Pipfile 
~/devel/dopper:fixes: pipenv --three
Creating a virtualenv for this project...
Pipfile: /home/carlos/devel/dopper/Pipfile
Using /usr/bin/python3 (3.6.6) to create virtualenv...
⠋Running virtualenv with interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /home/carlos/.local/share/virtualenvs/dopper-0v9QQl0s/bin/python3
Also creating executable in /home/carlos/.local/share/virtualenvs/dopper-0v9QQl0s/bin/python
Installing setuptools, pip, wheel...done.
Setting project for dopper-0v9QQl0s to /home/carlos/devel/dopper

Virtualenv location: /home/carlos/.local/share/virtualenvs/dopper-0v9QQl0s
Creating a Pipfile for this project...
~/devel/dopper:fixes: pipenv install numpy
Installing numpy...
Collecting numpy
  Using cached https://files.pythonhosted.org/packages/88/29/f4c845648ed23264e986cdc5fbab5f8eace1be5e62144ef69ccc7189461d/numpy-1.15.0-cp36-cp36m-manylinux1_x86_64.whl
Installing collected packages: numpy
Successfully installed numpy-1.15.0

Adding numpy to Pipfile's [packages]...
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...

Waiting forever...

Suppose it's a network issue, now why this works then? :

~:: pip install --force --upgrade --user numpy
Collecting numpy
  Downloading https://files.pythonhosted.org/packages/88/29/f4c845648ed23264e986cdc5fbab5f8eace1be5e62144ef69ccc7189461d/numpy-1.15.0-cp36-cp36m-manylinux1_x86_64.whl (13.9MB)
    100% |████████████████████████████████| 13.9MB 386kB/s 
Installing collected packages: numpy
  Found existing installation: numpy 1.15.0
    Uninstalling numpy-1.15.0:
      Successfully uninstalled numpy-1.15.0
Successfully installed numpy-1.15.0

[memeplex]

After a lot of attempts (installing dependencies one by one, avoiding local dependencies, avoiding editable dependencies, installing with skip lock then locking, etc) I was unable to find out any pattern. Sometimes it works in a sensible amount of time, sometimes it aborts after a few minutes with a timeout error, sometimes it hangs "forever" (which definition is contingent on my diminishing patience). I assume my problem is related to the many issues reporting some variant of "locking is too slow".

[techalchemy]

Again, please stop taking random guesses and posting random snippets of your output. If you want help with a specific problem, fill out an issue with the entire issue template. We are not going on a wild goose chase with anyone in this issue while they selectively post output that is mostly unrelated to pipenv. We have seen what the screen looks like when it hangs. Pasting that to us is not useful.

[memeplex]

Ok, if you think failing to install numpy in a clean environment is a random problem I know I'm taking my chances when using pipenv, thanks for the heads up and I'll shut my mouth up.

On Wed, Jul 25, 2018 at 6:27 PM, Dan Ryan notifications@github.com wrote:

Again, please stop taking random guesses and posting random snippets of your output. If you want help with a specific problem, fill out an issue with the entire issue template. We are not going on a wild goose chase with anyone in this issue while they selectively post output that is mostly unrelated to pipenv. We have seen what the screen looks like when it hangs. Pasting that to us is not useful.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/pypa/pipenv/issues/1816#issuecomment-407901419, or mute the thread https://github.com/notifications/unsubscribe-auth/ACtq-WfYHCgtRnbIu_yljQV7m0JvwxA7ks5uKOKqgaJpZM4S0kt- .

[techalchemy]

@memeplex it's not about it being a random problem, it's about the output not being helpful for troubleshooting. In no place did I call your problem random -- I told you specifically what you can do if you are interested in having a productive conversation about the issue you're facing, since it's impossible to troubleshoot or diagnose a system with selected fragments of terminal output which includes only messages we ourselves wrote and which don't actually include any errors or any way for us to reproduce it.

So far we know:

1. Nothing about your environment
2. Nothing about any of the additional circumstances surrounding your system
3. The exaggerated claim that you 'waited forever' but with no additional info about whether the process was doing anything, whether an environment existed with anything in it, what version of pipenv you are running, whether this happens with things besides numpy, how you installed python, what your hardware is like, what your system configuration is like, I could go on.

We triage a considerable number of issues, which is why we use issue templates to help us gather this information quickly to help assess what might be happening. When people simply paste '[locking]...' and tell us 'this takes forever', we feel for you, but it's kind of an early indication that you aren't actually here to be productive. I try to give the benefit of the doubt and urge people toward filing an issue with the issue template and a reproducible test case, but if you can't do that I can only assume your intention wasn't actually to troubleshoot and that you only wanted to be negative. That is not really welcome on the issue tracker, there are plenty of other places you can do that.

This tracker is used for identifying and solving problems with the codebase, not for piling on anytime you see something that you experienced. The rule is essentially that if you can't or aren't willing to contribute anything beyond restating something and telling us that we need to fix it pronto, you should probably refrain from posting. We are volunteers and we do this in our free time; if you are coming to the tracker you are directly asking for us to spend our time on you. We are usually able to do that, but we ask in return that you follow our process to help us triage efficiently.

[memeplex]

I really don't want to continue this discussion because of the other people subscribed to the thread, I'll answer this one and then you have the last saying if you so desire. I posted some anecdotal evidence, I promptly recognized it wasn't more than that and tried to eliminate the most contingent aspects by installing the local dependencies by other means and checking there were no problems with them and finally by removing every dependence altogether, starting an env from scratch and installing just numpy. Those steps (creating an env and installing numpy) are reproducible, I'm sincerely sorry I can't provide you with more useful information because I myself am unable to find any pattern, as I remarked when listing my attempts (and nor can I attach an iso of my entire setup). Thus, in the end, I suggested a relationship with a number of issues, all too similar, that were reported and closed, by that recognizing that my first impression had been wrong. I understand that reproducible is better but I can't even post a verbose output because the command just shows that minimal output that I (selectively or randomly or whatever) posted. At the beginning I thought my problem could be related to this issue and after my unsuccessful attempts to find some pattern I now feel more inclined to believe it's on the "locking is too slow" side. The reason I didn't create a new issue is that there are many of them closed by the same reason by which you certainly would have closed mine if I were to post it. Now, I think this is an important issue, I'm being unable to consistently install a popular package in a mostly standard setup, it's not about the snake emoji font not working on my new hipster terminal emulator and, while I do understand your position, I can't help feeling that you have been a bit too harsh. That said I really appreciate your detailed explanation after my last, bad tempered, comment. Btw, the claim that I 'waited forever', besides being a figure of speech, was indeed toned down by me, tongue in cheek, by defining forever as the limit of my patience, but I did provide input of it being above several minutes; since we are talking about something that should have taken a few seconds at most here, the only relevance of my imprecision is that you interpreted it as an early symptom of my ill intentions which weren't there to begin with. My apologies to everyone else here, at least to everyone else with the patience to read up to this point.

[techalchemy]

If you feel you have something new to add on this topic please create a new issue and fill out the issue template in full. We are aware of the problems and their various manifestations, and we have several fixes in process