VCS Dependency Installation fails when GIT_CONFIG is set

[techalchemy]

Environment

• pip version: 19.1.1
• Python version: 3.7.3
• OS: lubuntu 18.10

Description

Editable installs are failing in some edge cases during pipenv's test suite, seems possibly related to being in a certain level of subprocess nesting but I'm not too sure.

In nested subprocess calls (e.g. pipenv install running via pytest invoked itself via pipenv, which calls pip install with a subprocess, which then invokes pep517, git, etc, hard to know where this is failing), pip fails to install editable VCS dependencies as it is unable to find the source URL. This behavior is new as of pip 19.x and I was not able to test with earlier versions of the 19.x series as they didn't work with editable VCS dependencies for various reasons.

I know there have been changes in this part of the code lately, so I'll just go ahead and /cc @cjerdonek -- any thoughts on what's going on? This works totally fine when I run it myself even in a clean install.

Expected behavior Well, ideally this would just work, although I am not too sure what is precisely going wrong to be honest.

How to Reproduce

$ git clone git@github.com:pypa/pipenv.git
$ cd pipenv
$ git checkout -b bugfix/3809 --track origin/bugfix/3809
$ make test-specific tests=test_editable_vcs_install

Output

click here:

``` Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning. Loading .env environment variables… ===================================================================================== test session starts ===================================================================================== platform linux -- Python 3.7.3, pytest-4.6.3, py-1.8.0, pluggy-0.12.0 rootdir: /home/hawk/git/pipenv, inifile: pytest.ini, testpaths: tests plugins: devpi-server-1.7.0, server-fixtures-1.7.0, shutil-1.7.0, devpi-server-4.9.0, xdist-1.29.0, forked-1.0.2, flaky-3.5.3, tap-2.3, pypi-0.1.1 collected 251 items / 250 deselected / 1 selected tests/integration/test_install_uri.py F [100%] ========================================================================================== FAILURES =========================================================================================== __________________________________________________________________________________ test_editable_vcs_install __________________________________________________________________________________ PipenvInstance_NoPyPI = @pytest.mark.e @pytest.mark.vcs @pytest.mark.install @pytest.mark.needs_internet def test_editable_vcs_install(PipenvInstance_NoPyPI): # ! This is failing with PipenvInstance_NoPyPI(chdir=True) as p: c = p.pipenv( "install -e git+https://github.com/kennethreitz/requests.git#egg=requests --verbose" ) > assert c.return_code == 0 E AssertionError: assert 1 == 0 E + where 1 = .return_code tests/integration/test_install_uri.py:134: AssertionError ------------------------------------------------------------------------------------ Captured stdout call ------------------------------------------------------------------------------------- $ pipenv install -e git+https://github.com/kennethreitz/requests.git#egg=requests --verbose Installing -e git+https://github.com/kennethreitz/requests.git#egg=requests… Installing package: -e git+https://github.com/kennethreitz/requests.git#egg=requests Created temporary directory: /tmp/pip-ephem-wheel-cache-hbm8ev4s Created temporary directory: /tmp/pip-req-tracker-nm6qeuqm Created requirements tracker '/tmp/pip-req-tracker-nm6qeuqm' Created temporary directory: /tmp/pip-install-ednsivfo Obtaining requests from git+https://github.com/kennethreitz/requests.git#egg=requests Cloning https://github.com/kennethreitz/requests.git to ./.venv/src/requests Added requests from git+https://github.com/kennethreitz/requests.git#egg=requests to build tracker '/tmp/pip-req-tracker-nm6qeuqm' Running setup.py (path:/tmp/pipenv-e3gccq0u-project/.venv/src/requests/setup.py) egg_info for package requests Source in ./.venv/src/requests has version 2.22.0, which satisfies requirement requests from git+https://github.com/kennethreitz/requests.git#egg=requests Removed requests from git+https://github.com/kennethreitz/requests.git#egg=requests from build tracker '/tmp/pip-req-tracker-nm6qeuqm' Collecting chardet<3.1.0,>=3.0.2 (from requests) ...[pip verbose find-links output snipped]... '/tmp/pip-req-tracker-nm6qeuqm' Collecting idna<2.9,>=2.5 (from requests) ...[pip verbose find-links output snipped]... from build tracker '/tmp/pip-req-tracker-nm6qeuqm' Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests) Looking up "https://pypi.org/simple/urllib3/" in the cache ...[pip verbose find-links output snipped]... from build tracker '/tmp/pip-req-tracker-nm6qeuqm' Collecting certifi>=2017.4.17 (from requests) [...[pip verbose find-links output snipped]...'/tmp/pip-req-tracker-nm6qeuqm' Installing collected packages: chardet, idna, urllib3, certifi, requests changing mode of /tmp/pipenv-e3gccq0u-project/.venv/bin/chardetect to 775 Running setup.py develop for requests Successfully installed certifi-2019.6.16 chardet-3.0.4 idna-2.8 requests urllib3-1.25.3 Cleaning up... Removed build tracker '/tmp/pip-req-tracker-nm6qeuqm' Installing dependencies from Pipfile.lock (176165)… Created temporary directory: /tmp/pip-ephem-wheel-cache-fc7y5q_e Created temporary directory: /tmp/pip-req-tracker-lxmbdioj Created requirements tracker '/tmp/pip-req-tracker-lxmbdioj' Created temporary directory: /tmp/pip-install-g876weqg ... [pip verbose find-links output snipped]... Installed version (2019.6.16) is most up-to-date (past versions: 2019.6.16) Requirement already up-to-date: certifi==2019.6.16 in ./.venv/lib/python3.7/site-packages (from -r /tmp/pipenv-eday9hn0-requirements/pipenv-9kcfp6yy-requirement.txt (line 1)) (2019.6.16) Since it is already installed, we are trusting this package without checking its hash. To ensure a completely repeatable environment, install into an empty virtualenv. Cleaning up... Removed build tracker '/tmp/pip-req-tracker-lxmbdioj' Created temporary directory: /tmp/pip-ephem-wheel-cache-lh_xz_5_ Created temporary directory: /tmp/pip-req-tracker-o9gtsfz1 Created requirements tracker '/tmp/pip-req-tracker-o9gtsfz1' Created temporary directory: /tmp/pip-install-yween9rx ... [pip verbose find-links output snipped]... Installed version (3.0.4) is most up-to-date (past versions: 3.0.4) Requirement already up-to-date: chardet==3.0.4 in ./.venv/lib/python3.7/site-packages (from -r /tmp/pipenv-eday9hn0-requirements/pipenv-25t2vu3e-requirement.txt (line 1)) (3.0.4) Since it is already installed, we are trusting this package without checking its hash. To ensure a completely repeatable environment, install into an empty virtualenv. Cleaning up... Removed build tracker '/tmp/pip-req-tracker-o9gtsfz1' Created temporary directory: /tmp/pip-ephem-wheel-cache-3gx5t7ii Created temporary directory: /tmp/pip-req-tracker-u4m9b_y7 Created requirements tracker '/tmp/pip-req-tracker-u4m9b_y7' Created temporary directory: /tmp/pip-install-9zj6aaml ... [pip verbose output finding links snipped]... Since it is already installed, we are trusting this package without checking its hash. To ensure a completely repeatable environment, install into an empty virtualenv. Cleaning up... Removed build tracker '/tmp/pip-req-tracker-2wchoug_' Installing initially failed dependencies… Created temporary directory: /tmp/pip-ephem-wheel-cache-7qy5ch10 Created temporary directory: /tmp/pip-req-tracker-cx6miims Created requirements tracker '/tmp/pip-req-tracker-cx6miims' Created temporary directory: /tmp/pip-install-5279js0i Obtaining requests from git+https://github.com/kennethreitz/requests.git@4983a9bde39c6320aa4f3e34e50dac6e263dab6f#egg=requests Checking in /tmp/pipenv-e3gccq0u-project/.venv/src/requests for .git (git)... Cleaning up... Removed build tracker '/tmp/pip-req-tracker-cx6miims' Command failed... ------------------------------------------------------------------------------------ Captured stderr call ------------------------------------------------------------------------------------- Using python: 3.7 Path to python: /mnt/ramdisk/.venv/bin/python3.7 Creating a virtualenv for this project… Pipfile: /tmp/pipenv-e3gccq0u-project/Pipfile Using /mnt/ramdisk/.venv/bin/python3.7 (3.7.3) to create virtualenv… Already using interpreter /mnt/ramdisk/.venv/bin/python3.7 Using real prefix '/home/hawk/.pyenv/versions/3.7.3' New python executable in /tmp/pipenv-e3gccq0u-project/.venv/bin/python3.7 Also creating executable in /tmp/pipenv-e3gccq0u-project/.venv/bin/python Installing setuptools, pip, wheel... done. Running virtualenv with interpreter /mnt/ramdisk/.venv/bin/python3.7 Successfully created virtual environment! Virtualenv location: /tmp/pipenv-e3gccq0u-project/.venv Creating a Pipfile for this project… Installing 'requests' $ ['/tmp/pipenv-e3gccq0u-project/.venv/bin/pip', 'install', '--verbose', '--upgrade', '-e', 'git+https://github.com/kennethreitz/requests.git#egg=requests', '-i', 'https://pypi.org/simple'] Using source directory: '/tmp/pipenv-e3gccq0u-project/.venv/src' Adding requests to Pipfile's [packages]… Installation Succeeded Pipfile.lock not found, creating… Locking [dev-packages] dependencies… Locking [packages] dependencies… Building requirements... INFO:pipenv.patched.notpip._internal.vcs.git:Cloning https://github.com/kennethreitz/requests.git to /tmp/reqlib-srcse4lyuzz Resolving dependencies... ROUND 1 Current constraints: certifi>=2017.4.17 chardet<3.1.0,>=3.0.2 idna<2.9,>=2.5 requests from git+https://github.com/kennethreitz/requests.git#egg=requests (from -r /tmp/pipenv84p19_fkrequirements/pipenv-9cdtf0ii-constraints.txt (line 6)) urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 Finding the best candidates: found candidate certifi==2019.6.16 (constraint was >=2017.4.17) found candidate chardet==3.0.4 (constraint was >=3.0.2,<3.1.0) found candidate idna==2.8 (constraint was >=2.5,<2.9) found candidate -e git+https://github.com/kennethreitz/requests.git#egg=requests (constraint was ) found candidate urllib3==1.25.3 (constraint was >=1.21.1,!=1.25.0,!=1.25.1,<1.26) Finding secondary dependencies: chardet==3.0.4 not in cache, need to check index chardet==3.0.4 requires - urllib3==1.25.3 not in cache, need to check index urllib3==1.25.3 requires - certifi==2019.6.16 not in cache, need to check index certifi==2019.6.16 requires - idna==2.8 not in cache, need to check index idna==2.8 requires - New dependencies found in this round: adding ['certifi', '>=2017.4.17', '[]'] adding ['chardet', '<3.1.0,>=3.0.2', '[]'] adding ['idna', '<2.9,>=2.5', '[]'] adding ['urllib3', '!=1.25.0,!=1.25.1,<1.26,>=1.21.1', '[]'] Removed dependencies in this round: Unsafe dependencies in this round: ------------------------------------------------------------ Result of round 1: not stable ROUND 2 Current constraints: certifi>=2017.4.17 chardet<3.1.0,>=3.0.2 idna<2.9,>=2.5 requests from git+https://github.com/kennethreitz/requests.git#egg=requests (from -r /tmp/pipenv84p19_fkrequirements/pipenv-9cdtf0ii-constraints.txt (line 6)) urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 Finding the best candidates: found candidate certifi==2019.6.16 (constraint was >=2017.4.17) found candidate chardet==3.0.4 (constraint was >=3.0.2,<3.1.0) found candidate idna==2.8 (constraint was >=2.5,<2.9) found candidate -e git+https://github.com/kennethreitz/requests.git#egg=requests (constraint was ) found candidate urllib3==1.25.3 (constraint was >=1.21.1,!=1.25.0,!=1.25.1,<1.26) Finding secondary dependencies: urllib3==1.25.3 requires - idna==2.8 requires - certifi==2019.6.16 requires - chardet==3.0.4 requires - ------------------------------------------------------------ Result of round 2: stable, done ROUND 1 Current constraints: certifi>=2017.4.17 chardet<3.1.0,>=3.0.2 idna<2.9,>=2.5 requests from git+https://github.com/kennethreitz/requests.git#egg=requests (from -r /tmp/pipenv84p19_fkrequirements/pipenv-9cdtf0ii-constraints.txt (line 6)) urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 Finding the best candidates: found candidate certifi==2019.6.16 (constraint was >=2017.4.17) found candidate chardet==3.0.4 (constraint was >=3.0.2,<3.1.0) found candidate idna==2.8 (constraint was >=2.5,<2.9) found candidate -e git+https://github.com/kennethreitz/requests.git#egg=requests (constraint was ) found candidate urllib3==1.25.3 (constraint was >=1.21.1,!=1.25.0,!=1.25.1,<1.26) Finding secondary dependencies: chardet==3.0.4 not in cache, need to check index chardet==3.0.4 requires - urllib3==1.25.3 not in cache, need to check index urllib3==1.25.3 requires - certifi==2019.6.16 not in cache, need to check index certifi==2019.6.16 requires - idna==2.8 not in cache, need to check index idna==2.8 requires - New dependencies found in this round: adding ['certifi', '>=2017.4.17', '[]'] adding ['chardet', '<3.1.0,>=3.0.2', '[]'] adding ['idna', '<2.9,>=2.5', '[]'] adding ['urllib3', '!=1.25.0,!=1.25.1,<1.26,>=1.21.1', '[]'] Removed dependencies in this round: Unsafe dependencies in this round: ------------------------------------------------------------ Result of round 1: not stable ROUND 2 Current constraints: certifi>=2017.4.17 chardet<3.1.0,>=3.0.2 idna<2.9,>=2.5 requests from git+https://github.com/kennethreitz/requests.git#egg=requests (from -r /tmp/pipenv84p19_fkrequirements/pipenv-9cdtf0ii-constraints.txt (line 6)) urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 Finding the best candidates: found candidate certifi==2019.6.16 (constraint was >=2017.4.17) found candidate chardet==3.0.4 (constraint was >=3.0.2,<3.1.0) found candidate idna==2.8 (constraint was >=2.5,<2.9) found candidate -e git+https://github.com/kennethreitz/requests.git#egg=requests (constraint was ) found candidate urllib3==1.25.3 (constraint was >=1.21.1,!=1.25.0,!=1.25.1,<1.26) Finding secondary dependencies: urllib3==1.25.3 requires - idna==2.8 requires - certifi==2019.6.16 requires - chardet==3.0.4 requires - ------------------------------------------------------------ Result of round 2: stable, done Success! Updated Pipfile.lock (176165)! Writing supplied requirement line to temporary file: 'certifi==2019.6.16 --hash=sha256:046832c04d4e752f37383b628bc601a7ea7211496b4638f6514d0e5b9acc4939 --hash=sha256:945e3ba63a0b9f577b1395204e13c3a231f9bc0223888be653286534e5873695' Installing 'certifi' $ ['/tmp/pipenv-e3gccq0u-project/.venv/bin/pip', 'install', '--verbose', '--upgrade', '--require-hashes', '--no-deps', '-r', '/tmp/pipenv-eday9hn0-requirements/pipenv-9kcfp6yy-requirement.txt', '-i', 'https://pypi.org/simple'] Using source directory: '/tmp/pipenv-e3gccq0u-project/.venv/src' Writing supplied requirement line to temporary file: 'chardet==3.0.4 --hash=sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691 --hash=sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae' Installing 'chardet' $ ['/tmp/pipenv-e3gccq0u-project/.venv/bin/pip', 'install', '--verbose', '--upgrade', '--require-hashes', '--no-deps', '-r', '/tmp/pipenv-eday9hn0-requirements/pipenv-25t2vu3e-requirement.txt', '-i', 'https://pypi.org/simple'] Using source directory: '/tmp/pipenv-e3gccq0u-project/.venv/src' Writing supplied requirement line to temporary file: 'idna==2.8 --hash=sha256:c357b3f628cf53ae2c4c05627ecc484553142ca23264e593d327bcde5e9c3407 --hash=sha256:ea8b7f6188e6fa117537c3df7da9fc686d485087abf6ac197f9c46432f7e4a3c' Installing 'idna' $ ['/tmp/pipenv-e3gccq0u-project/.venv/bin/pip', 'install', '--verbose', '--upgrade', '--require-hashes', '--no-deps', '-r', '/tmp/pipenv-eday9hn0-requirements/pipenv-wz2pyxfh-requirement.txt', '-i', 'https://pypi.org/simple'] Using source directory: '/tmp/pipenv-e3gccq0u-project/.venv/src' Installing 'requests' $ ['/tmp/pipenv-e3gccq0u-project/.venv/bin/pip', 'install', '--verbose', '--upgrade', '--no-deps', '-e', 'git+https://github.com/kennethreitz/requests.git@4983a9bde39c6320aa4f3e34e50dac6e263dab6f#egg=requests', '-i', 'https://pypi.org/simple'] Using source directory: '/tmp/pipenv-e3gccq0u-project/.venv/src' Writing supplied requirement line to temporary file: "urllib3==1.25.3 ; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3' and python_version < '4' --hash=sha256:b246607a25ac80bedac05c6f282e3cdaf3afb65420fd024ac94435cabe6e18d1 --hash=sha256:dbe59173209418ae49d485b87d1681aefa36252ee85884c31346debd19463232" Installing 'urllib3' $ ['/tmp/pipenv-e3gccq0u-project/.venv/bin/pip', 'install', '--verbose', '--upgrade', '--require-hashes', '--no-deps', '-r', '/tmp/pipenv-eday9hn0-requirements/pipenv-0gevanyv-requirement.txt', '-i', 'https://pypi.org/simple'] Using source directory: '/tmp/pipenv-e3gccq0u-project/.venv/src' ***** ERROR LOG: Running command git config --get-regexp 'remote\..*\.url' ERROR: Exception: Traceback (most recent call last): File "/tmp/pipenv-e3gccq0u-project/.venv/lib/python3.7/site-packages/pip/_internal/vcs/git.py", line 272, in get_remote_url found_remote = remotes[0] IndexError: list index out of range During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/tmp/pipenv-e3gccq0u-project/.venv/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 178, in main status = self.run(options, args) File "/tmp/pipenv-e3gccq0u-project/.venv/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 352, in run resolver.resolve(requirement_set) File "/tmp/pipenv-e3gccq0u-project/.venv/lib/python3.7/site-packages/pip/_internal/resolve.py", line 131, in resolve self._resolve_one(requirement_set, req) File "/tmp/pipenv-e3gccq0u-project/.venv/lib/python3.7/site-packages/pip/_internal/resolve.py", line 294, in _resolve_one abstract_dist = self._get_abstract_dist_for(req_to_install) File "/tmp/pipenv-e3gccq0u-project/.venv/lib/python3.7/site-packages/pip/_internal/resolve.py", line 226, in _get_abstract_dist_for req, self.require_hashes, self.use_user_site, self.finder, File "/tmp/pipenv-e3gccq0u-project/.venv/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 391, in prepare_editable_requirement req.update_editable(not self._download_should_save) File "/tmp/pipenv-e3gccq0u-project/.venv/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 793, in update_editable vcs_backend.obtain(self.source_dir) File "/tmp/pipenv-e3gccq0u-project/.venv/lib/python3.7/site-packages/pip/_internal/vcs/__init__.py", line 442, in obtain existing_url = self.get_remote_url(dest) File "/tmp/pipenv-e3gccq0u-project/.venv/lib/python3.7/site-packages/pip/_internal/vcs/git.py", line 274, in get_remote_url raise RemoteNotFoundError pip._internal.vcs.RemoteNotFoundError ***** OUTPUT LOG: Created temporary directory: /tmp/pip-ephem-wheel-cache-q26j1sh1 Created temporary directory: /tmp/pip-req-tracker-emwse09n Created requirements tracker '/tmp/pip-req-tracker-emwse09n' Created temporary directory: /tmp/pip-install-6t56vjqz Obtaining requests from git+https://github.com/kennethreitz/requests.git@4983a9bde39c6320aa4f3e34e50dac6e263dab6f#egg=requests Checking in /tmp/pipenv-e3gccq0u-project/.venv/src/requests for .git (git)... Cleaning up... Removed build tracker '/tmp/pip-req-tracker-emwse09n' INFO:pipenv.patched.notpip._internal.vcs.git:Cloning https://github.com/kennethreitz/requests.git (to revision 4983a9bde39c6320aa4f3e34e50dac6e263dab6f) to /tmp/reqlib-srcqe0bgu33 An error occurred while installing -e git+https://github.com/kennethreitz/requests.git@4983a9bde39c6320aa4f3e34e50dac6e263dab6f#egg=requests! Will try again. Installing 'requests' $ ['/tmp/pipenv-e3gccq0u-project/.venv/bin/pip', 'install', '--verbose', '--upgrade', '--no-use-pep517', '--no-deps', '-e', 'git+https://github.com/kennethreitz/requests.git@4983a9bde39c6320aa4f3e34e50dac6e263dab6f#egg=requests', '-i', 'https://pypi.org/simple'] Using source directory: '/tmp/pipenv-e3gccq0u-project/.venv/src' ***** ERROR LOG: Running command git config --get-regexp 'remote\..*\.url' ERROR: Exception: Traceback (most recent call last): File "/tmp/pipenv-e3gccq0u-project/.venv/lib/python3.7/site-packages/pip/_internal/vcs/git.py", line 272, in get_remote_url found_remote = remotes[0] IndexError: list index out of range During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/tmp/pipenv-e3gccq0u-project/.venv/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 178, in main status = self.run(options, args) File "/tmp/pipenv-e3gccq0u-project/.venv/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 352, in run resolver.resolve(requirement_set) File "/tmp/pipenv-e3gccq0u-project/.venv/lib/python3.7/site-packages/pip/_internal/resolve.py", line 131, in resolve self._resolve_one(requirement_set, req) File "/tmp/pipenv-e3gccq0u-project/.venv/lib/python3.7/site-packages/pip/_internal/resolve.py", line 294, in _resolve_one abstract_dist = self._get_abstract_dist_for(req_to_install) File "/tmp/pipenv-e3gccq0u-project/.venv/lib/python3.7/site-packages/pip/_internal/resolve.py", line 226, in _get_abstract_dist_for req, self.require_hashes, self.use_user_site, self.finder, File "/tmp/pipenv-e3gccq0u-project/.venv/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 391, in prepare_editable_requirement req.update_editable(not self._download_should_save) File "/tmp/pipenv-e3gccq0u-project/.venv/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 793, in update_editable vcs_backend.obtain(self.source_dir) File "/tmp/pipenv-e3gccq0u-project/.venv/lib/python3.7/site-packages/pip/_internal/vcs/__init__.py", line 442, in obtain existing_url = self.get_remote_url(dest) File "/tmp/pipenv-e3gccq0u-project/.venv/lib/python3.7/site-packages/pip/_internal/vcs/git.py", line 274, in get_remote_url raise RemoteNotFoundError pip._internal.vcs.RemoteNotFoundError ***** OUTPUT LOG: Created temporary directory: /tmp/pip-ephem-wheel-cache-7qy5ch10 Created temporary directory: /tmp/pip-req-tracker-cx6miims Created requirements tracker '/tmp/pip-req-tracker-cx6miims' Created temporary directory: /tmp/pip-install-5279js0i Obtaining requests from git+https://github.com/kennethreitz/requests.git@4983a9bde39c6320aa4f3e34e50dac6e263dab6f#egg=requests Checking in /tmp/pipenv-e3gccq0u-project/.venv/src/requests for .git (git)... Cleaning up... ```

[cjerdonek]

For the repository directory where it's checking for the remote and getting RemoteNotFoundError, I would put some debug statements to (1) find out why the git config command isn't returning any remotes (does the repo not have any remotes?), and then (2) work backwards to find out how that repo was created, and why it's not in the expected state. From the logs, it looks like patched versions of things are being used (e.g. "INFO:pipenv.patched.notpip._internal.vcs.git:Cloning https://github.com/kennethreitz/requests.git"), so it's not clear whether pip is really to blame here..

[cjerdonek]

I see now that this is pip's code being invoked. Are you doing anything special in the way that you're invoking the subprocess? Git's clone documentation says this, so it should be creating remotes:

This default configuration is achieved by creating references to the remote branch heads under refs/remotes/origin and by initializing remote.origin.url and remote.origin.fetch configuration variables.

(from https://git-scm.com/docs/git-clone#_description )

[cjerdonek]

Notice that when it's failing to install requests, it has the words "Checking in" instead of "Cloning https://github.com/kennethreitz/requests.git to ./.venv/src/requests", so the Git repo appears to be left over from previously during the execution. Could something have cleaned up the info about the remotes?

First time--

Obtaining requests from git+https://github.com/kennethreitz/requests.git#egg=requests
  Cloning https://github.com/kennethreitz/requests.git to ./.venv/src/requests
  Added requests from git+https://github.com/kennethreitz/requests.git#egg=requests to build tracker '/tmp/pip-req-tracker-nm6qeuqm'
  ...
  Removed requests from git+https://github.com/kennethreitz/requests.git#egg=requests from build tracker '/tmp/pip-req-tracker-nm6qeuqm'

Later in the process--

Obtaining requests from git+https://github.com/kennethreitz/requests.git@4983a9bde39c6320aa4f3e34e50dac6e263dab6f#egg=requests
  Checking in /tmp/pipenv-e3gccq0u-project/.venv/src/requests for .git (git)...
Cleaning up...

Basically, pip needs to decide what to do if it's trying to install something in editable mode into a directory that already has contents (like a Git repo). Should it choose a different directory? (Does pip have a choice?) Should it write into that same directory?

[cjerdonek]

I think a workaround / way for you to avoid this could be to avoid installing requests in editable mode in the same directory that pip will be using later on in the process. Or else be sure the directory is empty by the time pip gets to it if you were the one that created it and you know you can delete it.

[techalchemy]

Hmmm that's helpful, has this behavior changed at all in the last ~3-5 releases that you're aware of? I haven't noticed changes around this, but I haven't followed that closely. In the past, I was pretty sure pip would just use the existing clone if it was there. I clone the source there myself (using a patched version of pip as you noticed) in order to resolve dependencies on it. My hope was that if I cloned to the same place pip is going to use, I could avoid excessive network traffic.

Is there any logic in pip right now to handle the case when a user has installed a VCS dependency in editable mode and re-executes pip install -e <vcs-dependency-url>@<some-other-ref>#egg=<name> that doesn't involve simply uninstalling and re-cloning the entire repository? If so, how can I trick pip into doing this for me?

[cjerdonek]

No, I don't think anything has changed around this (except perhaps that pip freeze is more relaxed in that it doesn't require VCS repositories to have a remote to display a more detailed URL, but that doesn't affect you here).

Yes, pip should already work if there's an existing clone. It even has logic in one corner of its code to "fix" the remote url if it finds that it doesn't match. The pip reinstall case you described should work for pip, and the reason is that the first time that pip does a clone, it will create the remotes (Git does this automatically by default), and then when pip inspects the repo directory the second time, it will find the remotes still there, so it should be able to pick things up normally. The issue here that I tried to communicate above is that the remotes aren't present in your case when you try to install the second time. So I would look into (1) are any remotes actually present the second time (or is pip misreporting this), (2) if remotes aren't present, are they getting added the first time (and if not, why not), and (3) if they were added, are they getting deleted somewhere in the process (e.g. in any of the "cleaning" stages you see in the logs).

[cjerdonek]

For example, when pipenv does its clone, does it pass any environment variables that could affect Git's behavior?

[cjerdonek]

So that you can see it's working as far as I can tell, here's what I get when I install a VCS url in editable mode twice in succession as you described (from pip master). Notice the difference in the logs for the second time from what you're getting.

$ pip install -v -e git+https://github.com/kennethreitz/requests.git#egg=requests
Created temporary directory: /private/var/folders/q9/j0_5hxt88v5592006s6dd3n80000gs/T/pip-ephem-wheel-cache-iwrs4qfh
Created temporary directory: /private/var/folders/q9/j0_5hxt88v5592006s6dd3n80000gs/T/pip-req-tracker-g2c_yppj
Created requirements tracker '/private/var/folders/q9/j0_5hxt88v5592006s6dd3n80000gs/T/pip-req-tracker-g2c_yppj'
Created temporary directory: /private/var/folders/q9/j0_5hxt88v5592006s6dd3n80000gs/T/pip-install-osa6jxr3
Obtaining requests from git+https://github.com/kennethreitz/requests.git#egg=requests
  Cloning https://github.com/kennethreitz/requests.git to /.../envs/pip/src/requests
  Running command git clone -q https://github.com/kennethreitz/requests.git /.../envs/pip/src/requests
  Added requests from git+https://github.com/kennethreitz/requests.git#egg=requests to build tracker '/private/var/folders/q9/j0_5hxt88v5592006s6dd3n80000gs/T/pip-req-tracker-g2c_yppj'
    Running setup.py (path:/.../envs/pip/src/requests/setup.py) egg_info for package requests
    ...
  Removed requests from git+https://github.com/kennethreitz/requests.git#egg=requests from build tracker '/private/var/folders/q9/j0_5hxt88v5592006s6dd3n80000gs/T/pip-req-tracker-g2c_yppj'
...
Successfully installed requests
Cleaning up...
...
$ pip install -v -e git+https://github.com/kennethreitz/requests.git@2e4329d153a01407be6a7d1217f1ea25b14138bc#egg=requests
Created temporary directory: /private/var/folders/q9/j0_5hxt88v5592006s6dd3n80000gs/T/pip-ephem-wheel-cache-ogu6t195
Created temporary directory: /private/var/folders/q9/j0_5hxt88v5592006s6dd3n80000gs/T/pip-req-tracker-mr9pqkn8
Created requirements tracker '/private/var/folders/q9/j0_5hxt88v5592006s6dd3n80000gs/T/pip-req-tracker-mr9pqkn8'
Created temporary directory: /private/var/folders/q9/j0_5hxt88v5592006s6dd3n80000gs/T/pip-install-3z3u6fz5
Obtaining requests from git+https://github.com/kennethreitz/requests.git@2e4329d153a01407be6a7d1217f1ea25b14138bc#egg=requests
  Checking in /.../envs/pip/src/requests for .git (git)...
  Running command git config --get-regexp 'remote\..*\.url'
  remote.origin.url https://github.com/kennethreitz/requests.git
  Clone in /.../envs/pip/src/requests exists, and has correct URL (https://github.com/kennethreitz/requests.git)
  Running command git rev-parse HEAD
  4983a9bde39c6320aa4f3e34e50dac6e263dab6f
  Updating /.../envs/pip/src/requests clone (to revision 2e4329d153a01407be6a7d1217f1ea25b14138bc)
  Running command git version
  git version 2.16.2
  Running command git fetch -q --tags
  Running command git show-ref 2e4329d153a01407be6a7d1217f1ea25b14138bc
  Running command git reset --hard -q 2e4329d153a01407be6a7d1217f1ea25b14138bc
  Added requests from git+https://github.com/kennethreitz/requests.git@2e4329d153a01407be6a7d1217f1ea25b14138bc#egg=requests to build tracker '/private/var/folders/q9/j0_5hxt88v5592006s6dd3n80000gs/T/pip-req-tracker-mr9pqkn8'
    Running setup.py (path:/.../envs/pip/src/requests/setup.py) egg_info for package requests
    ...
  Removed requests from git+https://github.com/kennethreitz/requests.git@2e4329d153a01407be6a7d1217f1ea25b14138bc#egg=requests from build tracker '/private/var/folders/q9/j0_5hxt88v5592006s6dd3n80000gs/T/pip-req-tracker-mr9pqkn8'
...
Successfully installed requests
Cleaning up...
...

[cjerdonek]

Also, if I manually remove the origin remote from the requests repo and try again, here is what I get.

$ pip install -v -e git+https://github.com/kennethreitz/requests.git@6f81e78dfe0b7419bde97cae594febfffab8e431#egg=requests
Created temporary directory: /private/var/folders/q9/j0_5hxt88v5592006s6dd3n80000gs/T/pip-ephem-wheel-cache-nyyk2v04
Created temporary directory: /private/var/folders/q9/j0_5hxt88v5592006s6dd3n80000gs/T/pip-req-tracker-zmt_rva9
Created requirements tracker '/private/var/folders/q9/j0_5hxt88v5592006s6dd3n80000gs/T/pip-req-tracker-zmt_rva9'
Created temporary directory: /private/var/folders/q9/j0_5hxt88v5592006s6dd3n80000gs/T/pip-install-djxg84oa
Obtaining requests from git+https://github.com/kennethreitz/requests.git@6f81e78dfe0b7419bde97cae594febfffab8e431#egg=requests
  Checking in /.../envs/pip/src/requests for .git (git)...
  Running command git config --get-regexp 'remote\..*\.url'
Cleaning up...
Removed build tracker '/private/var/folders/q9/j0_5hxt88v5592006s6dd3n80000gs/T/pip-req-tracker-zmt_rva9'
ERROR: Exception:
...

[techalchemy]

Sorry for being slow, we set several environment variables so I figured this would be easier with a bunch of debug info.

Firstly, here are the environment variables passed into the subprocess call:

Using source directory: '/tmp/pipenv-7_ibrn8d-project/.venv/src'
*** Environment found:
    PIP_CACHE_DIR: /tmp/pipenv-hofugbgb-cache
    PIP_WHEEL_DIR: /tmp/pipenv-hofugbgb-cache/wheels
    PIP_DESTINATION_DIR: /tmp/pipenv-hofugbgb-cache/pkgs
    PIP_EXISTS_ACTION: w
    PATH: /tmp/tmp.lfbEmdLF3g/pipenv_venv/bin:/tmp/tmp.lfbEmdLF3g/pipenv_venv/bin:/home/hawk/.pyenv/shims:/home/hawk/.pyenv/shims:/home/hawk/.pyenv/bin:/home/hawk/.local/bin:/home/hawk/bin:/home/hawk/go/bin:/home/hawk/go/bin:/usr/bin:/home/hawk/.pyenv/libexec:/home/hawk/.pyenv/plugins/python-build/bin:/home/hawk/.pyenv/plugins/pyenv-virtualenv/bin:/home/hawk/.pyenv/plugins/pyenv-update/bin:/home/hawk/.pyenv/plugins/pyenv-installer/bin:/home/hawk/.pyenv/plugins/pyenv-doctor/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
    PIP_SRC: /tmp/pipenv-7_ibrn8d-project/.venv/src

And I threw in an exception handler to pick up the directory this code gets invoked from in case that should matter:

*** FAILED IN DIRECTORY: /tmp/pipenv-7_ibrn8d-project

At that point the exception handler checks the contents of PIP_SRC to see if I cloned anything there that might cause this issue (in this case we see requests is there already, so there is some collision happening as you say)

*** SRC DIR CONTENTS: requests

Then I listed out the contents of the directory we care about just to see what it contains in case something weird is going on there...

*** PACKAGE SRC DIR CONTENTS: pytest.ini
    setup.cfg
    .travis.yml
    docs
    HISTORY.md
    _appveyor
    .git
    Pipfile
    ext
    requests.egg-info
    MANIFEST.in
    requests
    CODE_OF_CONDUCT.md
    README.md
    appveyor.yml
    LICENSE
    .github
    tox.ini
    CONTRIBUTING.md
    .gitignore
    tests
    .coveragerc
    setup.py
    Pipfile.lock
    Makefile
    AUTHORS.rst

And finally I am including the git config contents from that directory which I wanted to use to verify that the clone happened correctly:

*** GIT CONFIG CONTENTS: [core]
    repositoryformatversion = 0
    filemode = true
    bare = false
    logallrefupdates = true
[remote "origin"]
    url = https://github.com/kennethreitz/requests.git
    fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
    remote = origin
    merge = refs/heads/master

So I guess where I'm at on this is -- it looks correct to me. However, this is not the first time I've seen issues with git repository directories and permissions, especially as relates to subprocess hierarchies. Git is pretty aggressive about acquiring locks on its indexes, and if somewhere along the way we are not technically done cleaning that up I guess I could see it causing issues... do you know anything about that? What I know is mostly vague and pieced together over many build failures...

[techalchemy]

Also I want to be clear that this all works completely fine as far as I can tell exactly the way it is right now in pipenv too, the only thing that is failing for me is the test suite and I'm just trying to assess whether this is some bizarre circumstance about the test suite in particular or if it is actually a legitimate situation that simply isn't happening for the people who have tested it manually

[cjerdonek]

And finally I am including the git config contents from that directory which I wanted to use to verify that the clone happened correctly:

And how about the result after the first clone of executing the git-config invocation that pip uses: git config --get-regexp 'remote\..*\.url'? Is it working the first time and not the second, or not even after the first?

[techalchemy]

OK, here is the most interesting output yet

I have the same issue running --get-regexp as you do:

**** Ran command: git config --get-regexp 'remote\..*\.url'
**** Command status: 1
**** Command output: 
**** Commaned stderr:

However, get-url works fine (invoked directly after):

**** Ran command: git remote get-url origin
**** Command status: 0
**** Command output: https://github.com/kennethreitz/requests.git

Interestingly I've done a bunch of testing and there is basically no way I can get the get-regexp command to work while the current interpreter is running, but if I for instance run pipenv install... myself, and then I run an interactive session and use the git config --get-regexp... command with cwd=<virtualenv_src_dir>/requests/, the command runs successfully.

I still feel like this is a permissions or filesystem lock thing and that the git clone subprocess(es) are not being allowed to clean up before their parents are exiting? I tried to account for that, I even added a permission-setting bit of code as well as a time.sleep(5) before trying and it still fails:

**** Ran command: git config --get-regexp 'remote\..*\.url'
**** Command status: 1
**** Command output: 
**** Commaned stderr: 

[cjerdonek]

I have the same issue running --get-regexp as you do:

Just to clarify, I don't have any issues running the command!

A couple more things I would try: What version of git is it (git --version)? Is the version different when running your tests? What about running a bare git config (no options, should see a usage string) or git config -l? Do you still get no output? I would also try other combinations like --global and --show-origin.

[techalchemy]

Sorry, typing the comment over the course of several hours -- what I meant was that I have the same issue as 'you' where 'you' == 'pip' (sorry, I do not equate your identity with this codebase but it's been a long week)

That last series of commands however, was incredibly helpful.

You'll see the error message that pops up: error: only one config file at a time

This is because in the pipenv test suite we set GIT_CONFIG to a file we write for isolation. If I unset that the tests actually pass! So that's awesome, thanks for working through that with me

--version: (this does align with my version)

**** Ran command: $ git --version
**** Command status: 0
**** Command output: git version 2.19.1

config:

**** Ran command: $ git config
**** Command status: 129
**** Command output: 
**** Commaned stderr: usage: git config [<options>]

Config file location
    --global              use global config file
    --system              use system config file
    --local               use repository config file
    -f, --file <file>     use given config file
    --blob <blob-id>      read config from given blob object
[...]

config -l:

**** Ran command: $ git config -l
**** Command status: 0
**** Command output: user.name=pipenv
user.email=pipenv@pipenv.org

config --global:

**** Commaned stderr: 
**** Ran command: $ git config --global
**** Command status: 129
**** Command output: 
**** Commaned stderr: error: only one config file at a time
usage: git config [<options>]

--show-origin:

**** Ran command: $ git config --show-origin
**** Command status: 129
**** Command output: 
**** Commaned stderr: usage: git config [<options>]

[techalchemy]

One thing you may want to consider is temporarily unsetting GIT_CONFIG if it's set when you run that command

[cjerdonek]

Great! Re: options like --global and --show-origin, though, what I meant was to suggest that they be tried in conjunction with appropriate commands. (You will always get non-zero exit status when trying them in isolation I found.) So the things to try:

$ git config -l --global
$ git config -l --show-origin
$ git config -l --local

(I'm especially curious to know if passing --local will cause the original --get-regexp variant to work, even in the presence of a GIT_CONFIG.)

[techalchemy]

The output is the same, because of the environment override from setting GIT_CONFIG. You can test it out, just make a fake config file (e.g. /tmp/gitconfig and set GIT_CONFIG to point at it, move into a repository directory and try anything out:

 ~/g/pipenv   bugfix/3809     echo "[user]
                                       name = pipenv
                                       email = pipenv@pipenv.org" > /tmp/gitconfig
 ~/g/pipenv   bugfix/3809     set -gx GIT_CONFIG /tmp/gitconfig
 ~/g/pipenv   bugfix/3809     git config -l --local
error: only one config file at a time
usage: git config [<options>]
...

The only command that works is git config -l --show-origin, so not too sure what to make of that.

[cjerdonek]

Do you need to be passing GIT_CONFIG to pip for some reason? Passing GIT_CONFIG is a way of telling the command to ignore a repository's .git/config. I'm wondering if pip should respect that.

[pradyunsg]

If you do, I think it might be a good idea to "include" the original config file:

[include]
    path = /path/to/foo/.git/config  # include by absolute path