Install from Git access denied on Windows

joshkreud commented 3 years ago

Issue description

Hello,

I'm experiencing some issues Trying to get packages installed using git as the source. In this example I just took the setuptools Library but it seems to happen with other Git Repos too.

Possibly Related: #3391

Take the following command in a new Directory:

pipenv install git+https://github.com/pypa/setuptools.git#egg=setuptools

The Outcome will not change, even when suppling the -e/editable=true flag.

When looking at the Temp folder specified in the err.log, it seems a cleanup is failing due to .git files being locked by another Process. (Maybe git)

Expected result

A Environment to be created and setuptools (or whatever library) to be installed.

Actual result

$pipenv install git+https://github.com/pypa/setuptools.git#egg=setuptools

Creating a Pipfile for this project...
Installing git+https://github.com/pypa/setuptools.git#egg=setuptools...
Adding setuptools to Pipfile's [packages]...
Installation Succeeded
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
 Locking...Building requirements...
Resolving dependencies...
Success!
Warning: Exception in thread Thread-30:
Traceback (most recent call last):
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\lib\threading.py", line 932, in _bootstrap_inner
    self.run()
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\lib\subprocess.py", line 1366, in _readerthread
    buffer.append(fh.read())
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\lib\codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x81 in position 121: invalid start byte
Traceback (most recent call last):
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\lib\shutil.py", line 613, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Zugriff verweigert: 'C:\\Users\\joshi\\AppData\\Local\\Temp\\requirementslib-g7u_2aoe-src\\setuptools\\.git\\objects\\pack\\pack-5dc29dadfb20dc32be55969af763b231d355b87e.idx'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python39\Lib\site-packages\pipenv\vendor\vistir\contextmanagers.py", line 205, in spinner
    yield _spinner
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python39\Lib\site-packages\pipenv\vendor\vistir\misc.py", line 618, in run
    return _create_subprocess(
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python39\Lib\site-packages\pipenv\vendor\vistir\misc.py", line 542, in _create_subprocess
    c.out, c.err = c.communicate()
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\lib\subprocess.py", line 1024, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\lib\subprocess.py", line 1418, in _communicate
    stderr = stderr[0]
IndexError: list index out of range

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\lib\shutil.py", line 613, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Zugriff verweigert: 'C:\\Users\\joshi\\AppData\\Local\\Temp\\requirementslib-g7u_2aoe-src\\setuptools\\.git\\objects\\pack\\pack-5dc29dadfb20dc32be55969af763b231d355b87e.idx'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python39\Lib\site-packages\pipenv\vendor\vistir\path.py", line 433, in rmtree
    shutil.rmtree(directory, ignore_errors=ignore_errors, onerror=onerror)
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\lib\shutil.py", line 737, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\lib\shutil.py", line 610, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\lib\shutil.py", line 610, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\lib\shutil.py", line 610, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  [Previous line repeated 1 more time]
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\lib\shutil.py", line 615, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python39\Lib\site-packages\pipenv\vendor\vistir\path.py", line 496, in handle_remove_readonly
    set_write_bit(path)
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python39\Lib\site-packages\pipenv\vendor\vistir\path.py", line 383, in set_write_bit
    c = run(
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python39\Lib\site-packages\pipenv\vendor\vistir\misc.py", line 618, in run
    return _create_subprocess(
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python39\Lib\site-packages\pipenv\vendor\vistir\misc.py", line 542, in _create_subprocess
    c.out, c.err = c.communicate()
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\lib\subprocess.py", line 1024, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\lib\subprocess.py", line 1418, in _communicate
    stderr = stderr[0]
IndexError: list index out of range
Exception in thread Thread-32:
Traceback (most recent call last):
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\lib\threading.py", line 932, in _bootstrap_inner
    self.run()
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\lib\subprocess.py", line 1366, in _readerthread
    buffer.append(fh.read())
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\lib\codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x81 in position 121: invalid start byte
Traceback (most recent call last):
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\lib\shutil.py", line 613, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Zugriff verweigert: 'C:\\Users\\joshi\\AppData\\Local\\Temp\\pipenv-69le28rx-src\\setuptools\\.git\\objects\\pack\\pack-3c69dcebc05cd3a92fd6f1047706e9a45e6c893c.idx'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\joshi\appdata\local\programs\python\python39\lib\site-packages\pipenv\vendor\vistir\contextmanagers.py", line 205, in spinner
    yield _spinner
  File "c:\users\joshi\appdata\local\programs\python\python39\lib\site-packages\pipenv\vendor\vistir\misc.py", line 618, in run
    return _create_subprocess(
  File "c:\users\joshi\appdata\local\programs\python\python39\lib\site-packages\pipenv\vendor\vistir\misc.py", line 542, in _create_subprocess
    c.out, c.err = c.communicate()
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\lib\subprocess.py", line 1024, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\lib\subprocess.py", line 1418, in _communicate
    stderr = stderr[0]
IndexError: list index out of range

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\lib\shutil.py", line 613, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Zugriff verweigert: 'C:\\Users\\joshi\\AppData\\Local\\Temp\\pipenv-69le28rx-src\\setuptools\\.git\\objects\\pack\\pack-3c69dcebc05cd3a92fd6f1047706e9a45e6c893c.idx'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\joshi\appdata\local\programs\python\python39\lib\site-packages\pipenv\vendor\vistir\path.py", line 433, in rmtree
    shutil.rmtree(directory, ignore_errors=ignore_errors, onerror=onerror)
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\lib\shutil.py", line 737, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\lib\shutil.py", line 610, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\lib\shutil.py", line 610, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\lib\shutil.py", line 610, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  [Previous line repeated 1 more time]
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\lib\shutil.py", line 615, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "c:\users\joshi\appdata\local\programs\python\python39\lib\site-packages\pipenv\vendor\vistir\path.py", line 496, in handle_remove_readonly
    set_write_bit(path)
  File "c:\users\joshi\appdata\local\programs\python\python39\lib\site-packages\pipenv\vendor\vistir\path.py", line 383, in set_write_bit
    c = run(
  File "c:\users\joshi\appdata\local\programs\python\python39\lib\site-packages\pipenv\vendor\vistir\misc.py", line 618, in run
    return _create_subprocess(
  File "c:\users\joshi\appdata\local\programs\python\python39\lib\site-packages\pipenv\vendor\vistir\misc.py", line 542, in _create_subprocess
    c.out, c.err = c.communicate()
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\lib\subprocess.py", line 1024, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\lib\subprocess.py", line 1418, in _communicate
    stderr = stderr[0]
IndexError: list index out of range

Steps to replicate

pipenv install When the Pipfile already existst. Otherwihse the install command from above.

$ pipenv --support

Pipenv version: `'2020.11.15'` Pipenv location: `'c:\\users\\joshi\\appdata\\local\\programs\\python\\python39\\lib\\site-packages\\pipenv'` Python location: `'c:\\users\\joshi\\appdata\\local\\programs\\python\\python39\\python.exe'` Python installations found: - `3.9.1`: `C:\Users\joshi\AppData\Local\Programs\Python\Python39\python.exe` - `3.8.5`: `C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\python.exe` - `3.6.6`: `C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\python.exe` - `3.8.3`: `C:\Users\joshi\Miniconda3\python.exe` - `3.6.5`: `C:\Program Files (x86)\Microsoft Visual Studio\Shared\Anaconda3_64\python.exe` PEP 508 Information: ``` {'implementation_name': 'cpython', 'implementation_version': '3.9.1', 'os_name': 'nt', 'platform_machine': 'AMD64', 'platform_python_implementation': 'CPython', 'platform_release': '10', 'platform_system': 'Windows', 'platform_version': '10.0.18362', 'python_full_version': '3.9.1', 'python_version': '3.9', 'sys_platform': 'win32'} ``` System environment variables: - `ALLUSERSPROFILE` - `APPDATA` - `COMMONPROGRAMFILES` - `COMMONPROGRAMFILES(X86)` - `COMMONPROGRAMW6432` - `COMPUTERNAME` - `COMSPEC` - `DRIVERDATA` - `ELM` - `GOPATH` - `HOMEDRIVE` - `HOMEPATH` - `IWBPATH` - `JRE_HOME` - `LOCALAPPDATA` - `LOGONSERVER` - `LOTUS_NOTES_HOMESERVER` - `MSMPI_BIN` - `NUMBER_OF_PROCESSORS` - `ONEDRIVE` - `ONEDRIVECOMMERCIAL` - `OS` - `PATH` - `PATHEXT` - `PROCESSOR_ARCHITECTURE` - `PROCESSOR_IDENTIFIER` - `PROCESSOR_LEVEL` - `PROCESSOR_REVISION` - `PROGRAMDATA` - `PROGRAMFILES` - `PROGRAMFILES(X86)` - `PROGRAMW6432` - `PSMODULEPATH` - `PUBLIC` - `RFC_TRACE_DIR` - `SESSIONNAME` - `SNC_LIB` - `SNC_LIB_64` - `SNOW_AGENT` - `SPLM_LICENSE_SERVER` - `SW_SIM_HYDRA` - `SW_SIM_MPIT` - `SW_SIM_TEMP` - `SYSTEMDRIVE` - `SYSTEMROOT` - `TCVIS_CLUSTER_PATH` - `TEMP` - `TMP` - `UGII_3DCONNEXION_DIR` - `UGII_3DCONNEXION_DIR32` - `UGII_BASE_DIR` - `UGII_LANG` - `USERDNSDOMAIN` - `USERDOMAIN` - `USERDOMAIN_ROAMINGPROFILE` - `USERNAME` - `USERPROFILE` - `VBOX_MSI_INSTALL_PATH` - `VS140COMNTOOLS` - `WINDIR` - `WSLENV` - `WT_PROFILE_ID` - `WT_SESSION` - `PIP_DISABLE_PIP_VERSION_CHECK` - `PYTHONDONTWRITEBYTECODE` - `PIP_SHIMS_BASE_MODULE` - `PIP_PYTHON_PATH` - `PYTHONFINDER_IGNORE_UNSUPPORTED` Pipenv???specific environment variables: Debug???specific environment variables: - `PATH`: `C:\Program Files\Microsoft MPI\Bin\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files (x86)\IBM\Notes\;C:\Apps\tcvis\11_DAF\Products\Mockup\ClearanceDB;C:\Program Files\Citrix\System32\;C:\Program Files\Citrix\ICAService\;C:\Program Files\Intel\WiFi\bin\;C:\Program Files\Common Files\Intel\WirelessCommon\;C:\Program Files\dotnet\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Go\bin;C:\Program Files\nodejs\;C:\Program Files (x86)\Windows Kits\8.1\Windows Performance Toolkit\;C:\HashiCorp\Vagrant\bin;C:\Program Files\Docker\Docker\resources\bin;C:\ProgramData\DockerDesktop\version-bin;C:\Program Files\Git\cmd;C:\Users\joshi\AppData\Local\Programs\Python\Python39\Scripts\;C:\Users\joshi\AppData\Local\Programs\Python\Python39\;C:\Users\joshi\.poetry\bin;C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\Scripts\;C:\Users\joshi\AppData\Local\Programs\Python\Python38-32\;C:\Users\joshi\AppData\Local\Microsoft\WindowsApps;C:\Users\joshi\AppData\Local\Programs\Microsoft VS Code\bin;C:\Users\joshi\AppData\Local\Pandoc\;C:\Users\joshi\go\bin;C:\Users\joshi\AppData\Roaming\npm;C:\src\flutter\bin;;C:\Users\joshi\AppData\Roaming\vba-blocks\bin;C:\Users\joshi\.dotnet\tools` --------------------------- Contents of `Pipfile` ('C:\\Users\\joshi\\Documents\\JK_Work\\Git_Repos\\Pathsync\\Pipfile'): ```toml [[source]] url = "https://pypi.org/simple" verify_ssl = true name = "pypi" [packages] setuptools = {git = "https://github.com/pypa/setuptools.git"} [dev-packages] [requires] python_version = "3.8" ``` Contents of `Pipfile.lock` ('C:\\Users\\joshi\\Documents\\JK_Work\\Git_Repos\\Pathsync\\Pipfile.lock'): ```json { "_meta": { "hash": { "sha256": "20346182f999cf9dd37c67fbfb91c3dc938702459eeddf627dc1ed962e8fe5be" }, "pipfile-spec": 6, "requires": { "python_version": "3.8" }, "sources": [ { "name": "pypi", "url": "https://pypi.org/simple", "verify_ssl": true } ] }, "default": { "setuptools": { "git": "https://github.com/pypa/setuptools.git", "ref": "c121d289da5d19cf6df2bf6b64ac28916a060161" } }, "develop": {} } ```

matteius commented 2 years ago

@joshkreud Could you recheck with the newer version of pipenv 2022.1.8?

vashek commented 1 year ago

Using pipenv-2022.12.19, I don't get the IndexError but I do get the PermissionError. The problem is that when git clones a repo, it creates the idx files as read-only. On Linux, this doesn't prevent them being deleted, but on Windows it does. So pipenv fails when trying to delete the temporarily cloned repo.

vashek commented 1 year ago

Ah, but it doesn't actually prevent the installation, I was misled by a ResolutionFailure that then also reported the PermissionError when cleaning up atexit. So yes there is an issue with cleaning up the git repo but it doesn't actually break installations.

jacobfelknor commented 1 year ago

@matteius

Using pipenv-2022.12.19, I don't get the IndexError but I do get the PermissionError. The problem is that when git clones a repo, it creates the idx files as read-only. On Linux, this doesn't prevent them being deleted, but on Windows it does. So pipenv fails when trying to delete the temporarily cloned repo.

I am seeing the issue @vashek describes on newest version of pipenv==2023.3.20 when installing from a git source. Would it be possible to change the temporary folder cleanup process so that it first changes all files to read/write before attempting to unlink?

Just hacking around in Python console,

>>> import os
>>> os.unlink(".\\pipenv-fl4rgjgg-src\\pkg-name\\.git\\objects\\pack\\pack-360ea026c4b1b3e19452d958a35ed46fdeda4431.idx")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
PermissionError: [WinError 5] Access is denied
>>> from stat import S_IWUSR, S_IREAD
>>> os.chmod(".\\pipenv-fl4rgjgg-src\\pkg-name\\.git\\objects\\pack\\pack-360ea026c4b1b3e19452d958a35ed46fdeda4431.idx", 
S_IWUSR|S_IREAD)
>>> os.unlink(".\\pipenv-fl4rgjgg-src\\pkg-name\\.git\\objects\\pack\\pack-360ea026c4b1b3e19452d958a35ed46fdeda4431.idx")
# worked!

matteius commented 1 year ago

@jacobfelknor Could you provide some reproduction steps? I also have windows and haven't run into this issue. Other things to check, what version of setuptools and wheel do you have installed?

jacobfelknor commented 1 year ago

Sure. My specific case occurs when the install from git appears in the install_requires section of setup.py for a package I depend on, who is specified in my Pipfile.

For example, package_a = "*" is in my Pipfile and package_a's source code setup.py includes a pkg@git+https: in its install_requires

Unfortunately, I have been unable to reproduce using publicly available repository (instead of my internal git)..... which is interesting and something I hadn't noticed....

However I've attached a zip file containing a very simple package showing the package structure of when I see the problem. I'll reply if I find a case that can be reproduced with a public repo example.zip

matteius commented 1 year ago

@jacobfelknor For that example it appears the problem is packages=find_packages(include=["dummy_unlink_pkg"]), in the setup.py because this instructs the resolver to look in pypi for that package. This also seems kind of circular because the package itself is saying it required the package -- I think the example needs to be reworked somehow, because this should never be the case. Is it that your example editable package is actually requiring another package from a non-pypi server?

jacobfelknor commented 1 year ago

@matteius

Removed find_packages

The circular nature of this is only to condense this example into a single "project". The same problem happens if the editable install is coming from somewhere besides itself, but was convenient for it to just be from itself. The key is getting pipenv to install a project who's setup.py had the git install

I did find a reproduceable example (tested on two different machines). The key difference was requiring Python 3.7.9, which was the case where I was seeing problem.... this could be the culprit

example 2.zip

matteius commented 1 year ago

Had no issues with Using C:/Users/matte/AppData/Local/Programs/Python/Python311/python.exe (3.11.2) to create virtualenv... but pyenv installed 3.7 and it wasn't on my PATH and I ran out of time checking that scenario for now.

matteius commented 1 year ago

We just released a new version of pipenv=2023.5.19 that cleans up these pythonfinder edge cases (new major version of pythonfinder) -- please recheck the behavior there, but I believe this issue can be closed.

pypa / pipenv