search

pypa / pipenv

Python Development Workflow for Humans.
https://pipenv.pypa.io
MIT License
24.72k stars 1.86k forks source link

pipenv install generates file paths > 260 which break on Windows #5160

Open truekonrads opened 2 years ago

truekonrads commented 2 years ago

Issue description

pipenv can generate file paths during build which are more than 260 characters which breaks on Windows as described in this issue by https://github.com/netleibi/fastchunking/issues/6#issuecomment-877677653 by @netleibi

There are no suitable OS workarounds and therefore I propose that this is a pip bug.

Expected result

1) It should at least warn that long file paths are tehre and that users should apply a workaround to trim build path prefix using TMPDIR=C:\temp or similar 2) It should generate shorter paths

Actual result

When possible, provide the verbose output (--verbose), especially for locking and dependencies resolving issues.

Steps to replicate

On Windows 10, do:

Please run $ pipenv --support, and paste the results here. Don't put backticks (`) around it! The output already contains Markdown formatting.

$ pipenv --support Pipenv version: `'2022.1.8'` Pipenv location: `'C:\\Python39\\lib\\site-packages\\pipenv'` Python location: `'C:\\Python39\\python.exe'` Python installations found: - `3.9.7`: `C:\Python39\python.exe` - `3.8.3`: `C:\Python38-32\python.exe` - `3.7.9`: `C:\Python37\python.exe` - `3.6.8`: `C:\Python36\python.exe` - `3.4.0`: `C:\Python34\python.exe` - `2.7.18`: `C:\Python27\python.exe` PEP 508 Information: ``` {'implementation_name': 'cpython', 'implementation_version': '3.9.7', 'os_name': 'nt', 'platform_machine': 'AMD64', 'platform_python_implementation': 'CPython', 'platform_release': '10', 'platform_system': 'Windows', 'platform_version': '10.0.19044', 'python_full_version': '3.9.7', 'python_version': '3.9', 'sys_platform': 'win32'} ``` System environment variables: - `ALLUSERSPROFILE` - `APPDATA` - `ASL.LOG` - `CHOCOLATEYINSTALL` - `CHOCOLATEYLASTPATHUPDATE` - `COMMONPROGRAMFILES` - `COMMONPROGRAMFILES(X86)` - `COMMONPROGRAMW6432` - `COMPUTERNAME` - `COMSPEC` - `DRIVERDATA` - `GIT_SSH` - `GOPATH` - `GRADLE_HOME` - `HOMEDRIVE` - `HOMEPATH` - `JAVA_HOME` - `LOCALAPPDATA` - `LOGONSERVER` - `NUMBER_OF_PROCESSORS` - `NVM_HOME` - `NVM_SYMLINK` - `ONEDRIVE` - `ONEDRIVECOMMERCIAL` - `OS` - `PATH` - `PATHEXT` - `PIPENV_ACTIVE` - `PIP_DISABLE_PIP_VERSION_CHECK` - `PIP_PYTHON_PATH` - `PROCESSOR_ARCHITECTURE` - `PROCESSOR_IDENTIFIER` - `PROCESSOR_LEVEL` - `PROCESSOR_REVISION` - `PROGRAMDATA` - `PROGRAMFILES` - `PROGRAMFILES(X86)` - `PROGRAMW6432` - `PROMPT` - `PSMODULEPATH` - `PUBLIC` - `PYTHONDONTWRITEBYTECODE` - `SYSTEMDRIVE` - `SYSTEMROOT` - `TEMP` - `TMP` - `TMPDIR` - `USERDOMAIN` - `USERDOMAIN_ROAMINGPROFILE` - `USERNAME` - `USERPROFILE` - `VIRTUAL_ENV` - `WINDIR` - `WSLENV` - `WT_PROFILE_ID` - `WT_SESSION` - `PIP_SHIMS_BASE_MODULE` - `PYTHONFINDER_IGNORE_UNSUPPORTED` Pipenv–specific environment variables: - `PIPENV_ACTIVE`: `1` Debug–specific environment variables: - `PATH`: `C:\Users\admin\.virtualenvs\New_parser-S96RmPhF\Scripts;C:\Program Files\Amazon Corretto\jdk11.0.8_10\bin;C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin;C:\Python39\Scripts\;C:\Python39\;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0\;C:\windows\System32\OpenSSH\;C:\Program Files\dotnet\;C:\ProgramData\chocolatey\bin;C:\Program Files\Git\cmd;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn\;C:\Program Files\Amazon\AWSCLIV2\;C:\Program Files (x86)\GNU\GnuPG\pub;C:\ProgramData\nvm;C:\Program Files\nodejs;C:\Go\bin;C:\Program Files\Microsoft Windows Performance Toolkit\;C:\Program Files (x86)\Oberthur Technologies\AWP\Dlls;C:\Program Files\Oberthur Technologies\AWP\Dlls;C:\Program Files (x86)\IDEMIA\AWP\Dlls;C:\Program Files\IDEMIA\AWP\Dlls;C:\Program Files\Gemalto\Classic Client\BIN;C:\Program Files (x86)\Gemalto\Classic Client\BIN;C:\Program Files (x86)\eParaksts Token Signing\;C:\Program Files (x86)\dotnet\;C:\Program Files\Amazon\AWSSAMCLI\bin\;C:\Program Files\Docker\Docker\resources\bin;C:\ProgramData\DockerDesktop\version-bin;C:\Users\admin\.cargo\bin;C:\Users\admin\AppData\Local\Microsoft\WindowsApps;C:\Users\admin\AppData\Local\Programs\Microsoft VS Code\bin;C:\Program Files\NipperStudio;C:\Program Files (x86)\Nmap;C:\Users\admin\go\bin;C:\Tools\apache-maven-3.8.1\bin;C:\Users\admin\AppData\Local\gitkraken\bin;C:\Users\admin\AppData\Local\Keybase\` - `VIRTUAL_ENV`: `C:\Users\admin\.virtualenvs\New_parser-S96RmPhF` ---------------------------

If you're on macOS, run the following:

$ pipenv --support | pbcopy

If you're on Windows, run the following:

> pipenv --support | clip

If you're on Linux, run the following:

$ pipenv --support | xclip
matteius commented 2 years ago

@truekonrads I believe you should be reporting this bug directly to the pip backlog as well, as pipenv just vendors in pip and we can see that trying to pip install fastchunking also fails out-right on Windows because of this issue:

$ pip install fastchunking
Collecting fastchunking
  Downloading fastchunking-0.0.3.tar.gz (24 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting pybindgen
  Using cached PyBindGen-0.22.1-py2.py3-none-any.whl (152 kB)
Using legacy 'setup.py install' for fastchunking, since package 'wheel' is not installed.
Installing collected packages: pybindgen, fastchunking
  Running setup.py install for fastchunking: started
  Running setup.py install for fastchunking: finished with status 'error'
  error: subprocess-exited-with-error

  Running setup.py install for fastchunking did not run successfully.
  exit code: 1

  [50 lines of output]
  Generating file C:\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\build\rabinkarprh.cpp
  running install
  running build
  running build_py
  creating build\lib.win-amd64-3.10
  creating build\lib.win-amd64-3.10\fastchunking
  copying fastchunking\benchmark.py -> build\lib.win-amd64-3.10\fastchunking
  copying fastchunking\test.py -> build\lib.win-amd64-3.10\fastchunking
  copying fastchunking\__init__.py -> build\lib.win-amd64-3.10\fastchunking
  creating build\lib.win-amd64-3.10\lib
  copying lib\rabinkarpgen.py -> build\lib.win-amd64-3.10\lib
  copying lib\rabinkarp_gen.py -> build\lib.win-amd64-3.10\lib
  copying lib\__init__.py -> build\lib.win-amd64-3.10\lib
  running build_ext
  building 'fastchunking._rabinkarprh' extension
  creating build\temp.win-amd64-3.10
  creating build\temp.win-amd64-3.10\Release
  creating build\temp.win-amd64-3.10\Release\Users
  creating build\temp.win-amd64-3.10\Release\Users\matte
  creating build\temp.win-amd64-3.10\Release\Users\matte\AppData
  creating build\temp.win-amd64-3.10\Release\Users\matte\AppData\Local
  creating build\temp.win-amd64-3.10\Release\Users\matte\AppData\Local\Temp
  creating build\temp.win-amd64-3.10\Release\Users\matte\AppData\Local\Temp\pip-install-ii196agq
  creating build\temp.win-amd64-3.10\Release\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7
  creating build\temp.win-amd64-3.10\Release\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\build
  creating build\temp.win-amd64-3.10\Release\lib
  C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Ilib -IC:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\include -IC:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\Include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\include -IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt /EHsc /TpC:\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\build\rabinkarprh.cpp /Fobuild\temp.win-amd64-3.10\Release\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\build\rabinkarprh.obj
  rabinkarprh.cpp
  lib\rabinkarp.h(116): warning C4267: 'initializing': conversion from 'size_t' to 'unsigned int', possible loss of data
  lib\rabinkarp.h(171): warning C4267: 'initializing': conversion from 'size_t' to 'int', possible loss of data
  lib\rabinkarp.h(214): warning C4267: 'initializing': conversion from 'size_t' to 'unsigned int', possible loss of data
  C:\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\lib\characterhash.h(62): warning C4293: '>>': shift count negative or too big, undefined behavior
  C:\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\lib\characterhash.h(55): note: while compiling class template member function 'CharacterHash<uint32,unsigned char>::CharacterHash(hashvaluetype,uint32)'
          with
          [
              hashvaluetype=uint32
          ]
  lib\rabinkarp.h(34): note: see reference to function template instantiation 'CharacterHash<uint32,unsigned char>::CharacterHash(hashvaluetype,uint32)' being compiled
          with
          [
              hashvaluetype=uint32
          ]
  lib\rabinkarp.h(81): note: see reference to class template instantiation 'CharacterHash<uint32,unsigned char>' being compiled
  C:\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\lib\characterhash.h(64): warning C4293: '>>': shift count negative or too big, undefined behavior
  C:\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\lib\characterhash.h(67): warning C4293: '<<': shift count negative or too big, undefined behavior
  C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Ilib -IC:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\include -IC:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\Include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\include -IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt /EHsc /Tplib/rabinkarp.cpp /Fobuild\temp.win-amd64-3.10\Release\lib/rabinkarp.obj
  rabinkarp.cpp
  C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\libs /LIBPATH:C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\PCbuild\amd64 /LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\lib\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\ucrt\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\um\x64 /EXPORT:PyInit__rabinkarprh build\temp.win-amd64-3.10\Release\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\build\rabinkarprh.obj build\temp.win-amd64-3.10\Release\lib/rabinkarp.obj /OUT:build\lib.win-amd64-3.10\fastchunking\_rabinkarprh.cp310-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.10\Release\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\build\_rabinkarprh.cp310-win_amd64.lib
  LINK : fatal error LNK1104: cannot open file 'build\temp.win-amd64-3.10\Release\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\build\_rabinkarprh.cp310-win_amd64.exp'
  error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX86\\x64\\link.exe' failed with exit code 1104
  [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

Encountered error while trying to install package.

fastchunking

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
matteius commented 2 years ago

Seems related to this locked issue report, but I am fairly sure the version of python from the Microsoft Store doesn't ask you about disabling the path length limit. https://github.com/pypa/pip/issues/10996

matteius commented 2 years ago

I imagine the pip developers may not be interested in improving the default behavior for Windows, probably for existing installs this is the path forward: https://docs.python.org/3/using/windows.html#removing-the-max-path-limitation

However I note also that because fastchunking invokes C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX86\\x64\\link.exe it would appear that the max-path limitation cannot be changed and the install continues to fail even with modification of this registry value.

truekonrads commented 2 years ago

Seems related to this locked issue report, but I am fairly sure the version of python from the Microsoft Store doesn't ask you about disabling the path length limit. pypa/pip#10996

From the original bug report, the hero writes:

Unfortunately, enabling "Win32 long paths" support as described under the link above does not resolve the problem, as long paths would have to be explicitly supported by the Visual C++ linker, which is apparently not the case.

goyalyashpal commented 5 months ago

long paths would have to be explicitly supported by the Visual C++ linker, which is apparently not the case.

i don't understand why setuptools (or whatever is the tool responsible for invoking these compilers) don't allow using msys2's clang or gcc compilers. soooo many people are suffering from this MSVC thing