Closed jules-ch closed 2 years ago
Cross-platform dependency resolution is problematic and can fail in many cases. For example, the package has only sdist and it has some platform-specific requirements to build it. Locking will fail because the requirements can't be met on other platforms.
So the decision is made to not resolve the sub-dependencies at all. You can notice the lock file generated by 2018.11.26 is also incorrect: All sub-dependencies of python-ldap
isn't marked as Linux only.
So under the current situation, you should lock dependencies on Linux.
I would propose that there is at least some feedback shown when sub-dependencies are omitted while locking. The current behavior is unexpected from the user perspective and can only be detected by manually verifying the lockfile.
Cross-platform dependency resolution is problematic and can fail in many cases. For example, the package has only sdist and it has some platform-specific requirements to build it. Locking will fail because the requirements can't be met on other platforms.
Does this mean a dependency has to be able to be built on the current system to enumerate its dependencies? If yes, what is the motivation behind it?
I would propose that there is at least some feedback shown when sub-dependencies are omitted while locking. The current behavior is unexpected from the user perspective and can only be detected by manually verifying the lockfile.
Agreed with this; it would help figure out cross-platform build issues much sooner. In the meantime, I've started hunting platform-specific sub-dependencies and pinning them with a marker in my Pipenv
file, as suggested here: https://github.com/pypa/pipenv/issues/4408#issuecomment-668324177
Does this mean a dependency has to be able to be built on the current system to enumerate its dependencies? If yes, what is the motivation behind it?
I have found that, for at least packages with native dependencies like spicy
, this seems true; pipenv lock
will attempt to compile the native bits, and on macOS Big Sur, I sometimes have to prefix it with SYSTEM_VERSION_COMPAT=1
to ensure older packages that can't deal with Big Sur's native toolchain major version bump can still resolve correctly.
Can this be rechecked with pipenv==2022.9.2
?
Issue description
When installing a package with a simple requirement specifier, pipenv.lock doesn't show any dependencies if locking was made with a platform which doesn't match the constraint.
It is working with pipenv 2018.11.26 on windows.
Pipfile.lock with pipenv 2018.11.26 on both Windows 10 & Ubuntu 20.04
```json { "_meta": { "hash": { "sha256": "7624ac65f711c91e061df6c8be22fb968565da45132a67f6dd553f97c531ad95" }, "pipfile-spec": 6, "requires": { "python_version": "3.8" }, "sources": [ { "name": "pypi", "url": "https://pypi.org/simple", "verify_ssl": true } ] }, "default": { "pyasn1": { "hashes": [ "sha256:39c7e2ec30515947ff4e87fb6f456dfc6e84857d34be479c9d4a4ba4bf46aa5d", "sha256:aef77c9fb94a3ac588e87841208bdec464471d9871bd5050a287cc9a475cd0ba" ], "version": "==0.4.8" }, "pyasn1-modules": { "hashes": [ "sha256:905f84c712230b2c592c19470d3ca8d552de726050d1d1716282a1f6146be65e", "sha256:a50b808ffeb97cb3601dd25981f6b016cbb3d31fbf57a8b8a87428e6158d0c74" ], "version": "==0.2.8" }, "python-ldap": { "hashes": [ "sha256:4711cacf013e298754abd70058ccc995758177fb425f1c2d30e71adfc1d00aa5" ], "index": "pypi", "markers": "platform_system == 'Linux'", "version": "==3.3.1" } }, "develop": {} } ```Expected result
Lock file with specified package & dependencies like doing it on the specified platform. At least like pipenv 2018.11.26
This is the generated lock file on WSL.
Actual result
Steps to replicate
Pipfile :
pipenv lock
$ pipenv --support
Pipenv version: `'2020.6.2'` Pipenv location: `'C:\\Users\\Jules\\AppData\\Roaming\\Python\\Python38\\site-packages\\pipenv'` Python location: `'c:\\python38\\python.exe'` Python installations found: - `3.8.2`: `C:\Python38\python.exe` - `3.7.3`: `C:\Python37\python.exe` - `3.6.8`: `C:\Python36\python.exe` - `2.7.17`: `C:\Python27\python.exe` PEP 508 Information: ``` {'implementation_name': 'cpython', 'implementation_version': '3.8.2', 'os_name': 'nt', 'platform_machine': 'AMD64', 'platform_python_implementation': 'CPython', 'platform_release': '10', 'platform_system': 'Windows', 'platform_version': '10.0.19041', 'python_full_version': '3.8.2', 'python_version': '3.8', 'sys_platform': 'win32'} ``` System environment variables: - `ALLUSERSPROFILE` - `APPDATA` - `APPLICATION_INSIGHTS_NO_DIAGNOSTIC_CHANNEL` - `CHOCOLATEYINSTALL` - `CHOCOLATEYLASTPATHUPDATE` - `COMMONPROGRAMFILES` - `COMMONPROGRAMFILES(X86)` - `COMMONPROGRAMW6432` - `COMPOSE_CONVERT_WINDOWS_PATHS` - `COMPUTERNAME` - `COMSPEC` - `CUDA_PATH` - `CUDA_PATH_V10_0` - `CUDA_PATH_V10_1` - `CUDA_PATH_V9_0` - `CUDA_PATH_V9_1` - `DIGITALOCEAN_ACCESS_TOKEN` - `DIGITALOCEAN_IMAGE` - `DIGITALOCEAN_REGION` - `DIGITALOCEAN_SSH_KEY_FINGERPRINT` - `DRIVERDATA` - `FPS_BROWSER_APP_PROFILE_STRING` - `FPS_BROWSER_USER_PROFILE_STRING` - `HOMEDRIVE` - `HOMEPATH` - `INCLUDE` - `JAVA_HOME` - `LD_LIBRARY_PATH` - `LOCALAPPDATA` - `LOGONSERVER` - `NUMBER_OF_PROCESSORS` - `NVCUDASAMPLES10_0_ROOT` - `NVCUDASAMPLES10_1_ROOT` - `NVCUDASAMPLES9_0_ROOT` - `NVCUDASAMPLES9_1_ROOT` - `NVCUDASAMPLES_ROOT` - `NVTOOLSEXT_PATH` - `OCULUSBASE` - `ONEDRIVE` - `ONEDRIVECONSUMER` - `ORIGINAL_XDG_CURRENT_DESKTOP` - `OS` - `PATH` - `PATHEXT` - `PROCESSOR_ARCHITECTURE` - `PROCESSOR_IDENTIFIER` - `PROCESSOR_LEVEL` - `PROCESSOR_REVISION` - `PROGRAMDATA` - `PROGRAMFILES` - `PROGRAMFILES(X86)` - `PROGRAMW6432` - `PROMPT` - `PSMODULEPATH` - `PUBLIC` - `PYCHARM` - `SESSIONNAME` - `SYSTEMDRIVE` - `SYSTEMROOT` - `TEMP` - `TMP` - `USERDOMAIN` - `USERDOMAIN_ROAMINGPROFILE` - `USERNAME` - `USERPROFILE` - `VS140COMNTOOLS` - `WINDIR` - `TERM_PROGRAM` - `TERM_PROGRAM_VERSION` - `LANG` - `COLORTERM` - `VSCODE_GIT_IPC_HANDLE` - `GIT_ASKPASS` - `VSCODE_GIT_ASKPASS_NODE` - `VSCODE_GIT_ASKPASS_MAIN` - `PIP_DISABLE_PIP_VERSION_CHECK` - `PYTHONDONTWRITEBYTECODE` - `PIP_SHIMS_BASE_MODULE` - `PIP_PYTHON_PATH` - `PYTHONFINDER_IGNORE_UNSUPPORTED` PipenvÔÇôspecific environment variables: DebugÔÇôspecific environment variables: - `PATH`: `C:\openssl;C:\Program Files\Amazon Corretto\jdk11.0.6_10\bin;C:\Python38\Scripts\;C:\Python38\;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\libnvvp;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\libnvvp;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\libnvvp;C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Python37\Scripts;C:\Python37\;C:\Python36\Scripts\;C:\Python36\;C:\Program Files\Oculus\Support\oculus-runtime;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.1\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.1\libnvvp;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\ProgramData\chocolatey\bin;C:\WINDOWS\System32\OpenSSH\;C:\Program Files (x86)\OpenVPN\bin;C:\Strawberry\c\bin;C:\Strawberry\perl\site\bin;C:\Strawberry\perl\bin;C:\Program Files (x86)\Gpg4win\..\GnuPG\bin;C:\Program Files\Git\bin;C:\Program Files\Git\usr\bin;C:\Program Files\dotnet\;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files\nodejs\;C:\Program Files (x86)\Yarn\bin\;C:\Program Files\Amazon\AWSCLI\bin\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\Docker\Docker\resources\bin;C:\ProgramData\DockerDesktop\version-bin;C:\Users\Jules\.poetry\bin;C:\Users\Jules\.cargo\bin;C:\Users\Jules\AppData\Local\Microsoft\WindowsApps;C:\Program Files (x86)\Google\Chrome\Application;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin;C:\Users\Jules\AppData\Roaming\Mist;C:\Users\Jules\AppData\Roaming\Ethereum Wallet\binaries\Geth\unpacked;C:\Program Files\Git\bin;C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.11.25503\bin\HostX86\x86;C:\Miner\ethminer\bin;C:\Program Files\nodejs;D:\06.Dev-Tools\Qt\Qt5.11.2\5.11.2\msvc2017_64\bin;C:\Users\Jules\.dotnet\tools;C:\Program Files\JetBrains\PyCharm 2019.1.4\bin;;C:\Users\Jules\Downloads;C:\Program Files\ffmpeg-4.2.3-win64-static\bin;C:\Program Files (x86)\phantomjs-2.1.1-windows\bin;C:\Program Files (x86)\balena-cli;C:\Users\Jules\AppData\Local\Programs\Microsoft VS Code\bin;C:\Users\Jules\AppData\Local\Microsoft\WindowsApps;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\extras\CUPTI\libx64;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\include;C:\tools\cuda\bin;C:\Users\Jules\AppData\Roaming\npm;C:\Users\Jules\AppData\Local\Yarn\bin;C:\Program Files\Terraform;C:\Users\Jules\bin;C:\Gradle\gradle-6.3\bin;C:\Program Files (x86)\Graphviz2.38\bin;C:\Users\Jules\AppData\Local\Pandoc\;C:\Users\Jules\AppData\Roaming\Python\Python38\Scripts;` - `LANG`: `fr_FR.UTF-8` --------------------------- Contents of `Pipfile` ('C:\\Users\\Jules\\Code\\test-pipenv\\Pipfile'): ```toml [[source]] name = "pypi" url = "https://pypi.org/simple" verify_ssl = true [dev-packages] [packages] python-ldap = {markers = "platform_system=='Linux'", version = "*"} [requires] python_version = "3.8" ``` Contents of `Pipfile.lock` ('C:\\Users\\Jules\\Code\\test-pipenv\\Pipfile.lock'): ```json { "_meta": { "hash": { "sha256": "7624ac65f711c91e061df6c8be22fb968565da45132a67f6dd553f97c531ad95" }, "pipfile-spec": 6, "requires": { "python_version": "3.8" }, "sources": [ { "name": "pypi", "url": "https://pypi.org/simple", "verify_ssl": true } ] }, "default": { "python-ldap": { "hashes": [ "sha256:4711cacf013e298754abd70058ccc995758177fb425f1c2d30e71adfc1d00aa5" ], "markers": "platform_system == 'Linux'", "version": "==3.3.1" } }, "develop": {} } ```