pypa / pipenv

Python Development Workflow for Humans.
https://pipenv.pypa.io
MIT License
24.88k stars 1.87k forks source link

Install package with requirement specifiers doesn't write dependencies #4451

Closed jules-ch closed 2 years ago

jules-ch commented 4 years ago

Issue description

When installing a package with a simple requirement specifier, pipenv.lock doesn't show any dependencies if locking was made with a platform which doesn't match the constraint.

It is working with pipenv 2018.11.26 on windows.

Pipfile.lock with pipenv 2018.11.26 on both Windows 10 & Ubuntu 20.04 ```json { "_meta": { "hash": { "sha256": "7624ac65f711c91e061df6c8be22fb968565da45132a67f6dd553f97c531ad95" }, "pipfile-spec": 6, "requires": { "python_version": "3.8" }, "sources": [ { "name": "pypi", "url": "https://pypi.org/simple", "verify_ssl": true } ] }, "default": { "pyasn1": { "hashes": [ "sha256:39c7e2ec30515947ff4e87fb6f456dfc6e84857d34be479c9d4a4ba4bf46aa5d", "sha256:aef77c9fb94a3ac588e87841208bdec464471d9871bd5050a287cc9a475cd0ba" ], "version": "==0.4.8" }, "pyasn1-modules": { "hashes": [ "sha256:905f84c712230b2c592c19470d3ca8d552de726050d1d1716282a1f6146be65e", "sha256:a50b808ffeb97cb3601dd25981f6b016cbb3d31fbf57a8b8a87428e6158d0c74" ], "version": "==0.2.8" }, "python-ldap": { "hashes": [ "sha256:4711cacf013e298754abd70058ccc995758177fb425f1c2d30e71adfc1d00aa5" ], "index": "pypi", "markers": "platform_system == 'Linux'", "version": "==3.3.1" } }, "develop": {} } ```

Expected result

Lock file with specified package & dependencies like doing it on the specified platform. At least like pipenv 2018.11.26

This is the generated lock file on WSL.

{
    "_meta": {
        "hash": {
            "sha256": "7624ac65f711c91e061df6c8be22fb968565da45132a67f6dd553f97c531ad95"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.8"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "pyasn1": {
            "hashes": [
                "sha256:014c0e9976956a08139dc0712ae195324a75e142284d5f87f1a87ee1b068a359",
                "sha256:03840c999ba71680a131cfaee6fab142e1ed9bbd9c693e285cc6aca0d555e576",
                "sha256:0458773cfe65b153891ac249bcf1b5f8f320b7c2ce462151f8fa74de8934becf",
                "sha256:08c3c53b75eaa48d71cf8c710312316392ed40899cb34710d092e96745a358b7",
                "sha256:39c7e2ec30515947ff4e87fb6f456dfc6e84857d34be479c9d4a4ba4bf46aa5d",
                "sha256:5c9414dcfede6e441f7e8f81b43b34e834731003427e5b09e4e00e3172a10f00",
                "sha256:6e7545f1a61025a4e58bb336952c5061697da694db1cae97b116e9c46abcf7c8",
                "sha256:78fa6da68ed2727915c4767bb386ab32cdba863caa7dbe473eaae45f9959da86",
                "sha256:7ab8a544af125fb704feadb008c99a88805126fb525280b2270bb25cc1d78a12",
                "sha256:99fcc3c8d804d1bc6d9a099921e39d827026409a58f2a720dcdb89374ea0c776",
                "sha256:aef77c9fb94a3ac588e87841208bdec464471d9871bd5050a287cc9a475cd0ba",
                "sha256:e89bf84b5437b532b0803ba5c9a5e054d21fec423a89952a74f87fa2c9b7bce2",
                "sha256:fec3e9d8e36808a28efb59b489e4528c10ad0f480e57dcc32b4de5c9d8c9fdf3"
            ],
            "version": "==0.4.8"
        },
        "pyasn1-modules": {
            "hashes": [
                "sha256:0845a5582f6a02bb3e1bde9ecfc4bfcae6ec3210dd270522fee602365430c3f8",
                "sha256:0fe1b68d1e486a1ed5473f1302bd991c1611d319bba158e98b106ff86e1d7199",
                "sha256:15b7c67fabc7fc240d87fb9aabf999cf82311a6d6fb2c70d00d3d0604878c811",
                "sha256:426edb7a5e8879f1ec54a1864f16b882c2837bfd06eee62f2c982315ee2473ed",
                "sha256:65cebbaffc913f4fe9e4808735c95ea22d7a7775646ab690518c056784bc21b4",
                "sha256:905f84c712230b2c592c19470d3ca8d552de726050d1d1716282a1f6146be65e",
                "sha256:a50b808ffeb97cb3601dd25981f6b016cbb3d31fbf57a8b8a87428e6158d0c74",
                "sha256:a99324196732f53093a84c4369c996713eb8c89d360a496b599fb1a9c47fc3eb",
                "sha256:b80486a6c77252ea3a3e9b1e360bc9cf28eaac41263d173c032581ad2f20fe45",
                "sha256:c29a5e5cc7a3f05926aff34e097e84f8589cd790ce0ed41b67aed6857b26aafd",
                "sha256:cbac4bc38d117f2a49aeedec4407d23e8866ea4ac27ff2cf7fb3e5b570df19e0",
                "sha256:f39edd8c4ecaa4556e989147ebf219227e2cd2e8a43c7e7fcb1f1c18c5fd6a3d",
                "sha256:fe0644d9ab041506b62782e92b06b8c68cca799e1a9636ec398675459e031405"
            ],
            "version": "==0.2.8"
        },
        "python-ldap": {
            "hashes": [
                "sha256:4711cacf013e298754abd70058ccc995758177fb425f1c2d30e71adfc1d00aa5"
            ],
            "index": "pypi",
            "markers": "platform_system == 'Linux'",
            "version": "==3.3.1"
        }
    },
    "develop": {}
}

Actual result

Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning.
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
           Building requirements...
Resolving dependencies...
[====] Locking...
                          ROUND 1
Current constraints:

Finding the best candidates:

Finding secondary dependencies:
------------------------------------------------------------
Result of round 1: stable, done

Generating hashes:
[=== ] Locking...ROUND 1
Current constraints:

Finding the best candidates:

Finding secondary dependencies:
------------------------------------------------------------
Result of round 1: stable, done

Generating hashes:
Success!
Updated Pipfile.lock (31ad95)!
{
    "_meta": {
        "hash": {
            "sha256": "7624ac65f711c91e061df6c8be22fb968565da45132a67f6dd553f97c531ad95"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.8"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "python-ldap": {
            "hashes": [
                "sha256:4711cacf013e298754abd70058ccc995758177fb425f1c2d30e71adfc1d00aa5"
            ],
            "markers": "platform_system == 'Linux'",
            "version": "==3.3.1"
        }
    },
    "develop": {}
}

Steps to replicate

Pipfile :

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
python-ldap = {markers = "platform_system=='Linux'", version = "*"}

[requires]
python_version = "3.8"

pipenv lock


$ pipenv --support Pipenv version: `'2020.6.2'` Pipenv location: `'C:\\Users\\Jules\\AppData\\Roaming\\Python\\Python38\\site-packages\\pipenv'` Python location: `'c:\\python38\\python.exe'` Python installations found: - `3.8.2`: `C:\Python38\python.exe` - `3.7.3`: `C:\Python37\python.exe` - `3.6.8`: `C:\Python36\python.exe` - `2.7.17`: `C:\Python27\python.exe` PEP 508 Information: ``` {'implementation_name': 'cpython', 'implementation_version': '3.8.2', 'os_name': 'nt', 'platform_machine': 'AMD64', 'platform_python_implementation': 'CPython', 'platform_release': '10', 'platform_system': 'Windows', 'platform_version': '10.0.19041', 'python_full_version': '3.8.2', 'python_version': '3.8', 'sys_platform': 'win32'} ``` System environment variables: - `ALLUSERSPROFILE` - `APPDATA` - `APPLICATION_INSIGHTS_NO_DIAGNOSTIC_CHANNEL` - `CHOCOLATEYINSTALL` - `CHOCOLATEYLASTPATHUPDATE` - `COMMONPROGRAMFILES` - `COMMONPROGRAMFILES(X86)` - `COMMONPROGRAMW6432` - `COMPOSE_CONVERT_WINDOWS_PATHS` - `COMPUTERNAME` - `COMSPEC` - `CUDA_PATH` - `CUDA_PATH_V10_0` - `CUDA_PATH_V10_1` - `CUDA_PATH_V9_0` - `CUDA_PATH_V9_1` - `DIGITALOCEAN_ACCESS_TOKEN` - `DIGITALOCEAN_IMAGE` - `DIGITALOCEAN_REGION` - `DIGITALOCEAN_SSH_KEY_FINGERPRINT` - `DRIVERDATA` - `FPS_BROWSER_APP_PROFILE_STRING` - `FPS_BROWSER_USER_PROFILE_STRING` - `HOMEDRIVE` - `HOMEPATH` - `INCLUDE` - `JAVA_HOME` - `LD_LIBRARY_PATH` - `LOCALAPPDATA` - `LOGONSERVER` - `NUMBER_OF_PROCESSORS` - `NVCUDASAMPLES10_0_ROOT` - `NVCUDASAMPLES10_1_ROOT` - `NVCUDASAMPLES9_0_ROOT` - `NVCUDASAMPLES9_1_ROOT` - `NVCUDASAMPLES_ROOT` - `NVTOOLSEXT_PATH` - `OCULUSBASE` - `ONEDRIVE` - `ONEDRIVECONSUMER` - `ORIGINAL_XDG_CURRENT_DESKTOP` - `OS` - `PATH` - `PATHEXT` - `PROCESSOR_ARCHITECTURE` - `PROCESSOR_IDENTIFIER` - `PROCESSOR_LEVEL` - `PROCESSOR_REVISION` - `PROGRAMDATA` - `PROGRAMFILES` - `PROGRAMFILES(X86)` - `PROGRAMW6432` - `PROMPT` - `PSMODULEPATH` - `PUBLIC` - `PYCHARM` - `SESSIONNAME` - `SYSTEMDRIVE` - `SYSTEMROOT` - `TEMP` - `TMP` - `USERDOMAIN` - `USERDOMAIN_ROAMINGPROFILE` - `USERNAME` - `USERPROFILE` - `VS140COMNTOOLS` - `WINDIR` - `TERM_PROGRAM` - `TERM_PROGRAM_VERSION` - `LANG` - `COLORTERM` - `VSCODE_GIT_IPC_HANDLE` - `GIT_ASKPASS` - `VSCODE_GIT_ASKPASS_NODE` - `VSCODE_GIT_ASKPASS_MAIN` - `PIP_DISABLE_PIP_VERSION_CHECK` - `PYTHONDONTWRITEBYTECODE` - `PIP_SHIMS_BASE_MODULE` - `PIP_PYTHON_PATH` - `PYTHONFINDER_IGNORE_UNSUPPORTED` PipenvÔÇôspecific environment variables: DebugÔÇôspecific environment variables: - `PATH`: `C:\openssl;C:\Program Files\Amazon Corretto\jdk11.0.6_10\bin;C:\Python38\Scripts\;C:\Python38\;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\libnvvp;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\libnvvp;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\libnvvp;C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Python37\Scripts;C:\Python37\;C:\Python36\Scripts\;C:\Python36\;C:\Program Files\Oculus\Support\oculus-runtime;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.1\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.1\libnvvp;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\ProgramData\chocolatey\bin;C:\WINDOWS\System32\OpenSSH\;C:\Program Files (x86)\OpenVPN\bin;C:\Strawberry\c\bin;C:\Strawberry\perl\site\bin;C:\Strawberry\perl\bin;C:\Program Files (x86)\Gpg4win\..\GnuPG\bin;C:\Program Files\Git\bin;C:\Program Files\Git\usr\bin;C:\Program Files\dotnet\;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files\nodejs\;C:\Program Files (x86)\Yarn\bin\;C:\Program Files\Amazon\AWSCLI\bin\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\Docker\Docker\resources\bin;C:\ProgramData\DockerDesktop\version-bin;C:\Users\Jules\.poetry\bin;C:\Users\Jules\.cargo\bin;C:\Users\Jules\AppData\Local\Microsoft\WindowsApps;C:\Program Files (x86)\Google\Chrome\Application;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin;C:\Users\Jules\AppData\Roaming\Mist;C:\Users\Jules\AppData\Roaming\Ethereum Wallet\binaries\Geth\unpacked;C:\Program Files\Git\bin;C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.11.25503\bin\HostX86\x86;C:\Miner\ethminer\bin;C:\Program Files\nodejs;D:\06.Dev-Tools\Qt\Qt5.11.2\5.11.2\msvc2017_64\bin;C:\Users\Jules\.dotnet\tools;C:\Program Files\JetBrains\PyCharm 2019.1.4\bin;;C:\Users\Jules\Downloads;C:\Program Files\ffmpeg-4.2.3-win64-static\bin;C:\Program Files (x86)\phantomjs-2.1.1-windows\bin;C:\Program Files (x86)\balena-cli;C:\Users\Jules\AppData\Local\Programs\Microsoft VS Code\bin;C:\Users\Jules\AppData\Local\Microsoft\WindowsApps;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\extras\CUPTI\libx64;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\include;C:\tools\cuda\bin;C:\Users\Jules\AppData\Roaming\npm;C:\Users\Jules\AppData\Local\Yarn\bin;C:\Program Files\Terraform;C:\Users\Jules\bin;C:\Gradle\gradle-6.3\bin;C:\Program Files (x86)\Graphviz2.38\bin;C:\Users\Jules\AppData\Local\Pandoc\;C:\Users\Jules\AppData\Roaming\Python\Python38\Scripts;` - `LANG`: `fr_FR.UTF-8` --------------------------- Contents of `Pipfile` ('C:\\Users\\Jules\\Code\\test-pipenv\\Pipfile'): ```toml [[source]] name = "pypi" url = "https://pypi.org/simple" verify_ssl = true [dev-packages] [packages] python-ldap = {markers = "platform_system=='Linux'", version = "*"} [requires] python_version = "3.8" ``` Contents of `Pipfile.lock` ('C:\\Users\\Jules\\Code\\test-pipenv\\Pipfile.lock'): ```json { "_meta": { "hash": { "sha256": "7624ac65f711c91e061df6c8be22fb968565da45132a67f6dd553f97c531ad95" }, "pipfile-spec": 6, "requires": { "python_version": "3.8" }, "sources": [ { "name": "pypi", "url": "https://pypi.org/simple", "verify_ssl": true } ] }, "default": { "python-ldap": { "hashes": [ "sha256:4711cacf013e298754abd70058ccc995758177fb425f1c2d30e71adfc1d00aa5" ], "markers": "platform_system == 'Linux'", "version": "==3.3.1" } }, "develop": {} } ```
frostming commented 4 years ago

Cross-platform dependency resolution is problematic and can fail in many cases. For example, the package has only sdist and it has some platform-specific requirements to build it. Locking will fail because the requirements can't be met on other platforms.

So the decision is made to not resolve the sub-dependencies at all. You can notice the lock file generated by 2018.11.26 is also incorrect: All sub-dependencies of python-ldap isn't marked as Linux only.

So under the current situation, you should lock dependencies on Linux.

Systemcluster commented 4 years ago

I would propose that there is at least some feedback shown when sub-dependencies are omitted while locking. The current behavior is unexpected from the user perspective and can only be detected by manually verifying the lockfile.

Cross-platform dependency resolution is problematic and can fail in many cases. For example, the package has only sdist and it has some platform-specific requirements to build it. Locking will fail because the requirements can't be met on other platforms.

Does this mean a dependency has to be able to be built on the current system to enumerate its dependencies? If yes, what is the motivation behind it?

tinkerware commented 3 years ago

I would propose that there is at least some feedback shown when sub-dependencies are omitted while locking. The current behavior is unexpected from the user perspective and can only be detected by manually verifying the lockfile.

Agreed with this; it would help figure out cross-platform build issues much sooner. In the meantime, I've started hunting platform-specific sub-dependencies and pinning them with a marker in my Pipenv file, as suggested here: https://github.com/pypa/pipenv/issues/4408#issuecomment-668324177

Does this mean a dependency has to be able to be built on the current system to enumerate its dependencies? If yes, what is the motivation behind it?

I have found that, for at least packages with native dependencies like spicy, this seems true; pipenv lock will attempt to compile the native bits, and on macOS Big Sur, I sometimes have to prefix it with SYSTEM_VERSION_COMPAT=1 to ensure older packages that can't deal with Big Sur's native toolchain major version bump can still resolve correctly.

matteius commented 2 years ago

Can this be rechecked with pipenv==2022.9.2?