search

pypa / pipenv

Python Development Workflow for Humans.
https://pipenv.pypa.io
MIT License
24.92k stars 1.87k forks source link

Installing git dependencies over ssh often leads to corrupted pipfile.lock or pipfile dependencies #6076

Open Kroppeb opened 10 months ago

Kroppeb commented 10 months ago

Issue description

Installing git dependencies over ssh often results in errors.

Steps to replicate

Create a new project in a new folder, and run pipenv install git+ssh://git@github.com/pypa/pipenv.git#egg=pipenv

This fails. In the created lockfile contains this:

        "pipenv": {
            "git": "git+ssh://git",
            "markers": "python_version >= '3.8'",
            "ref": "d4483dd2a3cd55c45de2e3d5ca98094c67b64af3"
        },

Which results in pip trying to run the invalid command git clone --filter=blob:none --quiet 'ssh://git@d4483dd2a3cd55c45de2e3d5ca98094c67b64af3' ...

Running pipenv install git+ssh://git@github.com/pypa/pipenv.git@main#egg=pipenv instead does seem to fix the issue

$ pipenv --support Pipenv version: `'2023.12.0'` Pipenv location: `'C:\\Users\\Robbe\\AppData\\Local\\Programs\\Python\\Python312\\Lib\\site-packages\\pipenv'` Python location: `'C:\\Users\\Robbe\\AppData\\Local\\Programs\\Python\\Python312\\python.exe'` OS Name: `'nt'` User pip version: `'23.3.2'` user Python installations found: PEP 508 Information: ``` {'implementation_name': 'cpython', 'implementation_version': '3.12.1', 'os_name': 'nt', 'platform_machine': 'AMD64', 'platform_python_implementation': 'CPython', 'platform_release': '11', 'platform_system': 'Windows', 'platform_version': '10.0.22631', 'python_full_version': '3.12.1', 'python_version': '3.12', 'sys_platform': 'win32'} ``` System environment variables: - `ALLUSERSPROFILE` - `APPDATA` - `COMMONPROGRAMFILES` - `COMMONPROGRAMFILES(X86)` - `COMMONPROGRAMW6432` - `COMPUTERNAME` - `COMSPEC` - `DRIVERDATA` - `HOMEDRIVE` - `HOMEPATH` - `LOCALAPPDATA` - `LOGONSERVER` - `NUMBER_OF_PROCESSORS` - `ONEDRIVE` - `ONEDRIVECONSUMER` - `OS` - `PATH` - `PATHEXT` - `PROCESSOR_ARCHITECTURE` - `PROCESSOR_IDENTIFIER` - `PROCESSOR_LEVEL` - `PROCESSOR_REVISION` - `PROGRAMDATA` - `PROGRAMFILES` - `PROGRAMFILES(X86)` - `PROGRAMW6432` - `PSMODULEPATH` - `PUBLIC` - `SESSIONNAME` - `SYSTEMDRIVE` - `SYSTEMROOT` - `TEMP` - `TMP` - `USERDOMAIN` - `USERDOMAIN_ROAMINGPROFILE` - `USERNAME` - `USERPROFILE` - `WINDIR` - `WSLENV` - `WT_PROFILE_ID` - `WT_SESSION` - `ZES_ENABLE_SYSMAN` - `PIP_DISABLE_PIP_VERSION_CHECK` - `PYTHONDONTWRITEBYTECODE` - `PYTHONFINDER_IGNORE_UNSUPPORTED` Pipenv?specific environment variables: Debug?specific environment variables: - `PATH`: `C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files\Git\cmd;C:\Program Files (x86)\Gpg4win\..\GnuPG\bin;C:\Program Files\nodejs\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Users\Robbe\AppData\Local\Programs\Python\Python312\Scripts\;C:\Users\Robbe\AppData\Local\Programs\Python\Python312\;C:\Users\Robbe\AppData\Local\Programs\Python\Python311\Scripts\;C:\Users\Robbe\AppData\Local\Programs\Python\Python311\;C:\Users\Robbe\AppData\Local\Programs\Python\Python310\Scripts\;C:\Users\Robbe\AppData\Local\Programs\Python\Python310\;C:\Users\Robbe\AppData\Local\Microsoft\WindowsApps;C:\Users\Robbe\AppData\Local\JetBrains\Toolbox\scripts;C:\Users\Robbe\AppData\Local\Programs\Microsoft VS Code\bin;C:\Users\Robbe\AppData\Local\Google\Cloud SDK\google-cloud-sdk\bin;C:\Users\Robbe\AppData\Roaming\npm;C:\Users\Robbe\AppData\Local\Microsoft\WindowsApps;` --------------------------- Contents of `Pipfile` ('C:\\Users\\Robbe\\Personal\\pipenv-bug-test1\\Pipfile'): ```toml [[source]] url = "https://pypi.org/simple" verify_ssl = true name = "pypi" [packages] [dev-packages] [requires] python_version = "3.12" ``` Contents of `Pipfile.lock` ('C:\\Users\\Robbe\\Personal\\pipenv-bug-test1\\Pipfile.lock'): ```json { "_meta": { "hash": { "sha256": "86fe84858bd554474429681a04a3dca37e6755bcbfdc43990e5217ea4cfa89b3" }, "pipfile-spec": 6, "requires": { "python_version": "3.12" }, "sources": [ { "name": "pypi", "url": "https://pypi.org/simple", "verify_ssl": true } ] }, "default": { "certifi": { "hashes": [ "sha256:0569859f95fc761b18b45ef421b1290a0f65f147e92a1e5eb3e635f9a5e4e66f", "sha256:dc383c07b76109f368f6106eee2b593b04a011ea4d55f652c6ca24a754d1cdd1" ], "markers": "python_version >= '3.6'", "version": "==2024.2.2" }, "distlib": { "hashes": [ "sha256:034db59a0b96f8ca18035f36290806a9a6e6bd9d1ff91e45a7f172eb17e51784", "sha256:1530ea13e350031b6312d8580ddb6b27a104275a31106523b8f123787f494f64" ], "version": "==0.3.8" }, "filelock": { "hashes": [ "sha256:521f5f56c50f8426f5e03ad3b281b490a87ef15bc6c526f168290f0c7148d44e", "sha256:57dbda9b35157b05fb3e58ee91448612eb674172fab98ee235ccb0b5bee19a1c" ], "markers": "python_version >= '3.8'", "version": "==3.13.1" }, "pipenv": { "git": "git+ssh://git", "markers": "python_version >= '3.8'", "ref": "d4483dd2a3cd55c45de2e3d5ca98094c67b64af3" }, "platformdirs": { "hashes": [ "sha256:0614df2a2f37e1a662acbd8e2b25b92ccf8632929bc6d43467e17fe89c75e068", "sha256:ef0cc731df711022c174543cb70a9b5bd22e5a9337c8624ef2c2ceb8ddad8768" ], "markers": "python_version >= '3.8'", "version": "==4.2.0" }, "setuptools": { "hashes": [ "sha256:385eb4edd9c9d5c17540511303e39a147ce2fc04bc55289c322b9e5904fe2c05", "sha256:be1af57fc409f93647f2e8e4573a142ed38724b8cdd389706a867bb4efcf1e78" ], "markers": "python_version >= '3.8'", "version": "==69.0.3" }, "virtualenv": { "hashes": [ "sha256:4238949c5ffe6876362d9c0180fc6c3a824a7b12b80604eeb8085f2ed7460de3", "sha256:bf51c0d9c7dd63ea8e44086fa1e4fb1093a31e963b86959257378aef020e1f1b" ], "markers": "python_version >= '3.7'", "version": "==20.25.0" } }, "develop": {} } ```
Kroppeb commented 10 months ago

So the weird thing is that when we install our own private packages like this, this usually doesn't fail. It's often later when installing the project on a new system that issues arise. Not sure why installing pipenv seems to fail more reliably.

matteius commented 10 months ago

The problem is installing pipenv with pipenv -- it is not supported and it won't work right.

Kroppeb commented 10 months ago

I don't see why installing pipenv with pipenv is an issue? It's not that I'm using the virtualenv created by pipenv.

I have also been able to reproduce it with pipenv install git+ssh://git@github.com/cglwn/leftpad-pypi.git#egg=leftpad (and again, I can't using pipenv install git+ssh://git@github.com/cglwn/leftpad-pypi.git@master#egg=leftpad

matteius commented 10 months ago

Sounds like your issue relates more to ssh cloning, because you need an ssh key that gives you access to that project. Typically you want to use https urls for cloning since they are publicly available; for example pipenv hasn't granted you explicit access, to the repository so ssh is going to fail.

Kroppeb commented 10 months ago

No, not the issue. As mentioned in my report in the pipenv.lock file the following appears: "git": "git+ssh://git" instead of the expected "git" = "git+ssh://git@github.com/cglwn/leftpad-pypi.git" in case of leftpad.

As a result when trying to make pip install it, the following requirement is created: leftpad@ git+ssh://git@7184e2cd37256058cbece2aeebf4bff2533bd699 instead of leftpad@ git+ssh://git@github.com/cglwn/leftpad-pypi.git@7184e2cd37256058cbece2aeebf4bff2533bd699

Additionally, if you fix the pipenv.lock file manually, everything seems to work. But the requirement isn't mentioned in the pipenv file. Running the install command again, will again fail, but this time will add the following requirement

leftpad = {ref = "github.com/cglwn/leftpad-pypi.git", git = "git+ssh://git"}

It seems that somewhere along the way, pipenv splits the requirement on @ symbols and misinterprets the case where there is only one @ symbol.

What really bugs me is that in my experience, installing git dependencies over ssh usually works, and most issue we've encountered at my job is when cloning a project and trying to create the environment. I am really surprised that now I'm trying to trigger an issue, that it is consistently failing.

matteius commented 10 months ago

Even on pip, when I try to clone a repo with ssh that my key doesn't have access to, I get rejected. image

Kroppeb commented 10 months ago

I have access to the repo

matteius commented 10 months ago

This worked for me, I understand its https, but I don't have access to that repo to test ssh with: $ pipenv install git+https://github.com/cglwn/leftpad-pypi.git@master

I also tested it with pydantic:

$ cat Pipfile
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
pydantic = {ref = "main", git = "git+https://github.com/pydantic/pydantic.git"}
leftpad = {ref = "master", git = "git+https://github.com/cglwn/leftpad-pypi.git"}

[dev-packages]

[requires]
python_version = "3.11"

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/pipenv-6076
$ cat Pipfile.lock
{
    "_meta": {
        "hash": {
            "sha256": "1457336e0469ee62ed6e522c6c5018594aedc96868d5248a128929262bdc3b4e"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.11"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "annotated-types": {
            "hashes": [
                "sha256:0641064de18ba7a25dee8f96403ebc39113d0cb953a01429249d5c7564666a43",
                "sha256:563339e807e53ffd9c267e99fc6d9ea23eb8443c08f112651963e24e22f84a5d"
            ],
            "markers": "python_version >= '3.8'",
            "version": "==0.6.0"
        },
        "leftpad": {
            "git": "git+https://github.com/cglwn/leftpad-pypi.git",
            "ref": "7184e2cd37256058cbece2aeebf4bff2533bd699"
        },
        "pydantic": {
            "git": "git+https://github.com/pydantic/pydantic.git",
            "markers": "python_version >= '3.8'",
            "ref": "8364920acb272e2df5c6e261ca67622f3d6d1874"
        },
        "pydantic-core": {
            "hashes": [
                "sha256:06f0d5a1d9e1b7932477c172cc720b3b23c18762ed7a8efa8398298a59d177c7",
                "sha256:07982b82d121ed3fc1c51faf6e8f57ff09b1325d2efccaa257dd8c0dd937acca",
                "sha256:0f478ec204772a5c8218e30eb813ca43e34005dff2eafa03931b3d8caef87d51",
                "sha256:102569d371fadc40d8f8598a59379c37ec60164315884467052830b28cc4e9da",
                "sha256:10dca874e35bb60ce4f9f6665bfbfad050dd7573596608aeb9e098621ac331dc",
                "sha256:150ba5c86f502c040b822777e2e519b5625b47813bd05f9273a8ed169c97d9ae",
                "sha256:1661c668c1bb67b7cec96914329d9ab66755911d093bb9063c4c8914188af6d4",
                "sha256:1a2fe7b00a49b51047334d84aafd7e39f80b7675cad0083678c58983662da89b",
                "sha256:1ae8048cba95f382dba56766525abca438328455e35c283bb202964f41a780b0",
                "sha256:20f724a023042588d0f4396bbbcf4cffd0ddd0ad3ed4f0d8e6d4ac4264bae81e",
                "sha256:2133b0e412a47868a358713287ff9f9a328879da547dc88be67481cdac529118",
                "sha256:21e3298486c4ea4e4d5cc6fb69e06fb02a4e22089304308817035ac006a7f506",
                "sha256:21ebaa4bf6386a3b22eec518da7d679c8363fb7fb70cf6972161e5542f470798",
                "sha256:23632132f1fd608034f1a56cc3e484be00854db845b3a4a508834be5a6435a6f",
                "sha256:2d5bea8012df5bb6dda1e67d0563ac50b7f64a5d5858348b5c8cb5043811c19d",
                "sha256:300616102fb71241ff477a2cbbc847321dbec49428434a2f17f37528721c4948",
                "sha256:30a8259569fbeec49cfac7fda3ec8123486ef1b729225222f0d41d5f840b476f",
                "sha256:399166f24c33a0c5759ecc4801f040dbc87d412c1a6d6292b2349b4c505effc9",
                "sha256:3fac641bbfa43d5a1bed99d28aa1fded1984d31c670a95aac1bf1d36ac6ce137",
                "sha256:42c29d54ed4501a30cd71015bf982fa95e4a60117b44e1a200290ce687d3e640",
                "sha256:462d599299c5971f03c676e2b63aa80fec5ebc572d89ce766cd11ca8bcb56f3f",
                "sha256:4eebbd049008eb800f519578e944b8dc8e0f7d59a5abb5924cc2d4ed3a1834ff",
                "sha256:502c062a18d84452858f8aea1e520e12a4d5228fc3621ea5061409d666ea1706",
                "sha256:5317c04349472e683803da262c781c42c5628a9be73f4750ac7d13040efb5d2d",
                "sha256:5511f962dd1b9b553e9534c3b9c6a4b0c9ded3d8c2be96e61d56f933feef9e1f",
                "sha256:561be4e3e952c2f9056fba5267b99be4ec2afadc27261505d4992c50b33c513c",
                "sha256:601d3e42452cd4f2891c13fa8c70366d71851c1593ed42f57bf37f40f7dca3c8",
                "sha256:644904600c15816a1f9a1bafa6aab0d21db2788abcdf4e2a77951280473f33e1",
                "sha256:653a5dfd00f601a0ed6654a8b877b18d65ac32c9d9997456e0ab240807be6cf7",
                "sha256:694a5e9f1f2c124a17ff2d0be613fd53ba0c26de588eb4bdab8bca855e550d95",
                "sha256:71b4a48a7427f14679f0015b13c712863d28bb1ab700bd11776a5368135c7d60",
                "sha256:72bf9308a82b75039b8c8edd2be2924c352eda5da14a920551a8b65d5ee89253",
                "sha256:735dceec50fa907a3c314b84ed609dec54b76a814aa14eb90da31d1d36873a5e",
                "sha256:73802194f10c394c2bedce7a135ba1d8ba6cff23adf4217612bfc5cf060de34c",
                "sha256:780daad9e35b18d10d7219d24bfb30148ca2afc309928e1d4d53de86822593dc",
                "sha256:8655f55fe68c4685673265a650ef71beb2d31871c049c8b80262026f23605ee3",
                "sha256:877045a7969ace04d59516d5d6a7dee13106822f99a5d8df5e6822941f7bedc8",
                "sha256:87bce04f09f0552b66fca0c4e10da78d17cb0e71c205864bab4e9595122cb9d9",
                "sha256:8d4dfc66abea3ec6d9f83e837a8f8a7d9d3a76d25c9911735c76d6745950e62c",
                "sha256:8ec364e280db4235389b5e1e6ee924723c693cbc98e9d28dc1767041ff9bc388",
                "sha256:8fa00fa24ffd8c31fac081bf7be7eb495be6d248db127f8776575a746fa55c95",
                "sha256:920c4897e55e2881db6a6da151198e5001552c3777cd42b8a4c2f72eedc2ee91",
                "sha256:920f4633bee43d7a2818e1a1a788906df5a17b7ab6fe411220ed92b42940f818",
                "sha256:9795f56aa6b2296f05ac79d8a424e94056730c0b860a62b0fdcfe6340b658cc8",
                "sha256:98f0edee7ee9cc7f9221af2e1b95bd02810e1c7a6d115cfd82698803d385b28f",
                "sha256:99c095457eea8550c9fa9a7a992e842aeae1429dab6b6b378710f62bfb70b394",
                "sha256:99d3a433ef5dc3021c9534a58a3686c88363c591974c16c54a01af7efd741f13",
                "sha256:99f9a50b56713a598d33bc23a9912224fc5d7f9f292444e6664236ae471ddf17",
                "sha256:9c46e556ee266ed3fb7b7a882b53df3c76b45e872fdab8d9cf49ae5e91147fd7",
                "sha256:9f5d37ff01edcbace53a402e80793640c25798fb7208f105d87a25e6fcc9ea06",
                "sha256:a0b4cfe408cd84c53bab7d83e4209458de676a6ec5e9c623ae914ce1cb79b96f",
                "sha256:a497be217818c318d93f07e14502ef93d44e6a20c72b04c530611e45e54c2196",
                "sha256:ac89ccc39cd1d556cc72d6752f252dc869dde41c7c936e86beac5eb555041b66",
                "sha256:adf28099d061a25fbcc6531febb7a091e027605385de9fe14dd6a97319d614cf",
                "sha256:afa01d25769af33a8dac0d905d5c7bb2d73c7c3d5161b2dd6f8b5b5eea6a3c4c",
                "sha256:b1fc07896fc1851558f532dffc8987e526b682ec73140886c831d773cef44b76",
                "sha256:b49c604ace7a7aa8af31196abbf8f2193be605db6739ed905ecaf62af31ccae0",
                "sha256:b9f3e0bffad6e238f7acc20c393c1ed8fab4371e3b3bc311020dfa6020d99212",
                "sha256:ba07646f35e4e49376c9831130039d1b478fbfa1215ae62ad62d2ee63cf9c18f",
                "sha256:bd88f40f2294440d3f3c6308e50d96a0d3d0973d6f1a5732875d10f569acef49",
                "sha256:c0be58529d43d38ae849a91932391eb93275a06b93b79a8ab828b012e916a206",
                "sha256:c45f62e4107ebd05166717ac58f6feb44471ed450d07fecd90e5f69d9bf03c48",
                "sha256:c56da23034fe66221f2208c813d8aa509eea34d97328ce2add56e219c3a9f41c",
                "sha256:c94b5537bf6ce66e4d7830c6993152940a188600f6ae044435287753044a8fe2",
                "sha256:cebf8d56fee3b08ad40d332a807ecccd4153d3f1ba8231e111d9759f02edfd05",
                "sha256:d0bf6f93a55d3fa7a079d811b29100b019784e2ee6bc06b0bb839538272a5610",
                "sha256:d195add190abccefc70ad0f9a0141ad7da53e16183048380e688b466702195dd",
                "sha256:d25ef0c33f22649b7a088035fd65ac1ce6464fa2876578df1adad9472f918a76",
                "sha256:d6cbdf12ef967a6aa401cf5cdf47850559e59eedad10e781471c960583f25aa1",
                "sha256:d8c032ccee90b37b44e05948b449a2d6baed7e614df3d3f47fe432c952c21b60",
                "sha256:daff04257b49ab7f4b3f73f98283d3dbb1a65bf3500d55c7beac3c66c310fe34",
                "sha256:e83ebbf020be727d6e0991c1b192a5c2e7113eb66e3def0cd0c62f9f266247e4",
                "sha256:ed3025a8a7e5a59817b7494686d449ebfbe301f3e757b852c8d0d1961d6be864",
                "sha256:f1936ef138bed2165dd8573aa65e3095ef7c2b6247faccd0e15186aabdda7f66",
                "sha256:f5247a3d74355f8b1d780d0f3b32a23dd9f6d3ff43ef2037c6dcd249f35ecf4c",
                "sha256:fa496cd45cda0165d597e9d6f01e36c33c9508f75cf03c0a650018c5048f578e",
                "sha256:fb4363e6c9fc87365c2bc777a1f585a22f2f56642501885ffc7942138499bf54",
                "sha256:fb4370b15111905bf8b5ba2129b926af9470f014cb0493a67d23e9d7a48348e8",
                "sha256:fbec2af0ebafa57eb82c18c304b37c86a8abddf7022955d1742b3d5471a6339e"
            ],
            "markers": "python_version >= '3.8'",
            "version": "==2.16.1"
        },
        "typing-extensions": {
            "hashes": [
                "sha256:23478f88c37f27d76ac8aee6c905017a143b0b1b886c3c9f66bc2fd94f9f5783",
                "sha256:af72aea155e91adfc61c3ae9e0e342dbc0cba726d6cba4b6c72c1f34e47291cd"
            ],
            "markers": "python_version >= '3.8'",
            "version": "==4.9.0"
        }
    },
    "develop": {}
}
Kroppeb commented 10 months ago

Yeah, I wouldn't be surprised if this is a ssh only issue. I'll update the issue

matteius commented 10 months ago

I think I found the issue is colon vs slash -- not sure if github changed how they are exporting the ssh URL in the UI or not, but this has parity with how pip requires the ssh line be supplied: https://github.com/pypa/pipenv/issues/6085#issuecomment-1926901571

tehstu commented 4 months ago

This issue also affects our corporate Bitbucket (self-hosted), so may not be something specific to Github. I found the same problem occurring, where the lock file contains a ref value of the commit id, rather than the URL of the repo on Bitbucket.

However, if I include the branch name before the #egg parameter, as suggested by the OP, then I have no issues installing.

If it helps, Pipenv is 2023.12.1 and Python 3.9.2. Happy to provide any other info.

matteius commented 1 month ago

Could this be rechecked against pipenv==2024.3.0?

tehstu commented 1 month ago

I updated pipenv to 2024.3.0, still using Python 3.9.2. Unfortunately, I see the same behavior as documented by Kroppeb here and by myself above. Unless I include @branch_name before the #egg=, it fails in the same way.