search

pypa / pipenv

Python Development Workflow for Humans.
https://pipenv.pypa.io
MIT License
24.89k stars 1.87k forks source link

Incorrect dependency version installed when specifying a git commit. #5973

Closed tobygomersall closed 4 weeks ago

tobygomersall commented 1 year ago

Issue description

When I try to install a specific git commit of a repository, pipenv ignores the commit hash and installs the head.

Expected result

I expect to see the specified commit hash replicated in Pipfile.lock and the specified commit of the package installed in the virtual environment.

Actual result

The Pipfile.lock contains "ref": null in the package entry and the head of the package repository is installed in the virtual environment.

Steps to replicate

mkdir test
cd test
pipenv install git+https://github.com/myhdl/myhdl.git@ae25af4d593d20a26c85fbb17c0cd98a026e8595

Inspect the myhdl entry in Pipfile.lock. On my system the ref is set to null.

To confirm, perform an additional check:

pipenv shell
pip freeze

Which confirms that the installed myhdl is the head.

My debugging efforts

Release 2023.7.23 works as expected.

Releases 2023.8.19 and 2023.8.20 fail with an error.

Subsequent releases look to complete successfully but install the wrong version and set null in the lock file.

The problematic code looks to be on lines 115 to 118 of pipenv/pipenv/utils/locking.py

chrisroat commented 1 year ago

See same issue (ref is null using git source w/commit hash) on MacOS w/python 3.11. Going back to 2023.7.23 resolves issue.

matteius commented 1 year ago

I tried reproducing this without luck on Windows and ubuntu without luck, here is the output from Windows:

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/issue-5973
$ pipenv install git+https://github.com/myhdl/myhdl.git@ae25af4d593d20a26c85fbb17c0cd98a026e8595
Creating a virtualenv for this project...
Pipfile: C:\Users\matte\Projects\pipenv-triage\issue-5973\Pipfile
Using default python from C:\Users\matte\AppData\Local\Programs\Python\Python311
\python.exe (3.11.2) to create virtualenv...
[ ===] Creating virtual environment...created virtual environment CPython3.11.2.
final.0-64 in 4083ms
  creator CPython3Windows(dest=C:\c\users\matte\.virtualenvs\issue-5973-2dbfxMHD
, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=C:\Users\matte\AppData\Local\pypa\virtualenv)
    added seed packages: pip==23.2.1, setuptools==68.2.2, wheel==0.41.2
  activators BashActivator,BatchActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

Successfully created virtual environment!
Virtualenv location: C:\c\Users\matte\.virtualenvs\issue-5973-2dbfxMHD
Creating a Pipfile for this project...
Installing
git+https://github.com/myhdl/myhdl.git@ae25af4d593d20a26c85fbb17c0cd98a026e8595
...
Resolving
git+https://github.com/myhdl/myhdl.git@ae25af4d593d20a26c85fbb17c0cd98a026e8595
...
[    ] Installing...INFO:pipenv.patched.pip._internal.vcs.git:Cloning https://github.com/myhdl/myhdl.git (to revision ae25af4d593d20a26c85fbb17c0cd98a026e8595) to c:\users
\matte\appdata\local\temp\tmp9l5ajje7
[=   ] Installing None...INFO:pip.subprocessor:Running command git clone --filter=blob:none https://github.com/myhdl/myhdl.git 'C:\Users\matte\AppData\Local\Temp\tmp9l5ajj
e7'
INFO:pip.subprocessor:Cloning into 'C:\Users\matte\AppData\Local\Temp\tmp9l5ajje7'...
[=== ] Installing None...INFO:pip.subprocessor:Updating files:  54% (235/430)
INFO:pip.subprocessor:Updating files:  55% (237/430)
INFO:pip.subprocessor:Updating files:  56% (241/430)
INFO:pip.subprocessor:Updating files:  57% (246/430)
INFO:pip.subprocessor:Updating files:  58% (250/430)
INFO:pip.subprocessor:Updating files:  59% (254/430)
[==  ] Installing None...INFO:pip.subprocessor:Updating files:  60% (258/430)
INFO:pip.subprocessor:Updating files:  61% (263/430)
INFO:pip.subprocessor:Updating files:  62% (267/430)
INFO:pip.subprocessor:Updating files:  63% (271/430)
INFO:pip.subprocessor:Updating files:  64% (276/430)
INFO:pip.subprocessor:Updating files:  65% (280/430)
INFO:pip.subprocessor:Updating files:  66% (284/430)
INFO:pip.subprocessor:Updating files:  67% (289/430)
INFO:pip.subprocessor:Updating files:  68% (293/430)
INFO:pip.subprocessor:Updating files:  69% (297/430)
INFO:pip.subprocessor:Updating files:  70% (301/430)
INFO:pip.subprocessor:Updating files:  71% (306/430)
INFO:pip.subprocessor:Updating files:  72% (310/430)
INFO:pip.subprocessor:Updating files:  73% (314/430)
INFO:pip.subprocessor:Updating files:  74% (319/430)
INFO:pip.subprocessor:Updating files:  75% (323/430)
INFO:pip.subprocessor:Updating files:  76% (327/430)
INFO:pip.subprocessor:Updating files:  77% (332/430)
INFO:pip.subprocessor:Updating files:  78% (336/430)
INFO:pip.subprocessor:Updating files:  79% (340/430)
INFO:pip.subprocessor:Updating files:  80% (344/430)
INFO:pip.subprocessor:Updating files:  81% (349/430)
INFO:pip.subprocessor:Updating files:  82% (353/430)
INFO:pip.subprocessor:Updating files:  83% (357/430)
INFO:pip.subprocessor:Updating files:  84% (362/430)
INFO:pip.subprocessor:Updating files:  85% (366/430)
INFO:pip.subprocessor:Updating files:  86% (370/430)
INFO:pip.subprocessor:Updating files:  87% (375/430)
INFO:pip.subprocessor:Updating files:  88% (379/430)
INFO:pip.subprocessor:Updating files:  89% (383/430)
INFO:pip.subprocessor:Updating files:  90% (387/430)
INFO:pip.subprocessor:Updating files:  91% (392/430)
INFO:pip.subprocessor:Updating files:  92% (396/430)
INFO:pip.subprocessor:Updating files:  93% (400/430)
INFO:pip.subprocessor:Updating files:  94% (405/430)
[=   ] Installing None...INFO:pip.subprocessor:Updating files:  95% (409/430)
INFO:pip.subprocessor:Updating files:  96% (413/430)
INFO:pip.subprocessor:Updating files:  97% (418/430)
INFO:pip.subprocessor:Updating files:  98% (422/430)
INFO:pip.subprocessor:Updating files:  99% (426/430)
INFO:pip.subprocessor:Updating files: 100% (430/430)
INFO:pip.subprocessor:Updating files: 100% (430/430), done.
INFO:pip.subprocessor:Running command git rev-parse -q --verify 'sha^ae25af4d593d20a26c85fbb17c0cd98a026e8595'
INFO:pip.subprocessor:Running command git fetch -q https://github.com/myhdl/myhdl.git ae25af4d593d20a26c85fbb17c0cd98a026e8595
[   =] Installing None...INFO:pip.subprocessor:Running command git checkout -q ae25af4d593d20a26c85fbb17c0cd98a026e8595
[=   ] Installing None...INFO:pipenv.patched.pip._internal.vcs.git:Resolved https://github.com/myhdl/myhdl.git to commit ae25af4d593d20a26c85fbb17c0cd98a026e8595
Added myhdl to Pipfile's [packages] ...
Installation Succeeded
Pipfile.lock not found, creating...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
Success!
[==  ] Locking...Warning: INFO:pip.subprocessor:Running command git clone --filter=blob:none --quiet https://github.com/myhdl/myhdl.git 'C:\Users\matte\AppData\Local\Temp\
pip-temp-k48u2euc\myhdl_bcdd26e1a5b54e039fed9a83be4579ad'
INFO:pip.subprocessor:Running command git rev-parse -q --verify 'sha^ae25af4d593d20a26c85fbb17c0cd98a026e8595'
INFO:pip.subprocessor:Running command git fetch -q https://github.com/myhdl/myhdl.git ae25af4d593d20a26c85fbb17c0cd98a026e8595
INFO:pip.subprocessor:Running command git checkout -q ae25af4d593d20a26c85fbb17c0cd98a026e8595
INFO:pip.subprocessor:Running command git clone --filter=blob:none https://github.com/myhdl/myhdl.git 'C:\Users\matte\AppData\Local\Temp\tmp1bqge362'
INFO:pip.subprocessor:Cloning into 'C:\Users\matte\AppData\Local\Temp\tmp1bqge362'...
INFO:pip.subprocessor:Updating files:  69% (298/430)
INFO:pip.subprocessor:Updating files:  70% (301/430)
INFO:pip.subprocessor:Updating files:  71% (306/430)
INFO:pip.subprocessor:Updating files:  72% (310/430)
INFO:pip.subprocessor:Updating files:  73% (314/430)
INFO:pip.subprocessor:Updating files:  74% (319/430)
INFO:pip.subprocessor:Updating files:  75% (323/430)
INFO:pip.subprocessor:Updating files:  76% (327/430)
INFO:pip.subprocessor:Updating files:  77% (332/430)
INFO:pip.subprocessor:Updating files:  78% (336/430)
INFO:pip.subprocessor:Updating files:  79% (340/430)
INFO:pip.subprocessor:Updating files:  80% (344/430)
INFO:pip.subprocessor:Updating files:  81% (349/430)
INFO:pip.subprocessor:Updating files:  82% (353/430)
INFO:pip.subprocessor:Updating files:  83% (357/430)
INFO:pip.subprocessor:Updating files:  84% (362/430)
INFO:pip.subprocessor:Updating files:  85% (366/430)
INFO:pip.subprocessor:Updating files:  86% (370/430)
INFO:pip.subprocessor:Updating files:  87% (375/430)
INFO:pip.subprocessor:Updating files:  88% (379/430)
INFO:pip.subprocessor:Updating files:  89% (383/430)
INFO:pip.subprocessor:Updating files:  90% (387/430)
INFO:pip.subprocessor:Updating files:  91% (392/430)
INFO:pip.subprocessor:Updating files:  92% (396/430)
INFO:pip.subprocessor:Updating files:  93% (400/430)
INFO:pip.subprocessor:Updating files:  94% (405/430)
INFO:pip.subprocessor:Updating files:  95% (409/430)
INFO:pip.subprocessor:Updating files:  96% (413/430)
INFO:pip.subprocessor:Updating files:  97% (418/430)
INFO:pip.subprocessor:Updating files:  98% (422/430)
INFO:pip.subprocessor:Updating files:  99% (426/430)
INFO:pip.subprocessor:Updating files: 100% (430/430)
INFO:pip.subprocessor:Updating files: 100% (430/430), done.
INFO:pip.subprocessor:Running command git rev-parse -q --verify 'sha^ae25af4d593d20a26c85fbb17c0cd98a026e8595'
INFO:pip.subprocessor:Running command git fetch -q https://github.com/myhdl/myhdl.git ae25af4d593d20a26c85fbb17c0cd98a026e8595
INFO:pip.subprocessor:Running command git checkout -q ae25af4d593d20a26c85fbb17c0cd98a026e8595
Locking [dev-packages] dependencies...
Updated Pipfile.lock (15e8a5eda9949e87102c00fa1884a3c3f27288c8934043801db642adb16969ca)!
Installing dependencies from Pipfile.lock (6969ca)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/issue-5973
$ cat Pipfile
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
myhdl = {ref = "ae25af4d593d20a26c85fbb17c0cd98a026e8595", git = "git+https://github.com/myhdl/myhdl.git"}

[dev-packages]

[requires]
python_version = "3.9"

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/issue-5973
$ cat Pipfile.lock
{
    "_meta": {
        "hash": {
            "sha256": "15e8a5eda9949e87102c00fa1884a3c3f27288c8934043801db642adb16969ca"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.9"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "myhdl": {
            "git": "git+https://github.com/myhdl/myhdl.git",
            "ref": "ae25af4d593d20a26c85fbb17c0cd98a026e8595"
        }
    },
    "develop": {}
}
matteius commented 1 year ago

Also just verified this example worked on mac os using latest pipenv -- I am at a loss as to what you are saying.

tobygomersall commented 1 year ago

Could this caused by different python versions?

Diff between your Pipfile and mine:

12c12
< python_version = "3.10"
---
> python_version = "3.9"

Diff between your Pipfile.lock and mine:

4c4
<             "sha256": "e967d6ee54dd082c5c5035ebd37e2a7f61e5d627203f1775300d3181df5a91a2"
---
>             "sha256": "15e8a5eda9949e87102c00fa1884a3c3f27288c8934043801db642adb16969ca"
8c8
<             "python_version": "3.10"
---
>             "python_version": "3.9"
21c21
<             "ref": null
---
>             "ref": "ae25af4d593d20a26c85fbb17c0cd98a026e8595"
matteius commented 1 year ago

The windows test was actually using:

Using default python from C:\Users\matte\AppData\Local\Programs\Python\Python311 Not sure why it had 3.9 in Pipfile (possibly separate issue)

I used a variety of pythons across the three OS tests I ran and could not reproduce it. Doesn't mean its not an issue for you, but I am not sure yet what could be causing it.

chrisroat commented 1 year ago

I ran the same test as @matteius on a fresh install, and got the same results. So I scratched my head a moment. I realized this wasn't equivalent to what I was doing.

I find that when you add to an existing Pipfile, you run into the problem here. Simply doing an empty pipenv install first will repro the issue.

$ pipenv install
...
$ pipenv install git+https://github.com/myhdl/myhdl.git@ae25af4d593d20a26c85fbb17c0cd98a026e8595
...

Now you will find that the Pipfile.lock file as a null ref. (Pipfile has correct ref)

chrisroat commented 1 year ago

It is likely worth re-labelling this as a regression (in addition to a legitimate bug), unless it is unable to be reproduced.

Aiky30 commented 6 months ago

I can replicate this issue on Pipenv 2023.12.1.

Machine: x86 i7, Ubuntu 22.04.4 64bit

Pipfile.lock example:

somepackage = { ref = "Commit hash here", git = "ssh://git@github.com/somerepo/somepackage.git" }

PipLock example:

        "somepackage": {
            "git": "ssh://git@github.com/somerepo/somepackage.git",
            "ref": null
        },

Rolling back to 2023.7.23 as others have said does indeed work. Whatever changed after this version has broken this functionality.

ohshazbot commented 4 weeks ago

I'm experiencing this with 2024.1.0

matteius commented 4 weeks ago

@ohshazbot could you check if you experience it on this branch: https://github.com/pypa/pipenv/pull/6276

ohshazbot commented 4 weeks ago

Sorry matteius, I think I got to a resolution before I saw your reply.

But in my case, I'm not sure if it's unsupported behavior, as I was trying to install a specific commit whereas all the documentation only seems to support installing a git branch. And lookikng at the log difference between specifying branch and sha, the sha approach just pulls the latest commit from the default branch. So once I flipped my request over to using a branch identifier instead it worked as intended.

So-

I will say that this was also causing other random failures that manifested as 

  File "/home/john/.local/share/virtualenvs/portal-HU6noBrU/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 723, in resolve
    raise RuntimeError("Failed to lock Pipfile.lock!")
RuntimeError: Failed to lock Pipfile.lock!

, which were pretty useless until I realized there was a verbose flag that brought my attention to errors like

pipenv.patched.pip._vendor.packaging.requirements.InvalidRequirement: Expected end or semicolon (after name and no valid version specifier)
    cornice-swagger==

This was a package I recently migrated from a requirements.txt file and worked fine with the shas.

matteius commented 4 weeks ago

Thank you so much for this analysis @ohshazbot -- it is exactly as you describe and the cause of why I opened https://github.com/pypa/pipenv/issues/6266

Your findings will help me narrow this regression down further in the future.

ohshazbot commented 4 weeks ago

Coolio, this is a company setup to sanitizing is a bit of a pain, but if there's any other details I can share LMK