--require-hashes mode problem

[canassa]

Creating a lock file with pytest==7.3.1 generates an invalid lock file

Environment Versions

1. OS Type: OSX
2. Python version: 3.9.16
3. pip version: 22.0.4
4. pip-tools version: 6.13.0

Steps to replicate

Create a lockfile with pytest==7.3.1

pytest==7.3.1

Compile it with:

pip-compile --quiet --resolver=backtracking --upgrade --generate-hashes

Now install it:

pip install -r requirements/development.lock
Looking in indexes: https://hatch:****@gitlab.com/api/v4/projects/44758594/packages/pypi/simple
Collecting iniconfig==2.0.0
  Using cached iniconfig-2.0.0-py3-none-any.whl (5.9 kB)
Collecting packaging==23.1
  Using cached packaging-23.1-py3-none-any.whl (48 kB)
Collecting pluggy==1.0.0
  Using cached pluggy-1.0.0-py2.py3-none-any.whl (13 kB)
Collecting pytest==7.3.1
  Using cached pytest-7.3.1-py3-none-any.whl (320 kB)
Collecting tomli>=1.0.0
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    tomli>=1.0.0 from https://files.pythonhosted.org/packages/97/75/10a9ebee3fd790d20926a90a2547f0bf78f371b2f13aa822c759680ca7b9/tomli-2.0.1-py3-none-any.whl#sha256=939de3e7a6161af0c887ef91b7d41a53e7c5a1ca976325f429cb46ea9bc30ecc (from pytest==7.3.1->-r requirements/development.lock (line 21))
WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available.
You should consider upgrading via the '/Users/cesarcanassa/code/src/gitlab.com/new10/services/payment-service/.venv/bin/python -m pip install --upgrade pip' command.

Expected result

It should install correctly

Actual result

It fails while installing it

[webknjaz]

Looks like a pip issue, unless you have input and output files to show with a full reproducer...

[canassa]

I tested with the latest pip:

pip --version
pip 23.1.2

Same error:

pip install -r requirements/development.lock
Looking in indexes: https://hatch:****@gitlab.com/api/v4/projects/44758594/packages/pypi/simple
Collecting iniconfig==2.0.0 (from -r requirements/development.lock (line 9))
  Using cached iniconfig-2.0.0-py3-none-any.whl (5.9 kB)
Collecting packaging==23.1 (from -r requirements/development.lock (line 13))
  Using cached packaging-23.1-py3-none-any.whl (48 kB)
Collecting pluggy==1.0.0 (from -r requirements/development.lock (line 17))
  Using cached pluggy-1.0.0-py2.py3-none-any.whl (13 kB)
Collecting pytest==7.3.1 (from -r requirements/development.lock (line 21))
  Using cached pytest-7.3.1-py3-none-any.whl (320 kB)
Collecting exceptiongroup>=1.0.0rc8 (from pytest==7.3.1->-r requirements/development.lock (line 21))
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    exceptiongroup>=1.0.0rc8 from https://files.pythonhosted.org/packages/61/97/17ed81b7a8d24d8f69b62c0db37abbd8c0042d4b3fc429c73dab986e7483/exceptiongroup-1.1.1-py3-none-any.whl (from pytest==7.3.1->-r requirements/development.lock (line 21))

This is the input file

pytest==7.3.1

This is the lock file:

#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
#    pip-compile --generate-hashes --output-file=requirements/development.lock --resolver=backtracking requirements/development.txt
#

iniconfig==2.0.0 \
    --hash=sha256:2d91e135bf72d31a410b17c16da610a82cb55f6b0477d1a902134b24a455b8b3 \
    --hash=sha256:b6a85871a79d2e3b22d2d1b94ac2824226a63c6b741c88f7ae975f18b6778374
    # via pytest
packaging==23.1 \
    --hash=sha256:994793af429502c4ea2ebf6bf664629d07c1a9fe974af92966e4b8d2df7edc61 \
    --hash=sha256:a392980d2b6cffa644431898be54b0045151319d1e7ec34f0cfed48767dd334f
    # via pytest
pluggy==1.0.0 \
    --hash=sha256:4224373bacce55f955a878bf9cfa763c1e360858e330072059e10bad68531159 \
    --hash=sha256:74134bbf457f031a36d68416e1509f34bd5ccc019f0bcc952c7b909d06b37bd3
    # via pytest
pytest==7.3.1 \
    --hash=sha256:3799fa815351fea3a5e96ac7e503a96fa51cc9942c3753cda7651b93c1cfa362 \
    --hash=sha256:434afafd78b1d78ed0addf160ad2b77a30d35d4bdf8af234fe621919d9ed15e3
    # via -r requirements/development.txt

[AndydeCleyre]

You say you're using Python 3.9.16, and you may be for installing, but as your lockfile indicates, you've compiled with Python 3.11.

Pytest requires tomli only when using Python <3.11, so tomli does not make it into the lockfile (and so no hash for it either).

If you use the same environment (or at least the same Python version) for compiling as installing, it should work.

[webknjaz]

If you use the same environment (or at least the same Python version) for compiling as installing, it should work.

Only under pre-resolvelib pip or with -r and not -c — with the new dependency resolver, there's a bug with -c+hashes that hasn't yet been solved..

[jedie]

I have the same Problem: I add 'tomli;python_version<"3.11"' to dependencies and pip-compile add this:

...
tomli==2.0.1 ; python_version < "3.11" \
    --hash=sha256:939de3e7a6161af0c887ef91b7d41a53e7c5a1ca976325f429cb46ea9bc30ecc \
    --hash=sha256:de526c12914f0c550d15924c62d72abc48d6fe7364aa87328337a31007fe8a4f
...

This looks absolutely correct, isn't it?

pip failed installing with Python 3.11:

 ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    tomli>=1.1.0 from https://files.pythonhosted.org/packages/97/75/10a9ebee3fd790d20926a90a2547f0bf78f371b2f13aa822c759680ca7b9/tomli-2.0.1-py3-none-any.whl (from flynt==0.77->-r /tmp/tmpicdwstbt (line 309))

With Python 3.10 and 3.9 installing works. Full outputs are here: https://github.com/jedie/energymeter2mqtt/actions/runs/5235346165/jobs/9452127265?pr=5#step:4:233

So it's a pip bug https://github.com/pypa/pip/issues/9644 , isn't it?

[NfNitLoop]

It seems like the bug here is that pip-compile should resolve all dependencies for all python versions, so that the requirements.txt that it generates with --generate-hashes will work on multiple systems (& python versions). Locking down your dependencies seems less useful if you can't then get back to that state on another machine.

We ran into a build failure today because ipython specifies a conditional dependency like @jedie mentions above. As a result, pip install -r requirements.txt works on one build environment, and not on another.

Without this we'll just have to stop using --generate-hashes.