`--require-hashes` does not correctly handle pinned package with extras

[tonybajan]

What did you want to do?

Install this requirements.txt file with pip 21.0.1 or master in a new virtual environment.

A package is pinned to a non-latest version with an extra (here, requirements[security]) and another dependency requires this package without specifying the extra.

Output

Install fails with:

...
Collecting requests[security]==2.24.0
  Using cached requests-2.24.0-py2.py3-none-any.whl (61 kB)
Collecting six==1.15.0
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting urllib3==1.25.11
  Using cached urllib3-1.25.11-py2.py3-none-any.whl (127 kB)
Collecting requests<3
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    requests<3 from https://files.pythonhosted.org/packages/29/c1/24814557f1d22c56d50280771a17307e6bf87b70727d975fd6b2ce6b014a/requests-2.25.1-py2.py3-none-any.whl#sha256=c210084e36a42ae6b9219e00e48287def368a26d03a048ddad7bfee44f75871e (from requests-extra==1.0.0b3->-r requirements.txt (line 116))

The resolver does not recognise that requests[security]==2.24.0 fulfils requests<3, and tries to collect latest requests. This fails as it is has no pinned hash.

Additional information

Installation succeeds with --use-deprecated=legacy-resolver.

If the requirements file has no hashes, installation succeeds with the new resolver: Collecting requests<3 resolves to latest requests (2.25.1) in the install output, but the pinned version (2.24.0) is what ends up installed.

If all packages are already installed in the environment, pip install succeeds (with Requirement already satisfied) even with the new resolver.

[pradyunsg]

@anentropic You can only use --require-hashes with the entire tree of dependencies pinned. See https://pip.pypa.io/en/stable/topics/secure-installs/#hash-checking-mode which also lists this as an explicit restriction.

Your failure is unrelated to this issue AFAICT.

[anentropic]

@pradyunsg I have use pdm to generate a requirements file with hashes in it, based on the pyproject.toml and pdm.lock file

but it only adds direct dependencies and not transitive dependencies

are you saying that I need to generate a requirements file that outputs all the transitive dependencies as well?

[pfmoore]

Yes.

[anentropic]

hmm, I think the problem is more subtle...

I had a closer look and my requirements file does actually have transitive deps - just not greenlet

I tried generating one with pip-compile from pip-tools and it has the same problem

so it seems to be something specific about this sqlalchemy dependency that is not handled by the tooling? https://github.com/sqlalchemy/sqlalchemy/blob/rel_2_0_22/setup.cfg#L40

[pradyunsg]

That looks like a PDM bug, and regardless, it's unrelated to this issue. Let's not have further discussion about this here. If you have more questions around this, please file a new issue (which will ask for a bunch of details that are useful for us to know so that we're able to actually help you).

[anentropic]

That looks like a PDM bug

pip-tools has the same bug if so

[merwok]

pip-tools is fine (there have been troubles, I have seen that message from --require-hashes because of extras, but with up-to-date pip and pip-tools + not using --strip-extras it creates correct requirements files for pip install and pip-sync – but I am not 100% sure that the != dependency case is the same thing). Let’s keep this discussion on-topic please!

[devashish2203]

We started running into this issue yesterday and as far as I know we haven't updated the requirements.txt in over a month.

The requirements.txt was created with the following command

> pip-compile --allow-unsafe --generate-hashes --output-file=requirements.txt requirements.in

And then running the following fails with the below error.

> pip install -r requirements.txt

ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    googleapis-common-protos<2.0.dev0,>=1.56.2 from https://files.pythonhosted.org/packages/b7/14/3f8b5670e7e082a3735dc9a6411b4ba37af7e6662441ee66782906265632/googleapis_common_protos-1.64.0-py2.py3-none-any.whl#sha256=d1bfc569f70ed2e96ccf06ead265c2cf42b5abfc817cda392e3835f3b67b5c59 (from google-api-core[grpc]==2.19.1->-r requirements.txt (line 672))

Interestingly I'm not able to reproduce this on my M1 Mac laptop, but is consistently occuring in our CI which uses a Linux python3.10 docker image. Also can reproduce locally inside the docker image.