search

jazzband / pip-tools

A set of tools to keep your pinned Python dependencies fresh.
https://pip-tools.rtfd.io
BSD 3-Clause "New" or "Revised" License
7.76k stars 611 forks source link

pip-compile doesn't provide hashes for wheels hosted by simple index servers #1536

Open stefansjs opened 3 years ago

stefansjs commented 3 years ago

pip-compile doesn't compute hashes for all index servers, which leads to pip install failure. Specifically, any index server that provides hashes from its json API, will prevent pip-compile for checking if other installation candidates are available from other index servers.

Environment Versions

  1. OS Type
  2. Python version: $ python -V
  3. pip version: $ pip --version
  4. pip-tools version: $ pip-compile --version

Steps to replicate

  1. host a simple index server
  2. add wheels for source packages in pypi, e.g. avro-python3==1.9.2.1
  3. list this requirement in requirements.in
  4. run pip-compile --generate-hashes -i 'http://<my local index server>.com/' --extra-index-url https://pypi.org/simple
  5. run pip install -r requirements.txt

Expected result

The hashes should include the wheel in the local index server.

Actual result

Only hashes from pypi.org are listed in requirements.txt.

Looking at the implementation, it looks like piptools first tries to get hashes from the first index server that implements the json API. The first server that responds with a json blob is taken as the only hash candidates. Files are not manually hashed for any missing files.

In this case, because some files are on pypi, which implements the json API, other index servers are able to provide valid installation candidates to pip, but piptools does not bother to hash them, thus causing a pip installation failure due to the missing hashes.

stefansjs commented 3 years ago

I've got a PR underway. I'm just trying to make sure I add enough test coverage first.

snmishra commented 2 years ago

@stefansjs I'm running into this limitation as well. Any updates on progress? Could I offer assistance?

stefansjs commented 2 years ago

@snmishra I just got back to this PR. It sat on my back burner for a little longer than I'd like to admit. So far my feature is implemented on https://github.com/stefansjs/pip-tools/tree/all_hashes but I don't think it has enough test coverage to ensure that my implementation is right in all cases.

If it's easier, maybe I should start the PR and we can discuss changes in that CR discussion. Do you think that's the best way to discuss?

snmishra commented 2 years ago

@stefansjs Sounds good

jedie commented 1 year ago

Any News here?!?

atugushev commented 1 year ago

Any News here?!?

@jedie The work is in progress, see related PR https://github.com/jazzband/pip-tools/pull/1556.

jedie commented 1 month ago

Hm. Strange, i have just add piwheels as extra urls in pip-compile call to add the package hashes from there, too?!?

See: https://github.com/jedie/victron-ble2mqtt/pull/13/commits/972fe333e64a725608f96adcba68d993f88a603c