search

jazzband / pip-tools

A set of tools to keep your pinned Python dependencies fresh.
https://pip-tools.rtfd.io
BSD 3-Clause "New" or "Revised" License
7.67k stars 608 forks source link

Hashes are used although pip-compile was instructed to compile from scratch using --rebuild #2056

Open manuel-koch opened 6 months ago

manuel-koch commented 6 months ago

Environment Versions

  1. MacOS, 14.3.1 (23D60), arm64
  2. Python version: $ python -V: Python 3.10.6
  3. pip version: $ pip --version: pip 24.0
  4. pip-tools version: $ pip-compile --version: pip-compile, version 7.4.0

Scenario

Using a private PyPi repository, the same version of a package was published multiple times using different hashes between consecutive invocations of pip-compile.

The scenario is as follows:

The second run of pip-compile complains about a possible tampered package - but I have not put any hashes in the requirements.in or the previously compiled requirements.txt.

The output of the second run of pip-compile:

$ pip-compile --verbose --no-emit-index-url --rebuild --resolver=backtracking /Users/user/myproject/requirements.in
Using pip-tools configuration defaults found in 'pyproject.toml'.
Using indexes:
  https://user:****@custom-pypi/simple
  Looking in indexes: https://user:****@custom-pypi/simple

                          ROUND 1                           
  Collecting more-itertools~=10.2 (from -r /Users/user/myproject/requirements.in (line 2))
    Using cached https://custom-pypi/packages/packages/50/e2/8e10e465ee3987bb7c9ab69efb91d867d93959095f4807db102d07995d94/more_itertools-10.2.0-py3-none-any.whl (57 kB)
  ....
  Collecting my-lib-package~=5.0.4rc0 (from -r /Users/user/myproject/requirements.in (line 38))
    Using cached https://custom-pypi/my-lib-package/5.0.4rc0/my_lib_package-5.0.4rc0-py3-none-any.whl (72 kB)
Traceback (most recent call last):
  File "/python3.10/bin/pip-compile", line 8, in <module>
    sys.exit(cli())
  File "/python3.10/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/python3.10/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/python3.10/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/python3.10/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/python3.10/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/python3.10/lib/python3.10/site-packages/piptools/scripts/compile.py", line 469, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File "/python3.10/lib/python3.10/site-packages/piptools/resolver.py", line 604, in resolve
    is_resolved = self._do_resolve(
  File "/python3.10/lib/python3.10/site-packages/piptools/resolver.py", line 636, in _do_resolve
    resolver.resolve(
  File "/python3.10/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
    result = self._result = resolver.resolve(
  File "/python3.10/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/python3.10/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 397, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/python3.10/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
    if not criterion.candidates:
  File "/python3.10/lib/python3.10/site-packages/pip/_vendor/resolvelib/structs.py", line 156, in __bool__
    return bool(self._sequence)
  File "/python3.10/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
    return any(self)
  File "/python3.10/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr>
    return (c for c in iterator if id(c) not in self._incompatible_ids)
  File "/python3.10/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built
    candidate = func()
  File "/python3.10/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 182, in _make_candidate_from_link
    base: Optional[BaseCandidate] = self._make_base_candidate_from_link(
  File "/python3.10/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 228, in _make_base_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "/python3.10/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 290, in __init__
    super().__init__(
  File "/python3.10/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
    self.dist = self._prepare()
  File "/python3.10/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 222, in _prepare
    dist = self._prepare_distribution()
  File "/python3.10/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 301, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
  File "/python3.10/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 525, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "/python3.10/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 596, in _prepare_linked_requirement
    local_file = unpack_url(
  File "/python3.10/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 168, in unpack_url
    file = get_http_url(
  File "/python3.10/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 111, in get_http_url
    hashes.check_against_path(from_path)
  File "/python3.10/lib/python3.10/site-packages/pip/_internal/utils/hashes.py", line 106, in check_against_path
    return self.check_against_file(file)
  File "/python3.10/lib/python3.10/site-packages/pip/_internal/utils/hashes.py", line 102, in check_against_file
    return self.check_against_chunks(read_chunks(file))
  File "/python3.10/lib/python3.10/site-packages/pip/_internal/utils/hashes.py", line 91, in check_against_chunks
    self._raise(gots)
  File "/python3.10/lib/python3.10/site-packages/pip/_internal/utils/hashes.py", line 94, in _raise
    raise HashMismatch(self._allowed, gots)
pip._internal.exceptions.HashMismatch: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    my-lib-package~=5.0.4rc0 from https://custom-pypi/my-lib-package/5.0.4rc0/my_lib_package-5.0.4rc0-py3-none-any.whl#sha256=7f9d3fc72d7e1e1fa4aabae5327579e8291b0f103ebc57d89d7afac1c69462ff (from -r /Users/user/myproject/requirements.in (line 38)):
        Expected sha256 7f9d3fc72d7e1e1fa4aabae5327579e8291b0f103ebc57d89d7afac1c69462ff
             Got        7c5f0f5ed128d985f11a10353f0bb62a046892c292b5b7975e3bf0763ff89e0a

Where does pip-compile get the earlier hash from ? How can I configure pip-compile to really compile dependencies from scratch ? Why does --rebuild still reuse pre-existing hashes when those hashes are absent from any requirements.in or requirements.txt ?

manuel-koch commented 6 months ago

I already tried running pip cache purge but the problem still happens.

webknjaz commented 6 months ago

Do you have a config file lying around? Do you use -c anywhere? Have you tried passing an explicit argument?

manuel-koch commented 6 months ago

No config-file used ( except the requirements.in and requirements.txt ) and no use of -c anywhere AFAIK.

manuel-koch commented 6 months ago

I suspect that pip is comparing the hash of the currently installed package my-lib-package with the (now updated) hash of the remote package from the private pypi repo.

But why should this be the case - I wanted pip-compile to compile requirements from scratch ( hence the --rebuild option ), regardless of whether I have (some) of the packages already installed in my python environment.

webknjaz commented 6 months ago

I suspect that pip is comparing the hash of the currently installed package my-lib-package with the (now updated) hash of the remote package from the private pypi repo.

Was it previously recorded in the constraints txt file? Normally, pip demands hashes everywhere when it sees at least one --hash. Have you tried adding something like -vvvvv --pip-args='-vvvv' to your pip-compile command?

webknjaz commented 6 months ago

I already tried running pip cache purge but the problem still happens.

Does it still output that Using cached https://custom-pypi/my-lib-package/5.0.4rc0/my_lib_package-5.0.4rc0-py3-none-any.whl (72 kB) after the cache purge?

webknjaz commented 6 months ago

Also, how is my-lib-package~=5.0.4rc0 defined in requirements.in? Can you find any other entries of my-lib-package with something like grep -r? Do you reference . or -e . anywhere?

webknjaz commented 6 months ago

Also, grep for 7f9d3fc72d7e1e1fa4aabae5327579e8291b0f103ebc57d89d7afac1c69462ff too.

manuel-koch commented 6 months ago

Was it previously recorded in the constraints txt file?

Yes, but only as the resolved version, e.g. my-lib-package==5.0.4rc0

Also, how is my-lib-package~=5.0.4rc0 defined in requirements.in?

It was configured as my-lib-package~=5.0.4rc0

Can you find any other entries of my-lib-package with something like grep -r?

A sub-dependency was also depending on a compatible/same version, referring to it in its own dependencies as my-lib-package~=5.0.4rc0

Do you reference . or -e . anywhere?

No.

Does it still output that Using cached https://custom-pypi/my-lib-package/5.0.4rc0/my_lib_package-5.0.4rc0-py3-none-any.whl (72 kB) after the cache purge?

No, there is no corresponding *.whl in the cache and that message is not printed.

Also, grep for 7f9d3fc72d7e1e1fa4aabae5327579e8291b0f103ebc57d89d7afac1c69462ff too.

No results when executing grep -R 7f9d3fc72d7e1e1fa4aabae5327579e8291b0f103ebc57d89d7afac1c69462ff ~.

We have refactored the way we publish our packages to the private pypi repo, not re-using the same version on publishing an updated ( rc versioned ) package, as this seems to be in general a bad approach.

I will try to reproduce the issue again and try out the idea of -vvvvv --pip-args='-vvvv' to the pip-compile command.

gschaffner commented 4 months ago

we just encountered what looks like this issue:

pip-compile --quiet --upgrade --output-file requirements/dev-linux-3.12.lock requirements/dev.in
Traceback (most recent call last):
  File "/home/runner/work/pydux/pydux/pydux/.nox/devenv-3-12/bin/pip-compile", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/runner/work/pydux/pydux/pydux/.nox/devenv-3-12/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/pydux/pydux/pydux/.nox/devenv-3-12/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/runner/work/pydux/pydux/pydux/.nox/devenv-3-12/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/pydux/pydux/pydux/.nox/devenv-3-12/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/pydux/pydux/pydux/.nox/devenv-3-12/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/pydux/pydux/pydux/.nox/devenv-3-12/lib/python3.12/site-packages/piptools/scripts/compile.py", line 469, in cli
    results = resolver.resolve(max_rounds=max_rounds)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/pydux/pydux/pydux/.nox/devenv-3-12/lib/python3.12/site-packages/piptools/resolver.py", line 604, in resolve
    is_resolved = self._do_resolve(
                  ^^^^^^^^^^^^^^^^^
  File "/home/runner/work/pydux/pydux/pydux/.nox/devenv-3-12/lib/python3.12/site-packages/piptools/resolver.py", line 636, in _do_resolve
    resolver.resolve(
  File "/home/runner/work/pydux/pydux/pydux/.nox/devenv-3-12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 179, in resolve
    self.factory.preparer.prepare_linked_requirements_more(reqs)
  File "/home/runner/work/pydux/pydux/pydux/.nox/devenv-3-12/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 552, in prepare_linked_requirements_more
    self._complete_partial_requirements(
  File "/home/runner/work/pydux/pydux/pydux/.nox/devenv-3-12/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 487, in _complete_partial_requirements
    self._prepare_linked_requirement(req, parallel_builds)
  File "/home/runner/work/pydux/pydux/pydux/.nox/devenv-3-12/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 612, in _prepare_linked_requirement
    hashes.check_against_path(file_path)
  File "/home/runner/work/pydux/pydux/pydux/.nox/devenv-3-12/lib/python3.12/site-packages/pip/_internal/utils/hashes.py", line 106, in check_against_path
    return self.check_against_file(file)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/pydux/pydux/pydux/.nox/devenv-3-12/lib/python3.12/site-packages/pip/_internal/utils/hashes.py", line 102, in check_against_file
    return self.check_against_chunks(read_chunks(file))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/pydux/pydux/pydux/.nox/devenv-3-12/lib/python3.12/site-packages/pip/_internal/utils/hashes.py", line 91, in check_against_chunks
    self._raise(gots)
  File "/home/runner/work/pydux/pydux/pydux/.nox/devenv-3-12/lib/python3.12/site-packages/pip/_internal/utils/hashes.py", line 94, in _raise
    raise HashMismatch(self._allowed, gots)
pip._internal.exceptions.HashMismatch: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    unknown package:
        Expected sha256 ca1cb80a17d6999c827815ba10cfcb7fd8deaf0db176d06549ca266298390da8
             Got        405cf084fd8fc3cc7038a187ad233bce9c9f960ebf93281965c95e11e8925c04

here's the details of the environment this happened in:

(click to expand) cache state, dependency versions, config, and inputs CPython 3.12.2 in an ephemeral CI runner: GHA's ubuntu-latest. the job started with no pip/pip-tools caches. the first step was checkout, the second was ```yaml - name: Set up Nox uses: wntrblm/nox@6646b24d62e69442a55502e460e33c252d54beb4 with: python-versions: '3.12' ``` notably this step modifies pip's cache via `pipx install /path/to/nox`. the next step created a virtualenv (everything after ocurred in that virtualenv) and did ```console pip install -c requirements/dev-linux-3.12.lock pip-tools pip-sync requirements/dev-linux-3.12.lock pip install -c requirements/dev-linux-3.12.lock -e . pip install -c requirements/dev-linux-3.12.lock pytest-github-actions-annotate-failures pip-compile --quiet --upgrade --output-file requirements/dev-linux-3.12.lock requirements/dev.in # this is the one that raised the HashMismatch ``` the package at `.` does not specify any dependencies and just uses ```toml [build-system] requires = [ "hatchling", "hatch-vcs", ] build-backend = "hatchling.build" ``` pip-tools config files are just: ```toml [tool.pip-tools] strip-extras = true allow-unsafe = true ``` `requirements/dev-linux-3.12.lock` (i.e. the state of the venv): ``` # # This file is autogenerated by pip-compile with Python 3.12 # by the following command: # # https://github.com/.../actions/workflows/pin.yaml # attrs==23.2.0 # via flake8-bugbear black==24.2.0 # via # -r requirements/dev.in # flake8-black build==1.1.1 # via pip-tools click==8.1.7 # via # black # pip-tools contourpy==1.2.0 # via matplotlib coverage==7.4.3 # via # coverage # pytest-cov cycler==0.12.1 # via matplotlib execnet==2.0.2 # via pytest-xdist flake8==7.0.0 # via # flake8-artiq # flake8-black # flake8-bugbear # flake8-isort # flake8-pyproject flake8-artiq @ git+https://gitlab.com/duke-artiq/flake8-artiq.git # via -r requirements/dev.in flake8-black==0.3.6 # via -r requirements/dev.in flake8-bugbear==24.2.6 # via -r requirements/dev.in flake8-isort==6.1.1 # via -r requirements/dev.in flake8-pyproject==1.2.3 # via -r requirements/dev.in fonttools==4.49.0 # via matplotlib iniconfig==2.0.0 # via pytest isort==5.13.2 # via # -r requirements/dev.in # flake8-isort kiwisolver==1.4.5 # via matplotlib matplotlib==3.8.3 # via -r requirements/base.in mccabe==0.7.0 # via flake8 mpmath==1.3.0 # via sympy mypy-extensions==1.0.0 # via black numpy==1.26.4 # via # -r requirements/base.in # contourpy # matplotlib # qutip # scipy # thermocouples-reference packaging==23.2 # via # black # build # matplotlib # pytest # qutip pathspec==0.12.1 # via black pillow==10.2.0 # via # -r requirements/base.in # matplotlib pip-tools==7.4.0 # via -r requirements/dev.in platformdirs==4.2.0 # via black pluggy==1.4.0 # via pytest pycodestyle==2.11.1 # via flake8 pyflakes==3.2.0 # via flake8 pyparsing==3.1.1 # via matplotlib pyproject-hooks==1.0.0 # via # build # pip-tools pytest==8.0.2 # via # -r requirements/dev.in # pytest-cov # pytest-xdist pytest-cov==4.1.0 # via -r requirements/dev.in pytest-xdist==3.5.0 # via -r requirements/dev.in python-dateutil==2.8.2 # via matplotlib qutip==4.7.5 # via -r requirements/base.in scipy==1.12.0 # via # -r requirements/base.in # qutip six==1.16.0 # via # -r requirements/base.in # python-dateutil sympy==1.12 # via -r requirements/base.in thermocouples-reference==0.20 # via -r requirements/base.in wheel==0.42.0 # via pip-tools # The following packages are considered to be unsafe in a requirements file: pip==24.0 # via pip-tools setuptools==69.1.1 # via pip-tools ``` `requirements/dev.in`: ``` pip-tools black isort flake8-pyproject flake8-black flake8-isort flake8-bugbear flake8-artiq @ git+https://gitlab.com/duke-artiq/flake8-artiq.git -r base.in pytest pytest-xdist pytest-cov ``` `requirements/base.in`: ``` matplotlib numpy Pillow qutip scipy six sympy thermocouples_reference ```

some investigation revealed that ca1cb80a17d6999c827815ba10cfcb7fd8deaf0db176d06549ca266298390da8 corresponds to qutip-5.0.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl: https://pypi.org/simple/qutip/ has

<a href="https://files.pythonhosted.org/packages/9b/6e/141227fbab01d4f413702627fb286f3e118264d5e0b3300d0cf871eba991/qutip-5.0.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=ca1cb80a17d6999c827815ba10cfcb7fd8deaf0db176d06549ca266298390da8" data-dist-info-metadata="sha256=0cca2dcd4bf5f592a1f66c827f32bfe3a679662f65105b07b42ac1ffda916b7a" data-core-metadata="sha256=0cca2dcd4bf5f592a1f66c827f32bfe3a679662f65105b07b42ac1ffda916b7a">qutip-5.0.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl</a><br />

ca1cb80a17d6999c827815ba10cfcb7fd8deaf0db176d06549ca266298390da8 does actually match that wheel when downloaded.

beyond that, i don't know how it received/wrote/hashed a corrupted/different file.

second instance

i attempted many times to reproduce this with -vvvvvvvvvv --pip-args=-vvvvvvvvvv, locally and on GHA, but it is flaky and only ended up reproducing once. this time, on GHA's macos-latest. this was also on CPython 3.12.2, and with a requirements/dev-darwin-3.12.lock that is identical to requirements/dev-linux-3.12.lock. here's the very verbose log: https://gist.github.com/gschaffner/822423dd56436490b9f56685f9c216c2

some investigation revealed that 3f2f7b774b07c1c87c4872ffb3a49bf45fb6aefba011f35d18f36f1c51bec05e corresponds to qutip-5.0.0-cp312-cp312-macosx_10_9_x86_64.whl: https://pypi.org/simple/qutip/ has

<a href="https://files.pythonhosted.org/packages/4d/e2/e8f2f0ca8e65beef392ed8847ad6b30375d36cac55a7769d8af472831fef/qutip-5.0.0-cp312-cp312-macosx_10_9_x86_64.whl#sha256=3f2f7b774b07c1c87c4872ffb3a49bf45fb6aefba011f35d18f36f1c51bec05e" data-dist-info-metadata="sha256=0cca2dcd4bf5f592a1f66c827f32bfe3a679662f65105b07b42ac1ffda916b7a" data-core-metadata="sha256=0cca2dcd4bf5f592a1f66c827f32bfe3a679662f65105b07b42ac1ffda916b7a">qutip-5.0.0-cp312-cp312-macosx_10_9_x86_64.whl</a><br />

3f2f7b774b07c1c87c4872ffb3a49bf45fb6aefba011f35d18f36f1c51bec05e does actually match that wheel when downloaded.

anyway, here's grep -F 'qutip-5.0.0-cp312-cp312-macosx_10_9_x86_64.whl' that-vvvery-vverbose-log:

    Found link https://files.pythonhosted.org/packages/4d/e2/e8f2f0ca8e65beef392ed8847ad6b30375d36cac55a7769d8af472831fef/qutip-5.0.0-cp312-cp312-macosx_10_9_x86_64.whl (from https://pypi.org/simple/qutip/), version: 5.0.0
    Obtaining dependency information for qutip from https://files.pythonhosted.org/packages/4d/e2/e8f2f0ca8e65beef392ed8847ad6b30375d36cac55a7769d8af472831fef/qutip-5.0.0-cp312-cp312-macosx_10_9_x86_64.whl.metadata
    Looking up "https://files.pythonhosted.org/packages/4d/e2/e8f2f0ca8e65beef392ed8847ad6b30375d36cac55a7769d8af472831fef/qutip-5.0.0-cp312-cp312-macosx_10_9_x86_64.whl.metadata" in the cache
    https://files.pythonhosted.org:443 "GET /packages/4d/e2/e8f2f0ca8e65beef392ed8847ad6b30375d36cac55a7769d8af472831fef/qutip-5.0.0-cp312-cp312-macosx_10_9_x86_64.whl.metadata HTTP/1.1" 200 9613
    Downloading qutip-5.0.0-cp312-cp312-macosx_10_9_x86_64.whl.metadata (9.6 kB)
    Updating cache with response from "https://files.pythonhosted.org/packages/4d/e2/e8f2f0ca8e65beef392ed8847ad6b30375d36cac55a7769d8af472831fef/qutip-5.0.0-cp312-cp312-macosx_10_9_x86_64.whl.metadata"
  Looking up "https://files.pythonhosted.org/packages/4d/e2/e8f2f0ca8e65beef392ed8847ad6b30375d36cac55a7769d8af472831fef/qutip-5.0.0-cp312-cp312-macosx_10_9_x86_64.whl" in the cache
  https://files.pythonhosted.org:443 "GET /packages/4d/e2/e8f2f0ca8e65beef392ed8847ad6b30375d36cac55a7769d8af472831fef/qutip-5.0.0-cp312-cp312-macosx_10_9_x86_64.whl HTTP/1.1" 200 10187154
  Downloading qutip-5.0.0-cp312-cp312-macosx_10_9_x86_64.whl (10.2 MB)
  Downloading link https://files.pythonhosted.org/packages/4d/e2/e8f2f0ca8e65beef392ed8847ad6b30375d36cac55a7769d8af472831fef/qutip-5.0.0-cp312-cp312-macosx_10_9_x86_64.whl (from https://pypi.org/simple/qutip/) to /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-unpack-3r2m9058/qutip-5.0.0-cp312-cp312-macosx_10_9_x86_64.whl

i noticed that the logged HTTP response length 10187154 is the correct length of qutip-5.0.0-cp312-cp312-macosx_10_9_x86_64.whl, but nothing else in the logs caught my eye or my greps.

i'm not sure why both instances i have seen this bug in have both involved qutip 5.0.0 wheels. at time of writing qutip 5.0.0 was released to PyPI 3 days before both of the two buggy runs above. these wheels were uploaded 3 days ago too; qutip didn't do delayed uploading of wheels there.

anyway, let me know if you have any ideas based on this info. perhaps we just got unlucky with some errors that passed TCP's weak checksums? perhaps some tiny fraction of PyPI's nodes have some rotted files for qutip 5.0.0 cached? i've got no clue.