search

pypi / warehouse

The Python Package Index
https://pypi.org
Apache License 2.0
3.59k stars 966 forks source link

Intermittent "THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE" errors in CircleCI #14116

Closed brettlangdon closed 1 year ago

brettlangdon commented 1 year ago

Describe the bug We recently started noticing intermittent failures in our CircleCI jobs like the following error:

ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    gevent==22.10.2 from https://files.pythonhosted.org/packages/92/4c/d61af2f4e0319f12bb8e5643cd9c1626b1c42425829f0a0d73409a1c7a4b/gevent-22.10.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (from -r .riot/requirements/18b8c7c.txt (line 15)):
        Expected sha256 17b68f4c9e20e47ad49fe797f37f91d5bbeace8765ce2707f979a8d4ec197e4d
             Got        cc6bd65780100be4b522db0244854d7ec0af3fb290fa3a5d9931b5582b032f8f

This happens what seems at random, and has happened with a large variety of packages.

We are using requirements.in where we pin specific versions, but we do not define hashes.

Example: https://github.com/DataDog/dd-trace-py/blob/1.x/.riot/requirements/18b8c7c.txt

Expected behavior pip install does not fail with "ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE" errors.

To Reproduce I have not been able to reproduce yet outside of our CI environment, where it occurs randomly.

My Platform

Our CI jobs run in CircleCI in the docker container with the following image: https://github.com/DataDog/dd-trace-py/blob/2dbe9cfc04194777726338fbe1622fa7fdadef5b/docker/Dockerfile

Additional context I am not sure if this is actually a PyPI issue, but I am not sure what direction to take my investigation. Any help understanding which parts of our systems to dig further into would be greatly appreciated.

di commented 1 year ago

None of the "Got" hashes match actual hashes for files on PyPI, so it seems like whatever has recorded these hashes prior to install has done so incorrectly or incompletely (or PyPI has served incomplete files).

We are using requirements.in where we pin specific versions, but we do not define hashes.

In order for pip to fail like this, hashes must be defined somewhere. It seems like at least one error comes from this line. I'm not at all familiar with riot but I assume it's compiling dependencies with hashes somewhere, although after a quick glance I couldn't tell where.

emmettbutler commented 1 year ago

This is the command Riot generates to perform requirements locking. It's the only place Riot does so, and as far as I know it doesn't use hashes.

brettlangdon commented 1 year ago

There is a chance it could be related to CircleCI/Docker/proxy?

https://stackoverflow.com/questions/61551480/clean-docker-pip-install-results-in-error-these-packages-do-not-match-the-hashe/73744966#73744966

brettlangdon commented 1 year ago

Not sure if this is related at all, but I just noticed a new pip error in our CI.

ERROR: Exception:
Traceback (most recent call last):
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper
    status = run_func(*args)
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
    return func(self, options, args)
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 419, in run
    requirement_set = resolver.resolve(
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve
    result = self._result = resolver.resolve(
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 373, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 213, in _attempt_to_pin_criterion
    criteria = self._get_updated_criteria(candidate)
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 204, in _get_updated_criteria
    self._add_to_criteria(criteria, requirement, parent=candidate)
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 172, in _add_to_criteria
    if not criterion.candidates:
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_vendor/resolvelib/structs.py", line 151, in __bool__
    return bool(self._sequence)
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
    return any(self)
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr>
    return (c for c in iterator if id(c) not in self._incompatible_ids)
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 44, in _iter_built
    for version, func in infos:
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 279, in iter_index_candidate_infos
    result = self._finder.find_best_candidate(
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_internal/index/package_finder.py", line 890, in find_best_candidate
    candidates = self.find_all_candidates(project_name)
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_internal/index/package_finder.py", line 831, in find_all_candidates
    page_candidates = list(page_candidates_it)
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_internal/index/sources.py", line 134, in page_candidates
    yield from self._candidates_from_page(self._link)
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_internal/index/package_finder.py", line 795, in process_project_url
    page_links = list(parse_links(index_response))
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_internal/index/collector.py", line 223, in wrapper_wrapper
    return list(fn(page))
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_internal/index/collector.py", line 236, in parse_links
    data = json.loads(page.content)
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 166956 (char 166955)

https://app.circleci.com/pipelines/github/DataDog/dd-trace-py/40220/workflows/71f1fb13-394c-46d7-b42a-c36f44ea96f7/jobs/2703857?invite=true#step-108-424

brettlangdon commented 1 year ago

@di I have started a support case with CircleCI to explore that avenue.

Are you ok if we leave this issue open until we make some further progress? Maybe any outcomes we find can be useful for others who find this issue in the future.

di commented 1 year ago

@brettlangdon Yes, that's fine.

brettlangdon commented 1 year ago

I am going to close this issue, we weren't able to find a root cause, but the issue appears to have gone away.

I expect this was an intermittent network issue somewhere, and since it seems like only we reported it then it is not likely an issue with PyPI but with CircleCI or Docker.