Closed brettlangdon closed 1 year ago
None of the "Got" hashes match actual hashes for files on PyPI, so it seems like whatever has recorded these hashes prior to install has done so incorrectly or incompletely (or PyPI has served incomplete files).
We are using requirements.in where we pin specific versions, but we do not define hashes.
In order for pip to fail like this, hashes must be defined somewhere. It seems like at least one error comes from this line. I'm not at all familiar with riot
but I assume it's compiling dependencies with hashes somewhere, although after a quick glance I couldn't tell where.
This is the command Riot generates to perform requirements locking. It's the only place Riot does so, and as far as I know it doesn't use hashes.
There is a chance it could be related to CircleCI/Docker/proxy?
Not sure if this is related at all, but I just noticed a new pip error in our CI.
ERROR: Exception:
Traceback (most recent call last):
File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper
status = run_func(*args)
File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
return func(self, options, args)
File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 419, in run
requirement_set = resolver.resolve(
File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve
result = self._result = resolver.resolve(
File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 373, in resolve
failure_causes = self._attempt_to_pin_criterion(name)
File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 213, in _attempt_to_pin_criterion
criteria = self._get_updated_criteria(candidate)
File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 204, in _get_updated_criteria
self._add_to_criteria(criteria, requirement, parent=candidate)
File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 172, in _add_to_criteria
if not criterion.candidates:
File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_vendor/resolvelib/structs.py", line 151, in __bool__
return bool(self._sequence)
File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
return any(self)
File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr>
return (c for c in iterator if id(c) not in self._incompatible_ids)
File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 44, in _iter_built
for version, func in infos:
File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 279, in iter_index_candidate_infos
result = self._finder.find_best_candidate(
File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_internal/index/package_finder.py", line 890, in find_best_candidate
candidates = self.find_all_candidates(project_name)
File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_internal/index/package_finder.py", line 831, in find_all_candidates
page_candidates = list(page_candidates_it)
File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_internal/index/sources.py", line 134, in page_candidates
yield from self._candidates_from_page(self._link)
File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_internal/index/package_finder.py", line 795, in process_project_url
page_links = list(parse_links(index_response))
File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_internal/index/collector.py", line 223, in wrapper_wrapper
return list(fn(page))
File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_internal/index/collector.py", line 236, in parse_links
data = json.loads(page.content)
File "/root/.pyenv/versions/3.10.11/lib/python3.10/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/root/.pyenv/versions/3.10.11/lib/python3.10/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/root/.pyenv/versions/3.10.11/lib/python3.10/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 166956 (char 166955)
@di I have started a support case with CircleCI to explore that avenue.
Are you ok if we leave this issue open until we make some further progress? Maybe any outcomes we find can be useful for others who find this issue in the future.
@brettlangdon Yes, that's fine.
I am going to close this issue, we weren't able to find a root cause, but the issue appears to have gone away.
I expect this was an intermittent network issue somewhere, and since it seems like only we reported it then it is not likely an issue with PyPI but with CircleCI or Docker.
Describe the bug We recently started noticing intermittent failures in our CircleCI jobs like the following error:
This happens what seems at random, and has happened with a large variety of packages.
We are using
requirements.in
where we pin specific versions, but we do not define hashes.Example: https://github.com/DataDog/dd-trace-py/blob/1.x/.riot/requirements/18b8c7c.txt
Expected behavior
pip install
does not fail with "ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE" errors.To Reproduce I have not been able to reproduce yet outside of our CI environment, where it occurs randomly.
My Platform
Our CI jobs run in CircleCI in the docker container with the following image: https://github.com/DataDog/dd-trace-py/blob/2dbe9cfc04194777726338fbe1622fa7fdadef5b/docker/Dockerfile
Additional context I am not sure if this is actually a PyPI issue, but I am not sure what direction to take my investigation. Any help understanding which parts of our systems to dig further into would be greatly appreciated.