search

pypa / pip

The Python package installer
https://pip.pypa.io/
MIT License
9.51k stars 3.02k forks source link

pip ignores index pages with Content-Type not defined in PEP 691 #11340

Open hmstepanek opened 2 years ago

hmstepanek commented 2 years ago

Description

Our github action's CI has been failing sporadically with the following errors since 7/26:

Example 1:

https://github.com/newrelic/newrelic-python-agent/runs/7621668567?check_suite_focus=true

WARNING: Skipping page https://pypi.org/simple/pytest/ because the GET request got Content-Type: Unknown. The only supported Content-Types are application/vnd.pypi.simple.v1+json, application/vnd.pypi.simple.v1+html, and text/html
ERROR: Could not find a version that satisfies the requirement pytest==6.2.5 (from versions: none)
ERROR: No matching distribution found for pytest==6.2.5

Example 2:

https://github.com/newrelic/newrelic-python-agent/runs/7621668309?check_suite_focus=true

WARNING: Skipping page https://pypi.org/simple/beautifulsoup4/ because the GET request got Content-Type: Unknown. The only supported Content-Types are application/vnd.pypi.simple.v1+json, application/vnd.pypi.simple.v1+html, and text/html
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of pytest to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement beautifulsoup4 (from webtest) (from versions: none)
ERROR: No matching distribution found for beautifulsoup4

We tried explicitly disabling the caching directory but this did not fix the issue. At around the same time this issue began, we noticed the following issue was filled in the pip repo: https://github.com/pypi/warehouse/issues/11949. We also noticed that the following issue appears to be the same issue we are running into now though the root cause may be different: https://github.com/pypa/pip/issues/5345.

Expected behavior

.tox/postgres-datastore_psycopg2-py39-psycopg20208/bin/pip install pytest==6.2.5 succeeds.

pip version

22.2.1

Python version

I have personally seen failures on 3.9.13, 3.8.13, 2.7.18. These failures appear to be independent of the Python version.

OS

Ubuntu 20.04.4 LTS

How to Reproduce

  1. Kick off a test run by clicking the refresh icon (labeled "Re-run this job") on this page. image and continue to do so until the tests fail (sometimes it fails the first time, sometimes it takes 2 or 3 runs to fail).

**Note we have not been able to reproduce this locally on our machines, only inside our Github Actions CI.

Output

https://github.com/newrelic/newrelic-python-agent/runs/7634470208?check_suite_focus=true

[9046] /home/runner/work/newrelic-python-agent/newrelic-python-agent$ /home/runner/work/newrelic-python-agent/newrelic-python-agent/.tox/postgres-datastore_psycopg2-py39-psycopg20208/bin/pip install pytest==6.2.5 iniconfig pytest-cov WebTest==2.0.35 'psycopg2-binary<2.9'
WARNING: Skipping page https://pypi.org/simple/pytest/ because the GET request got Content-Type: Unknown. The only supported Content-Types are application/vnd.pypi.simple.v1+json, application/vnd.pypi.simple.v1+html, and text/html
ERROR: Could not find a version that satisfies the requirement pytest==6.2.5 (from versions: none)
ERROR: No matching distribution found for pytest==6.2.5
ERROR: invocation failed (exit code 1)
ERROR: could not install deps [pytest==6.2.5, iniconfig, pytest-cov, WebTest==2.0.35, psycopg2-binary<2.9]; v = InvocationError("/home/runner/work/newrelic-python-agent/newrelic-python-agent/.tox/postgres-datastore_psycopg2-py39-psycopg20208/bin/pip install pytest==6.2.5 iniconfig pytest-cov WebTest==2.0.35 'psycopg2-binary<2.9'", 1)
postgres-datastore_psycopg2-py39-psycopg20208 finish: getenv /home/runner/work/newrelic-python-agent/newrelic-python-agent/.tox/postgres-datastore_psycopg2-py39-psycopg20208 after 1.94 seconds
cleanup /home/runner/work/newrelic-python-agent/newrelic-python-agent/.tox/.tmp/package/2/newrelic-0.0.zip

=================================== log end ====================================
✖ FAIL postgres-datastore_psycopg2-py39-psycopg20208 in 4.419 seconds
postgres-datastore_psycopg2cffi-py36-psycopg2cffi0207 uses /opt/hostedtoolcache/Python/3.6.15/x64/bin/python3.6
postgres-datastore_psycopg2cffi-py36-psycopg2cffi0207 start: parallel postgres-datastore_psycopg2cffi-py36-psycopg2cffi0207 
[9112] /home/runner/work/newrelic-python-agent/newrelic-python-agent$ /opt/hostedtoolcache/Python/3.10.5/x64/bin/python3.10 /opt/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/tox/__main__.py -vv -e postgres-datastore_asyncpg-py37,postgres-datastore_asyncpg-py39,postgres-datastore_postgresql-py36,postgres-datastore_postgresql-py38,postgres-datastore_psycopg2-py27-psycopg20208,postgres-datastore_psycopg2-py37-psycopg20208,postgres-datastore_psycopg2-py39-psycopg20208,postgres-datastore_psycopg2cffi-py27-psycopg2cffi0207,postgres-datastore_psycopg2cffi-py36-psycopg2cffi0207,postgres-datastore_psycopg2cffi-pypy-psycopg2cffi0207,postgres-datastore_psycopg2cffi-py37-psycopg2cffi0208,postgres-datastore_psycopg2cffi-py39-psycopg2cffi0208 -p auto --installpkg .tox/.tmp/package/1/newrelic-0.0.zip >.tox/postgres-datastore_psycopg2cffi-py36-psycopg2cffi0207/log/postgres-datastore_psycopg2cffi-py36-psycopg2cffi0207-0.log
postgres-datastore_psycopg2cffi-py27-psycopg2cffi0207 finish: parallel postgres-datastore_psycopg2cffi-py27-psycopg2cffi0207  after 20.55 seconds
✔ OK postgres-datastore_psycopg2cffi-py27-psycopg2cffi0207 in 20.555 seconds
pypy (/opt/hostedtoolcache/PyPy/2.7.18/x64/bin/pypy) is {'executable': '/opt/hostedtoolcache/PyPy/2.7.18/x64/bin/pypy', 'implementation': 'PyPy', 'version_info': [2, 7, 18, 'final', 42], 'version': '2.7.18 (8e99af2f9b8e3a58dd8691378a36ef43c3139ee9, Mar 29 2022, 05:16:07)\n[PyPy 7.3.9 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)]', 'is_64': True, 'sysplatform': 'linux2', 'os_sep': '/', 'extra_version_info': [7, 3, 9, 'final', 0]}
postgres-datastore_psycopg2cffi-pypy-psycopg2cffi0207 uses /opt/hostedtoolcache/PyPy/2.7.18/x64/bin/pypy
postgres-datastore_psycopg2cffi-pypy-psycopg2cffi0207 start: parallel postgres-datastore_psycopg2cffi-pypy-psycopg2cffi0207 
[9962] /home/runner/work/newrelic-python-agent/newrelic-python-agent$ /opt/hostedtoolcache/Python/3.10.5/x64/bin/python3.10 /opt/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/tox/__main__.py -vv -e postgres-datastore_asyncpg-py37,postgres-datastore_asyncpg-py39,postgres-datastore_postgresql-py36,postgres-datastore_postgresql-py38,postgres-datastore_psycopg2-py27-psycopg20208,postgres-datastore_psycopg2-py37-psycopg20208,postgres-datastore_psycopg2-py39-psycopg20208,postgres-datastore_psycopg2cffi-py27-psycopg2cffi0207,postgres-datastore_psycopg2cffi-py36-psycopg2cffi0207,postgres-datastore_psycopg2cffi-pypy-psycopg2cffi0207,postgres-datastore_psycopg2cffi-py37-psycopg2cffi0208,postgres-datastore_psycopg2cffi-py39-psycopg2cffi0208 -p auto --installpkg .tox/.tmp/package/1/newrelic-0.0.zip >.tox/postgres-datastore_psycopg2cffi-pypy-psycopg2cffi0207/log/postgres-datastore_psycopg2cffi-pypy-psycopg2cffi0207-0.log
postgres-datastore_psycopg2cffi-py36-psycopg2cffi0207 finish: parallel postgres-datastore_psycopg2cffi-py36-psycopg2cffi0207  after 21.97 seconds
✔ OK postgres-datastore_psycopg2cffi-py36-psycopg2cffi0207 in 21.973 seconds
postgres-datastore_psycopg2cffi-py37-psycopg2cffi0208 uses /opt/hostedtoolcache/Python/3.7.13/x64/bin/python3.7
postgres-datastore_psycopg2cffi-py37-psycopg2cffi0208 start: parallel postgres-datastore_psycopg2cffi-py37-psycopg2cffi0208 
[10603] /home/runner/work/newrelic-python-agent/newrelic-python-agent$ /opt/hostedtoolcache/Python/3.10.5/x64/bin/python3.10 /opt/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/tox/__main__.py -vv -e postgres-datastore_asyncpg-py37,postgres-datastore_asyncpg-py39,postgres-datastore_postgresql-py36,postgres-datastore_postgresql-py38,postgres-datastore_psycopg2-py27-psycopg20208,postgres-datastore_psycopg2-py37-psycopg20208,postgres-datastore_psycopg2-py39-psycopg20208,postgres-datastore_psycopg2cffi-py27-psycopg2cffi0207,postgres-datastore_psycopg2cffi-py36-psycopg2cffi0207,postgres-datastore_psycopg2cffi-pypy-psycopg2cffi0207,postgres-datastore_psycopg2cffi-py37-psycopg2cffi0208,postgres-datastore_psycopg2cffi-py39-psycopg2cffi0208 -p auto --installpkg .tox/.tmp/package/1/newrelic-0.0.zip >.tox/postgres-datastore_psycopg2cffi-py37-psycopg2cffi0208/log/postgres-datastore_psycopg2cffi-py37-psycopg2cffi0208-0.log
postgres-datastore_psycopg2cffi-py37-psycopg2cffi0208 finish: parallel postgres-datastore_psycopg2cffi-py37-psycopg2cffi0208  after 20.76 seconds
✔ OK postgres-datastore_psycopg2cffi-py37-psycopg2cffi0208 in 20.764 seconds
postgres-datastore_psycopg2cffi-py39-psycopg2cffi0208 uses /opt/hostedtoolcache/Python/3.9.13/x64/bin/python3.9
postgres-datastore_psycopg2cffi-py39-psycopg2cffi0208 start: parallel postgres-datastore_psycopg2cffi-py39-psycopg2cffi0208 
[11484] /home/runner/work/newrelic-python-agent/newrelic-python-agent$ /opt/hostedtoolcache/Python/3.10.5/x64/bin/python3.10 /opt/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/tox/__main__.py -vv -e postgres-datastore_asyncpg-py37,postgres-datastore_asyncpg-py39,postgres-datastore_postgresql-py36,postgres-datastore_postgresql-py38,postgres-datastore_psycopg2-py27-psycopg20208,postgres-datastore_psycopg2-py37-psycopg20208,postgres-datastore_psycopg2-py39-psycopg20208,postgres-datastore_psycopg2cffi-py27-psycopg2cffi0207,postgres-datastore_psycopg2cffi-py36-psycopg2cffi0207,postgres-datastore_psycopg2cffi-pypy-psycopg2cffi0207,postgres-datastore_psycopg2cffi-py37-psycopg2cffi0208,postgres-datastore_psycopg2cffi-py39-psycopg2cffi0208 -p auto --installpkg .tox/.tmp/package/1/newrelic-0.0.zip >.tox/postgres-datastore_psycopg2cffi-py39-psycopg2cffi0208/log/postgres-datastore_psycopg2cffi-py39-psycopg2cffi0208-0.log
postgres-datastore_psycopg2cffi-pypy-psycopg2cffi0207 finish: parallel postgres-datastore_psycopg2cffi-pypy-psycopg2cffi0207  after 27.38 seconds
✔ OK postgres-datastore_psycopg2cffi-pypy-psycopg2cffi0207 in 29.762 seconds
postgres-datastore_psycopg2cffi-py39-psycopg2cffi0208 finish: parallel postgres-datastore_psycopg2cffi-py39-psycopg2cffi0208  after 20.10 seconds
✔ OK postgres-datastore_psycopg2cffi-py39-psycopg2cffi0208 in 20.1 seconds
___________________________________ summary ____________________________________
  postgres-datastore_asyncpg-py37: commands succeeded
  postgres-datastore_asyncpg-py39: commands succeeded
  postgres-datastore_postgresql-py36: commands succeeded
  postgres-datastore_postgresql-py38: commands succeeded
  postgres-datastore_psycopg2-py27-psycopg20208: commands succeeded
  postgres-datastore_psycopg2-py37-psycopg20208: commands succeeded
ERROR:   postgres-datastore_psycopg2-py39-psycopg20208: parallel child exit code 1
  postgres-datastore_psycopg2cffi-py27-psycopg2cffi0207: commands succeeded
  postgres-datastore_psycopg2cffi-py36-psycopg2cffi0207: commands succeeded
  postgres-datastore_psycopg2cffi-pypy-psycopg2cffi0207: commands succeeded
  postgres-datastore_psycopg2cffi-py37-psycopg2cffi0208: commands succeeded
  postgres-datastore_psycopg2cffi-py39-psycopg2cffi0208: commands succeeded
cleanup /home/runner/work/newrelic-python-agent/newrelic-python-agent/.tox/.tmp/package/1/newrelic-0.0.zip
Error: Process completed with exit code 1.



### Code of Conduct

- [X] I agree to follow the [PSF Code of Conduct](https://www.python.org/psf/conduct/).
uranusjr commented 2 years ago

This looks like a PyPI bug.

Skipping page https://pypi.org/simple/pytest/ because the GET request got Content-Type: Unknown

But that is not a valid Content-Type value. You should probably report this problem to https://github.com/pypa/warehouse.

For pip, maybe it’s acceptable to always fall back to the HTML parser if the Content-Type is invalid? PEP 503 does not specify what Content-Type the server should use…? @dstufft

joerick commented 2 years ago

I don't think this is a warehouse issue. We see this in cibuildwheel quite a bit too - I've been tracking it at https://github.com/pypa/cibuildwheel/issues/1254 , with examples of failures on 22.2.2 and 21.3.1.

I have a minimal recreation of the crash here - https://github.com/joerick/pip-concurrency-debug . Or see this actions run - ignore the cleanup errors after the crash.

My theory is that this is related to concurrent access to pip's cache (that was also an issue a few years back, in https://github.com/pypa/pip/issues/5345 - which gave me the idea). But I can only get the crash to reliably recreate when running different versions of pip at the same time. That is something that our integration tests on cibuildwheel do, because we run tests in parallel, and run different versions of Pip because we still support Python 3.6. I note that the newrelic crash above also is running tests in parallel via tox, and is running different versions of pip for the same reason- old versions of python.

In terms of specific versions, here are results from my tests-

pip versions running simultaneously result
22.2.2, 21.3.1, 20.3.4 💥
20.3.4 ✅
22.2.2 ✅
21.3.1 ✅
21.3.1, 20.3.4 ✅
22.2.2, 21.3.1, 20.3.4 💥
22.2.2, 20.3.4 💥
22.2.2, 21.3.1 💥
22.2.2, 22.2.1 ✅
22.2, 22.1.2 💥

So the issue seems to be using pip<22.2 and pip>=22.2 at the same time. Note that I have to spin up 10 concurrent threads constantly installing/uninstalling to hit this semi-reliably, so it's not easy to recreate!

Giving each pip version a separate PIP_CACHE_DIR appears to fix the issue, which seems to implicate the shared cache as the source of the problems. @hmstepanek I'd be very curious if you newrelic folks find that separate cache dirs alleviates the issue for you. The errors on cibuildwheel's CI are too sporadic to test any workaround there, and the runs take far too long.

hmstepanek commented 2 years ago

@joerick Thank you for the insight! What you said makes total sense and would explain why not a lot of people seem to have this issue. I was beginning to wonder if it was just us. 🙈 I will give that workaround a try and see if it fixes the issue!

hmstepanek commented 1 year ago

We were able to resolve this by disabling the cache when running pip on python 2.7. See Fix Pip Concurrency Issues for details. So we can confirm it does appear to be an issue with pip caches across pip versions.