Fix Python metadata for Linux Python 3.6 wheel

[cschwan]

pip install pineappl with Python 3.6 fails. Adding the --verbose switch three times gives some useful information:

  Link requires a different Python (3.6.9 not in: '>=3.7'): https://files.pythonhosted.org/packages/c3/8d/789678e46590233b0052045358056721693155f1715c85a9d577e9af155e/pineappl-0.7.3-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=2537b7430b3928ec2dc9c8f8c27efc5e59de6d78bc13528d3c30883e02877e03 (from https://pypi.org/simple/pineappl/) (requires-python:>=3.7)

and indeed, if one goes to https://pypi.org/simple/pineappl/ and looks at the source code of the link pineappl-0.7.3-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl it indeed requires at least Python 3.7:

<a
   href="https://files.pythonhosted.org/packages/c3/8d/789678e46590233b0052045358056721693155f1715c85a9d577e9af155e/pineappl-0.7.3-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=2537b7430b3928ec2dc9c8f8c27efc5e59de6d78bc13528d3c30883e02877e03"
   data-requires-python=">=3.7"
   data-dist-info-metadata="sha256=b278bdc3bd3712be9b860415b838f2fb5e12c4e16e7ed98a0354842250fbbe2a"
   data-core-metadata="sha256=b278bdc3bd3712be9b860415b838f2fb5e12c4e16e7ed98a0354842250fbbe2a">
     pineappl-0.7.3-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
</a>

Now the question becomes: who generates this data? It's described in PEP 503, but I suppose it should be generated from the wheel's metadata which is correct.

For the time being we can circumvent the problem with:

pip install --ignore-requires-python pineappl

[alecandido]

Ok, so the dependencies are resolved downloading the wheel and opening, but the Python requirement may be directly exposed. And in the case of PyPI it is, taken from a random source.

Since no wheel is better than any other, I could only think of two options:

1. PyPI is parsing the first uploaded wheel
2. PyPI is parsing the sdist

But what if the sdist is missing? That makes me think that 1. is more likely than 2. ...

[cschwan]

What I don't understand is that the 'simple' list should be in 1 to 1 correspondence of the metadata in the wheels. But then again https://pypi.org/project/pineappl/ also shows 'Requires: Python >=3.7' in the Meta column.

[alecandido]

https://github.com/pypi/warehouse/blob/874c3b2f62e7db5f69aaa257514cad28bd0b9a78/warehouse/templates/api/simple/detail.html#L23

Used in two places: https://github.com/pypi/warehouse/blob/874c3b2f62e7db5f69aaa257514cad28bd0b9a78/warehouse/api/simple.py#L57 https://github.com/pypi/warehouse/blob/874c3b2f62e7db5f69aaa257514cad28bd0b9a78/warehouse/packaging/utils.py#L99 but I will follow the first, since it seems the one serving the simple API.

Actually, they are both getting the context from the same utils module, and only _simple_detail() is specifying context["projects"]["requires-python"]: https://github.com/pypi/warehouse/blob/874c3b2f62e7db5f69aaa257514cad28bd0b9a78/warehouse/packaging/utils.py#L67-L71 and it is file.release.requires_python, where file is one element in the query above: https://github.com/pypi/warehouse/blob/874c3b2f62e7db5f69aaa257514cad28bd0b9a78/warehouse/packaging/utils.py#L44-L51

Now the problem is moved to find out how PyPI constructs that DB...

[alecandido]

However, the problem is already in the DB structure: there is only one field requires-python per release https://github.com/pypi/warehouse/blob/874c3b2f62e7db5f69aaa257514cad28bd0b9a78/warehouse/packaging/models.py#L503 but, as we know very well, a release may have multiple files: https://github.com/pypi/warehouse/blob/874c3b2f62e7db5f69aaa257514cad28bd0b9a78/warehouse/packaging/models.py#L539

EDIT: each file may know its own requires-python: https://github.com/pypi/warehouse/blob/874c3b2f62e7db5f69aaa257514cad28bd0b9a78/warehouse/packaging/models.py#L705

Then it could be a problem in the join: https://github.com/pypi/warehouse/blob/874c3b2f62e7db5f69aaa257514cad28bd0b9a78/warehouse/packaging/utils.py#L47 I should check if by default is a right join, and Release.requires_python is overwriting File.requires_python.

[cschwan]

It looks like this describes our problem: https://github.com/pypi/warehouse/issues/8090.

[alecandido]

I think that for Requires-Python, we display that in the Web UI, so it makes sense to want that to be consistent, but more importantly, the intent of that key is that you can make a new release that doesn't support some version(s) of Python without a new release breaking existing users. For that, I think it makes sense to keep it consistent, since every file should be the same version of the code.

https://github.com/pypi/warehouse/issues/8090#issuecomment-1152743364

[alecandido]

File: Requires-Python (Denormalized from Release via PostgreSQL)

https://github.com/pypi/warehouse/issues/8090#issuecomment-1152743364

They are not even reading all the files, it was really the first one uploaded, even for the individual files metadata...

[cschwan]

OK, then let's try to globally change to requires-python = ">=3.6".

[cschwan]

It worked!

$ ~/.local/bin/pip install --pre pineappl
Defaulting to user installation because normal site-packages is not writeable
Collecting pineappl
  Downloading pineappl-0.7.4_rc.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (866 kB)
     |████████████████████████████████| 866 kB 13.4 MB/s
Requirement already satisfied: numpy<2.0.0,>=1.16.0 in /opt/bwhpc/common/conda/envs/python-3.6.9/lib/python3.6/site-packages (from pineappl) (1.17.0)
Installing collected packages: pineappl
Successfully installed pineappl-0.7.4rc1

[cschwan]

I'm closing this; commit 3c83f8257c2740ebcbd47d0164378d61baad748d adds a comment documenting this quirk.