pypa / bandersnatch

A PyPI mirror client according to PEP 381 http://www.python.org/dev/peps/pep-0381/
Academic Free License v3.0
453 stars 141 forks source link

Add platform tags specified in PEP 599/600 to exclude_platform #830

Open rigelifland opened 3 years ago

rigelifland commented 3 years ago

Currently, if you use the exclude_platform filter to remove linux packages, newer package tags still make it through. (For example, tornado-6.1-cp35-cp35m-manylinux2014_aarch64.whl)

The exclude_platform filter currently doesn't exclude packages with newer linux platform tags as defined in PEP 599 and 600. Can these be added?

rigelifland commented 3 years ago

PEP 599 should be straight forward to implement:

_bandersnatch_filter_plugins/filenamename.py

    _linuxPlatformTypes = [
        "linux-i686",  # PEP 425
        "linux-x86_64",  # PEP 425
        "linux_armv7l",  # https://github.com/pypa/warehouse/pull/2010
        "linux_armv6l",  # https://github.com/pypa/warehouse/pull/2012
        "manylinux1_i686",  # PEP 513
        "manylinux1_x86_64",  # PEP 513
        "manylinux2010_i686",  # PEP 571
        "manylinux2010_x86_64",  # PEP 571
        "manylinux2014_x86_64",  # PEP 599
        "manylinux2014_i686",  # PEP 599
        "manylinux2014_aarch64",  # PEP 599
        "manylinux2014_armv7l",  # PEP 599
        "manylinux2014_ppc64",  # PEP 599
        "manylinux2014_ppc64le",  # PEP 599
        "manylinux2014_s390x"  # PEP 599
    ]
cooperlees commented 3 years ago

Sure PR will be accepted for sure.

Long term tho, is there a library or should we look at adding this to the packaging library (https://github.com/pypa/packaging) if it's not already there as I am sure multiple libraries need this information.

rigelifland commented 3 years ago

As far as I can tell, the packaging library is aware of PEP 599/600 but it doesn't have the ability to generate a list of tags which could be used as a replacement for _linuxPlatformTypes.

Some of the this problem stems from platforms in bandersnatch being simplified to just ['windows', 'linux', macos', 'freebsd'], where platforms in packaging includes architecture information (e.g.'win_amd64'). PEP 600 will exacerbate this with many new linux platform types - does it make sense to change from the current filter using in:

if i in fn:

to something more flexible like a regex match against the platform as parsed out by packaging:

if re.match(i, fn):
cooperlees commented 3 years ago

Sure, happy to improve this generic matching as I think it's nice to say "ignore all windows" if you're a Linux only shop etc.

Thanks for this work!