Investigate custom platform tags for ISA, ABI, and accelerator in wheel filename

[tiran]

The filename of wheels are standardized and contain the following information:

{distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl

Version is defined as <public version identifier>[+<local version label>] with public version [N!]N(.N)*[{a|b|rc}N][.postN][.devN] and optional local version [a-zA-Z0-9.]+. See #92 for a local version label discussion.

python tag and abi tag reflect the Python version and ABI version of a platlib wheel (wheel with platform-specific binary extensions). They are typically cp312-cp312 for a CPython 3.12 wheel or cp312-abi3 for a wheel with stable ABI that is compatible with Python >= 3.12, <4.0.

The platform tag contains information similar to target triplet, e.g. linux-x86-64 for Linux on AMD64 or manylinux2014_aarch64 for Linux on ARM64 on CentOS 6 or newer (manylinux2014 standard with glibc 2.17).

Tools like pip use packaging.tags.sys_tags() to generated an ordered list of supported tags. The tool dynamically creates tags based on current implementation (CPython, PyPy), version, sysconfig.get_platform(), and glibc/musllibc version. On my Fedora 39 with glibc 2.38, the platform tags are:

>>> packaging.tags
>>> list(packaging.tags.platform_tags())
['manylinux_2_38_x86_64', 'manylinux_2_37_x86_64', 'manylinux_2_36_x86_64', 'manylinux_2_35_x86_64', 'manylinux_2_34_x86_64', 'manylinux_2_33_x86_64', 'manylinux_2_32_x86_64', 'manylinux_2_31_x86_64', 'manylinux_2_30_x86_64', 'manylinux_2_29_x86_64', 'manylinux_2_28_x86_64', 'manylinux_2_27_x86_64', 'manylinux_2_26_x86_64', 'manylinux_2_25_x86_64', 'manylinux_2_24_x86_64', 'manylinux_2_23_x86_64', 'manylinux_2_22_x86_64', 'manylinux_2_21_x86_64', 'manylinux_2_20_x86_64', 'manylinux_2_19_x86_64', 'manylinux_2_18_x86_64', 'manylinux_2_17_x86_64', 'manylinux2014_x86_64', 'manylinux_2_16_x86_64', 'manylinux_2_15_x86_64', 'manylinux_2_14_x86_64', 'manylinux_2_13_x86_64', 'manylinux_2_12_x86_64', 'manylinux2010_x86_64', 'manylinux_2_11_x86_64', 'manylinux_2_10_x86_64', 'manylinux_2_9_x86_64', 'manylinux_2_8_x86_64', 'manylinux_2_7_x86_64', 'manylinux_2_6_x86_64', 'manylinux_2_5_x86_64', 'manylinux1_x86_64', 'linux_x86_64']

And sys_tags() returns > 1,000 tag variants:

>>> [str(t) for t in packaging.tags.sys_tags()]
['cp312-cp312-manylinux_2_38_x86_64', 'cp312-cp312-manylinux_2_37_x86_64', 'cp312-cp312-manylinux_2_36_x86_64', 'cp312-cp312-manylinux_2_35_x86_64',
...
'py34-none-any', 'py33-none-any', 'py32-none-any', 'py31-none-any', 'py30-none-any']

I have an idea how we can dynamically extend the platform tags with custom tags. At first this will require a bit of monkey patching with a meta import hook, dynamic loader, and a .pth file. Later we could ask upstream to include a API (e.g. entry point hook) to extend the tags.

• get current distro and version to define distro ABI (e.g. fc39, el9). I added platform.freedesktop_os_release several years ago
• platform and CPU extensions for machine and ISA version (e.g. x86_64_v3 for -march=x86-64-v3
• GPU hardware and support library version from a config file in $VIRTUAL_ENV, e.g. cuda12_1, rocm6_0

Example tag: linux_x86_64_v3_fc39_rocm6_0 Example wheel: example-1.2.3+fc39.rocm6.0-cp312-cp312-linux_x86_64_v3_fc39_rocm6_0.whl. The distro ABI and GPU extension is included twice, so it also shows up in pip list and packaging metadata.

[dhellmann]

There was some upstream discussion in https://discuss.python.org/t/what-to-do-about-gpus-and-the-built-distributions-that-support-them/7125/63 and https://github.com/conda-forge/conda-forge.github.io/issues/1059