serving index granularity

[dhellmann]

For which variables do we need “flavors” of index servers?

• Python versions
  • Covered by fields in filename
• Fedora versions, RHEL versions, etc.
  • ABI not encoded in filename, so would need different indexes
• Use cases (model tuning vs. model authoring)
• GPU versions, hardware architectures
  • No way to manage this with markers today, so likely need separate indexes

Can we "stack" indexes or otherwise combine them to avoid republishing the same content multiple times?

How do we track for any given package and build where it needs to be published?

[tiran]

The binary wheel file name is documented in packaging specs.

• File name spec: {distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl

• version spec can contain local version <public version identifier>[+<local version label>]

• distribution is the package name. We should never change it

• version can be extended to some degree. We can append .post versions and local version labels. The additional information will show up in importlib metadata and pip list.

• build tag could be used to indicate respins, although post releases might be better suited

• python tag for binary wheels will be current Python version (e.g. cp312 for CPython 3.12)

• abi tag is either the same as python tag or abi3 for a stable ABI / limited API build. IIRC Python 3.13 will also signal free threaded (no gil) builds here

• platform tag contains information similar to target triplet like linux-x86_64 or some manylinux / manymusl standard. Tags come from packaging.tags.sys_tags. I have an idea to define and enable custom dynamic tags like linux-x86_64-v3-rocm6.0. If my proposal works, then we don't need index stacking and can serve all our wheels from a single index.

A wheel for Fedora 39, X86_64 v3 micro arch, AMD ROCm 6.0 may look like this example-2.0.0.post100+fc39.v3.rocm6.0-cp12-cp12-linux-x86_64-v3-rocm6.0.whl. fc39, v3, and rocm6.0 are duplicated, so that pip list shows the package variant information, too.