Wheels have different metadata on different platforms

[konstin]

Hi! Some resolvers in Python such as poetry and uv try to create lockfiles from the user's requirements that work an any platform. For example, you could create a universal lockfile on linux and use it to install the project on windows.

For this, both poetry and uv read the METADATA file of a single wheel on the index (in this case, pypi) and assume its metadata applies to all other platforms, too. For tensorflow, there is currently different metadata for windows and for linux/mac. For windows, the requires-dist excluding the cuda packages is:

Requires-Dist: tensorflow-macos ==2.15.1 ; platform_system == "Darwin" and platform_machine == "arm64"
Requires-Dist: tensorflow-cpu-aws ==2.15.1 ; platform_system == "Linux" and (platform_machine == "arm64" or platform_machine == "aarch64")
Requires-Dist: tensorflow-intel ==2.15.1 ; platform_system == "Windows"

While for linux and mac it is:

Requires-Dist: absl-py (>=1.0.0)
Requires-Dist: astunparse (>=1.6.0)
Requires-Dist: flatbuffers (>=23.5.26)
Requires-Dist: gast (!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1)
Requires-Dist: google-pasta (>=0.1.1)
Requires-Dist: h5py (>=2.9.0)
Requires-Dist: libclang (>=13.0.0)
Requires-Dist: ml-dtypes (~=0.3.1)
Requires-Dist: numpy (<2.0.0,>=1.23.5)
Requires-Dist: opt-einsum (>=2.3.2)
Requires-Dist: packaging
Requires-Dist: protobuf (!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3)
Requires-Dist: setuptools
Requires-Dist: six (>=1.12.0)
Requires-Dist: termcolor (>=1.1.0)
Requires-Dist: typing-extensions (>=3.6.6)
Requires-Dist: wrapt (<1.15,>=1.11.0)
Requires-Dist: tensorflow-io-gcs-filesystem (>=0.23.1)
Requires-Dist: grpcio (<2.0,>=1.24.3)
Requires-Dist: tensorboard (<2.16,>=2.15)
Requires-Dist: tensorflow-estimator (<2.16,>=2.15.0)
Requires-Dist: keras (<2.16,>=2.15.0)

That means depending on whether we read a windows wheel or a unix wheel, we get a different lockfile.

Would it be possible for tensorflow to write the same METADATA for all platforms and gate the platform specific entries with platform_system markers?

For uv, we've considered reading the METADATA files for all wheels, but this has major drawbacks: We have to make 17 network requests for pypi instead of 1 for each version we try, slowing resolution down. There is also no perfect mapping between environment markers (which usually tell us which dependencies to install on which platform) and wheel tags, so when METADATA can be different between wheels we'd also have to capture this in lockfiles.

I hope I explained good enough why identical METADATA files across all wheels of a version are important for us, I can add more details about how the resolver works if you have more questions.

This is similar to the problem discussed at https://github.com/tensorflow/tensorflow/issues/62346#issuecomment-1798633528.

[tilakrayal]

@konstin, The mentioned platforms tensorflow-cpu-aws, tensorflow-intel are maintained by the other groups. Also could you try in the latest tensorflow v2.17 and provide the update whether facing the similar issue. Thank you!

[konstin]

Hi @tilakrayal, I see that tensorflow is forwarding to other packages for windows which totally makes sense; I'm asking about keeping the requires-dist metadata consistent between wheels, i.e. adding the metadata for unix to the windows wheel with platform_system != "Windows" and that from the windows wheel to the unix wheels with platform_system == "Windows". For example, this would change

Requires-Dist: absl-py (>=1.0.0)
Requires-Dist: astunparse (>=1.6.0)
Requires-Dist: flatbuffers (>=23.5.26)

to

Requires-Dist: absl-py (>=1.0.0); platform_system != "Windows"
Requires-Dist: astunparse (>=1.6.0); platform_system != "Windows"
Requires-Dist: flatbuffers (>=23.5.26); platform_system != "Windows"

The wheels would still work the same when you installed them, but a universal resolver gets reliable metadata for all platforms at once.

Sorry for picking 2.15.1 as example, i've looked at the 2.17.0 wheels and i'm facing the same issue: https://inspector.pypi.io/project/tensorflow/2.17.0/packages/8a/8e/0ad1eff787bf13f8dca87472414fbdfb73ea53f5a1a1c20489cfccfb7717/tensorflow-2.17.0-cp310-cp310-win_amd64.whl/tensorflow-2.17.0.dist-info/METADATA is different from https://inspector.pypi.io/project/tensorflow/2.17.0/packages/1f/a1/7d2042050159619a190db874913a2bc70645f8ac677d442f9aab4d29153e/tensorflow-2.17.0-cp310-cp310-macosx_12_0_arm64.whl/tensorflow-2.17.0.dist-info/METADATA.

[tilakrayal]

@belitskiy