jsirois
commented
1 year ago As I pointed out to @tgolsson offline, a multi-platform lock example (Pants does not support these) for this case looks like so:
$ pex3 lock create \
--index https://download.pytorch.org/whl/cpu/ \
--platform macosx-10.9-x86_64-cp-39-cp39 \
--platform linux-x86_64-cp-39-cp39 \
"torch==1.11.0+cpu ; platform_system != 'Darwin'" \
"torch===1.11.0 ; platform_system == 'Darwin'" \
--pip-version latest \
--resolver-version pip-2020-resolver \
--indent 2 \
-o lock.mp.jsonIn real life you'd almost certainly want to use a pair of --complete-platform - this was just for quick demo. NB: I use === to semi-ameliorate the local version abuse footguns of torch here.
The result is a lock file containing 2 locked resolves:
{
"allow_builds": true,
"allow_prereleases": false,
"allow_wheels": true,
"build_isolation": true,
"constraints": [],
"locked_resolves": [
{
"locked_requirements": [
{
"artifacts": [
{
"algorithm": "sha256",
"hash": "50fd9bf85c578c871c28f1cb0ace9dfc6024401c7f399b174fb0f370899f4454",
"url": "https://download.pytorch.org/whl/cpu/torch-1.11.0-cp39-none-macosx_10_9_x86_64.whl"
}
],
"project_name": "torch",
"requires_dists": [
"typing-extensions"
],
"requires_python": ">=3.7.0",
"version": "1.11.0"
},
{
"artifacts": [
{
"algorithm": "sha256",
"hash": "fb33085c39dd998ac16d1431ebc293a8b3eedd00fd4a32de0ff79002c19511b4",
"url": "https://files.pythonhosted.org/packages/31/25/5abcd82372d3d4a3932e1fa8c3dbf9efac10cc7c0d16e78467460571b404/typing_extensions-4.5.0-py3-none-any.whl"
}
],
"project_name": "typing-extensions",
"requires_dists": [],
"requires_python": ">=3.7",
"version": "4.5.0"
}
],
"platform_tag": [
"cp39",
"cp39",
"macosx_10_9_x86_64"
]
},
{
"locked_requirements": [
{
"artifacts": [
{
"algorithm": "sha256",
"hash": "544c13ef120531ec2f28a3c858c06e600d514a6dfe09b4dd6fd0262088dd2fa3",
"url": "https://download.pytorch.org/whl/cpu/torch-1.11.0%2Bcpu-cp39-cp39-linux_x86_64.whl"
}
],
"project_name": "torch",
"requires_dists": [
"typing-extensions"
],
"requires_python": ">=3.7.0",
"version": "1.11.0+cpu"
},
{
"artifacts": [
{
"algorithm": "sha256",
"hash": "fb33085c39dd998ac16d1431ebc293a8b3eedd00fd4a32de0ff79002c19511b4",
"url": "https://files.pythonhosted.org/packages/31/25/5abcd82372d3d4a3932e1fa8c3dbf9efac10cc7c0d16e78467460571b404/typing_extensions-4.5.0-py3-none-any.whl"
}
],
"project_name": "typing-extensions",
"requires_dists": [],
"requires_python": ">=3.7",
"version": "4.5.0"
}
],
"platform_tag": [
"cp39",
"cp39",
"linux_x86_64"
]
}
],
"path_mappings": {},
"pex_version": "2.1.136",
"pip_version": "23.1.2",
"prefer_older_binary": false,
"requirements": [
"torch==1.11.0+cpu; platform_system != \"Darwin\"",
"torch===1.11.0; platform_system == \"Darwin\""
],
"requires_python": [],
"resolver_version": "pip-2020-resolver",
"style": "strict",
"target_systems": [],
"transitive": true,
"use_pep517": null
}Pex has the smarts to pick the best fitting lock - if any - at use time.
tgolsson
This thread is intended to focus on actionable solutions for #18293, potentially multi-platform Pex locks, maybe with other components as well.
Is your feature request related to a problem? Please describe.
The big neural network libraries Tensorflow and Pytorch have significant hurdles in usage based on how they're versioned. In general, the basic wheels published to PyPi aren't usable by default, and any important usage requires custom indexes and compute-API-tagged builds. In the case of
tensorflowthese are differently named packages, and in the case oftorchthey are differentiated by local versions. These specific packages also differ in what platforms they support, and the local versions may not exist for all platforms.Currently, the universal lock provided by Pex specifies a maximum set of
{Linux, Mac}but makes no guarantee either platform will be supported. This compounds the complexity of version specification, as local versions shall be preferred. In practice, one has to jump through hoops to ensure a basic resolve works. For example:This seems like we'd end up with one lock that works on both Mac (CPU) and Linux (CPU + random CUDA), and one for Linux with only CPU support. However; due to the PEP440 requirements both locks end up picking +CPU, and having no install candidates for Mac. The proper requirement is to specify it like this:
This means that the more different compute variants you need the more specific the generic constraints have to be. If we allowed Pex to separate the versions between Mac and Linux (and Windows?) we would have an easier time ensuring something reasonable gets picked. However; even so it'd be required to exclude all available versions in each resolve as we'd not want a massive
+cudaresult in the generic for Linux.Describe the solution you'd like
The thread on Slack discusses multi-platform locks as a potential solution, but I think that still runs afoul of some footguns with the local-version specifiers as one still needs to exclude them. It might therefore be necessary to combine them with other solutions such as per-resolve extra-indexes.
Additional context
https://pantsbuild.slack.com/archives/C046T6T9U/p1683187396031649