Incompatible Python version + `--complete-platforms ...` + `--lock ...` + empty cache gives seemingly-spurious "requires different Python" errors

[huonw]

When trying to build a PEX with a requirement that restricts Python interpreter versions (e.g. >= 3.10), it seems one can get seemingly-spurious requires different Python errors if all of these conditions are true:

1. PEX is being executed with a Python version that isn't compatible with the requirement (e.g. 3.9)
2. --complete-platforms is being used to build for a different platform (which does satisfy the Python interpreter requirement)
3. PEX_ROOT has not had a PEX execution using compatible Python version manipulate that dependencies (this is a bit fuzzy, I'm not sure of the details)
4. --lock is being used to pin versions

(The first two are, I believe, the normal way to do a cross-build for a "foreign" platform, and so are just "table stakes". The last two are more interesting.)

In particular, with appropriate complete platform and lock files:

PEX_ROOT=$(mktemp -d) python3.9 ./pex pbipy==2.8.2 --complete-platform ./complete_platform_linux_arm64_py11.json -o bin1.pex --lock lock.json

Fails with:

There was 1 error downloading required artifacts:
1. pbipy 2.8.2 from https://files.pythonhosted.org/packages/61/74/62c07268b60093e3eb5a770d1cfcb072482095bd8cd754c502deaf832790/pbipy-2.8.2-py2.py3-none-any.whl
    ERROR: Package 'pbipy' requires a different Python: 3.9.17 not in '>=3.10'

https://gist.github.com/huonw/456ed93d53c992894cacb4efe71250fe has a full reproducer, including the full output of the failure above (with PEX_VERBOSE=1). This reproducer:

• uses https://pypi.org/project/pbipy/2.8.2/#files which provides a py2.py3-none-any wheel
• has a Linux arm64 lockfile (NB. if truly a "foreign" platform is required for 2, and you're on a Linux arm64 machine you may need to regenerate it or run within a docker image. I'm not sure)
• invokes PEX in 4 ways, first demonstrating the failure (as above), and then doing two ablations do demonstrate that conditions 3 and 4 seem to be required

The two ablations, which both succeed:

1. first build a PEX with a local Python 3.11 platform, and then, within the same PEX_ROOT, run the same command as above.

   PEX_ROOT=${root}/2 python3.11 ./pex pbipy==2.8.2 -o bin2.pex --lock lock.json
   PEX_ROOT=${root}/2 python3.9 ./pex pbipy==2.8.2 --complete-platform ./complete_platform_linux_arm64_py11.json -o bin3.pex --lock lock.json

2. run the same command (in a empty PEX_ROOT), without --lock lock.json:

   PEX_ROOT=${root}/3 python3.9 ./pex pbipy==2.8.2 --complete-platform ./complete_platform_linux_arm64_py11.json -o bin4.pex

I'm not 100% sure this is a bug, but I'm suspicious, based on: this is a platform-indepedent package, and it works with the cache pre-seeded or without a lockfile.

[jsirois]

Seems solidly buggy to me. Thanks for the repro case.

Pex already has Pip runtime patching technology it deploys to get around this sort of error in other code paths. That patching is almost certainly not being applied here in the downloader code.

FYI: I'll not be back to a keyboard to work on this or review fixes until 4/19.

[jsirois]

Ok, the runtime patching was happening, it was just happening against incomplete information. Fix coming today.