search

pex-tool / pex

A tool for generating .pex (Python EXecutable) files, lock files and venvs.
https://docs.pex-tool.org/
Apache License 2.0
2.61k stars 261 forks source link

When a pex is built and a platform-specific wheel is required, the interpreter constraints are not narrowed to the platform in question. #676

Closed jsirois closed 3 months ago

jsirois commented 5 years ago

If you build a pex as such:

./pex27 thing -o thing.pex

You may end up creating a pex that contains platform-specific wheels if thing itself or its transitive dependencies are platform-specific. In a case like this, where a ~random minimum python2.7 interpreter on the PATH is picked, that interpreter will either be ucs2 (cp27m) or ucs4 (cp27mu) and likewise the platform specific wheel(s) embedded in the pex. As such, the pex can then only execute properly at runtime with a python 2.7 interpreter that matches the built platform-specific wheels. PEX should probably detect this situation at build time and inject interpreter constraints that narrow the field to python 2.7 interpreters of correct ucs level.

At the time of writing I'm not sure ucs level can even be currently picked out via interpreter constraints, so some work may need to be done there, or else the runtime picking of interpreter will simply need to take this into account and fail with a meaningful error message if a matching ucs level interpreter cannot be found.

Eric-Arellano commented 5 years ago

For the record, this was discovered while working on https://github.com/pantsbuild/pants/pull/7235 with the CI run https://travis-ci.org/pantsbuild/pants/jobs/497208431#L891 (also see line 1931 in this run).

Eric-Arellano commented 5 years ago

I'm not sure ucs level can even be currently picked out via interpreter constraints, so some work may need to be done there.

It's possible!

>>> import sys
>>> print(sys.maxunicode)
1114111 # UCS4

# or....

65535 # UCS2

See https://stackoverflow.com/a/1446456.

jsirois commented 5 years ago

By interpreter constraints I mean --interpreter-constraints; eg: CPython>=3.6. But ~ack, we'll need to use something like the code snippet of yours at runtime in combination with some new metadata instead.

jsirois commented 3 months ago

Interpreter constraint narrowing has been obviated by testing interpreters on boot as implemented in #1770 fixing this problem as re-stated in #1020.