How does pex resolve defaut `--pip-version`

[maxkoretskyi]

I have pex==2.13.0, when I run this with double dependency pex fails:

$ pex 'kywy==0.18.3' 'kywy>=0.18.1' -o d.pex
pip: ERROR: Double requirement given: kywy>=0.18.1 (already in kywy==0.18.3, name='kywy')

Pip doesn't fail, so when I specify --pip-version latest pex works all good:

 pex 'kywy==0.18.3' 'kywy>=0.18.1' -o d.pex --pip-version latest

If I do pip --version I get 24:

pip 24.0 from /home/user/.virtualenvs/app-builder/lib/python3.10/site-packages/pip (python 3.10)

so why doesn't pex it pick this version of pip by default?

[jsirois]

Pex is hermetic. It actively ignores the surrounding venv contents if run from a venv very much on purpose. This is perhaps more obvious if you don't run it from a venv but instead run the Pex PEX binary When you say pex --pip-version X the X must be from the list of versions Pex supports, and if its not the vendored version Pex ships with (A very old patched Pip 20.3.4), then Pex dog-foods itself to fetch the requested Pip (plus supporting setuptools and wheel) into a hermetic PEX under the PEX_ROOT (~/.pex/pip/<version>/pip.pex/... by default).

If you haven't seen this, here's the list of supported versions as of Pex 2.13.0:

:; pex --help | grep -A5 "\--pip-version {"
  --pip-version {latest,vendored,20.3.4-patched,22.2.2,22.3,22.3.1,23.0,23.0.1,23.1,23.1.1,23.1.2,23.2,23.3.1,23.3.2,24.0,24.1,24.1.1,24.1.2,24.2}
                        The version of Pip to use for resolving dependencies.
                        The `latest` version refers to the latest version in
                        this list (24.2) which is not necessarily the latest
                        Pip version released on PyPI. (default:
                        20.3.4-patched)

[jsirois]

So, to be more clear perhaps - unlike most Python tools, you can install Pex globally on your machine once. You can specify --python, --interpreter-constraint, etc to get it to look at any python on the system at any time using that one global install. Again, since Pex happens to be written in Python, this might not be obvious. But consider it written in another language. If it has to invoke Python code, it always does so in a hermetic subprocess. If you're familiar with uv, like that - just slower ;).

[jsirois]

And as to why the --pip-version defaults so old - Pex never breaks users - ever ^1. You can still be on Python 2.7 and purposefully using the Pip 20.3.4 legacy resolver and you can still upgrade to Pex latest and have everything still work. This is great for backwards compatibility, but it means the Pex defaults are very often not what you want. It pays off to frequently re-read release notes and / or pex --help as a result to see what new, better options may be available to improve resolution or PEX execution speed, etc.

[maxkoretskyi]

@jsirois thanks a lot for the quick reply!

I wanted to confirm that it's not me doing something wrong, it's actually a common approach to "customize" pex with those options if I understood you correctly here:

You can specify --python, --interpreter-constraint, etc to get it to look at any python on the system at any time using that one global install.

unlike most Python tools, you can install Pex globally on your machine once.

I actually was only installing it through pip, but it seems that if I build an isolate env, e.g. docker, I could download (can I?) and put a binary there instead of trying to put it into the global python site, which might be problematic (ubuntu doesn't like it)

then Pex dog-foods itself to fetch the requested Pip

are there maybe some public design docs there offer a glipse into the pex machinery?

[jsirois]

I wanted to confirm that it's not me doing something wrong, it's actually a common approach to "customize" pex with those options if I understood you correctly here

Yes. There are a ton of features hidden behind the 3 console scripts Pex ships with (pex, pex3 and pex-tools). Unfortunately, the only real documentation right now is the command line help. There are some very basic notes at https://docs.pex-tool.org and recipes as well, but the coverage is very poor.

are there maybe some public design docs there offer a glipse into the pex machinery?

Nope. Just the code and tinkering. A PEX is a zip, so unzip it or use zipinfo to list it, etc. You can use pex-tools ./my.pex ... to further interrogate the PEX file itself. If you're curious you can learn a lot by just clearing the PEX_ROOT and then running a pex ... command or running a PEX file and looking back in the PEX_ROOT to see the directory structure of what got populated (see : ~/.pex/{pip,installed_wheels,venvs,interpreters,...} etc.)

[jsirois]

I actually was only installing it through pip, but it seems that if I build an isolate env, e.g. docker, I could download (can I?) and put a binary there instead of trying to put it into the global python site, which might be problematic (ubuntu doesn't like it)

Yes, you could install it in a myriad ways:

1. Use the Pex PEX binary (requires the target system has an answer to /usr/bin/env python):

   curl -fL https://github.com/pex-tool/pex/releases/download/v2.13.0/pex > /some/path/pex
   chmod +x /some/path/pex
   export PATH=/some/path:$PATH

2. Install in a venv:

   python -mvenv pex.venv
   pex.venv/bin/pip install pex
   export PATH=pex.venv/bin:$PATH

3. Install it using pipx if you happen to use pipx:

   pipx install pex

4. Build a hermetic Pex native binary - no Python on target system is required!

   # 1st get `pex` via one of the methods above!
   pex pex -c pex --venv --scie eager -o pex

   EXCEPT!, this doesn't work yet: https://github.com/pex-tool/pex/pull/2484

[jsirois]

@maxkoretskyi please let me know if you have any further questions. If not, I'd like to close this as an answered question.

[maxkoretskyi]

@jsirois yes, absolutely, thanks a for your elaborate answers!

[maxkoretskyi]

@jsirois need your help, if I install pex as binary like this

# installing pex directly as binary requires the `/usr/bin/env python` call to be available
RUN curl -fL https://github.com/pex-tool/pex/releases/download/v2.13.0/pex -o /usr/local/bin/pex
RUN chmod +x /usr/local/bin/pex
RUN chown ${uid}:${gid} /usr/local/bin/pex

the following simply code my.py fails when run with python m.py:

import subprocess

pex = ['/usr/local/bin/pex']

try:
    subprocess.run(pex,check=True,capture_output=True,env={'PEX_VERBOSE': '3'})
except subprocess.CalledProcessError as exc:
    error = f'Building pex failed with error: \n {exc.stderr.decode("unicode_escape")}'
    print(error)

The error is:

/usr/bin/env: 'python': No such file or directory

which is weird, because /usr/bin/env python is available:

$ /usr/bin/env python --version
Python 3.11.9

and if I run /usr/local/bin/pex directly, not as a subprocess, all works OK

[jsirois]

@maxkoretskyi your problem has nothing to do with Pex here, it has to do with your use of the subprocess API. When you say subprocess.run(..., env={'PEX_VERBOSE': '3'}) you tell Python that {'PEX_VERBOSE': '3'} is the complete and total environment to run the subprocess in. In other words, you nuke PATH and HOME and everything else from the environment the subprocess sees. Without PATH, the /usr/bin/env executable can't find python. So you probably want: subprocess.run(..., env={**os.environ, 'PEX_VERBOSE': '3'})

[maxkoretskyi]

@jsirois silly me! appreciate your insights, all is good now 🙋🏻

as a side note, do you by any chance do occasional consulting related to python? If not, maybe you know somebody good who does?

[jsirois]

@maxkoretskyi I do, but I'd need to know more details. You could DM me at https://pex-tool.org/discord or email john.sirois@gmail.com. I also can provide recommendations if the work is not a good fit.

[maxkoretskyi]

Thanks! I'll send an email