Vendored `pip` in `pex` not located inside created virtualenv on macOS

[stuhood]

On macOS, it's possible to encounter an error like the following:

  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'pip._internal.cli.main'

or

  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'pip'

Both of these are believed to stem from xcode upgrades shifting the symlinks/binaries of system python installs in ways that PEX does not currently detect. If you inspect the virtualenv in which the PEX is running, you can see that the site-packages of the virtualenv is specific to a different major version of Python than is actually executing, which causes the installed copy of pip not to be found.

To work around this issue you can wipe the "pex root" (if you're using https://github.com/pantsbuild/pants/, this will be located at ~/.cache/pants/named_caches/pex_root).

It may be possible to repro (and fix) this issue by using xcode-select to toggle between multiple xcode installs while PEX attempts to use the xcode-provided Python interpreter.

[jsirois]

As I understand it @stuhood you'll be taking up this issue at least to verify / repro the root cause since you run the OS and are in a better position to repro than I am. If the proposed root cause is correct, that would imply any venv (Pex aside) created using a macOS system interpreter that is updated via xcode upgrades will have this problem. Namely you'll need to blow away the venv. It might be useful to check that as you narrow in.

[stuhood]

Possibly related to #1422... will see if I can get a before/after repro.

[jsirois]

These should not be related. Prior to the just released Pex 2.1.46, the venv hashes did happen to include the full python interpreter version. The scheme is: $PEX_ROOT/venvs/<pex hash>/<runtime influences hash>/. The pex_hash is a hash of the json dump of PEX-INFO, and PEX-INFO prior to 2.1.46 had a build_properties object like so:

$ pex -oexample.pex
^jsirois@gill ~/dev/pantsbuild/jsirois-pex (issues/1422 *) $ pex-tools example.pex info | jq .build_properties
{
  "class": "CPython",
  "pex_version": "2.1.45",
  "platform": "manylinux_2_33_x86_64",
  "version": [
    3,
    9,
    6
  ]
}

[jsirois]

Assuming the root cause guess is correct - an xcode upgraded Python that's re-directed to through a shim binary ~like this: https://github.com/python/cpython/blob/v3.7.1/Mac/Tools/pythonw.c - we've had another report in Pants Slack where the only way to get Pants to see a newly upgraded python3 (3.7 -> 3.8), is to nuke ~/.cache/pants which forces a Pex re-scan of system interpreters since the ~/.cache/pants/named_caches/pex_root (and LMDB store) are gone.

[danshinkansama]

Muchachos, ¿Se pudo arreglar?

[Eric-Arellano]

Muchachos, ¿Se pudo arreglar?

Me parece que no todavía. ¿Usas Pants o Pex solo? El resto del equipo no habla español creo pero puedes unirte a nuestro Slack y yo intentaré ayudarte con gusto :) Usa este enlace nomás https://join.slack.com/t/pantsbuild/shared_invite/zt-d0uh0mok-RLvVosDiX6JDpvStH~bFBA