Closed jperkelens closed 3 years ago
As requested: @stuhood
Thanks! I repro the issue, with or without setuptools
explicitly installed alongside pex
. Investigating.
According to https://github.com/pypa/setuptools/issues/1167, this is likely a collision between multiple setuptools
installs, one of which is vendored inside pex, and the other of which comes from your sitepackages due to --system-site-packages
.
So I suspect that this might be bug in the vendoring that pex
does that is allowing an external copy of setuptools
to collide with the vendored copy.
Two workarounds:
--system-site-packages
flag and rely on explicitly pip installing the things you need in the virtualenv$ pex pex -o pex.pex -m pex
$ ./pex.pex --interpreter-constraint='CPython>=3.7' boto3
Python 3.7.9 (default, Aug 18 2020, 06:24:24)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>>
This backtrace is in a Pex execution of Pip. There, Pex dogfoods itself to run a Pip PEX in isolation - with the exception of setuptools and wheel whose vendored versions are deliberately exposed so Pip can execute setup.py
s as needed:
https://github.com/pantsbuild/pex/blob/2783d304daf43dc1d59753c7e5fc2599b16acd01/pex/pip.py#L35-L53
So that's where to start spelunking.
I'll dig on this a bit.
Getting back to the conversation above, I wanted to correct the record on:
- run pex from a pex, which will allow pex to isolate itself rather than relying on virtualenv to do so:
Pex isolates itself whether run from a pex file or not. It does this by:
My answer above about Pip PEX dogfood concerns point 2.
FWIW, I tested that both 1) and 2) from my answer resolved the issue in the repro. I might have rushed things though.
Aha, ok. That's not making sense but good to know.
That said, I don't repro:
$ cat <<EOF | docker build -
FROM databricksruntime/python-virtualenv:latest
RUN apt-get update && apt-get install -y --allow-unauthenticated apt-transport-https ca-certificates curl gnupg2 software-properties-common curl software-properties-common build-essential
# Install & Bootstrap Python 3.7
RUN add-apt-repository ppa:deadsnakes/ppa
RUN apt update
RUN apt install --yes python3.7 python3.7-dev
RUN virtualenv -p python3.7 --system-site-packages /databricks/python3
RUN /databricks/python3/bin/pip3.7 install pex setuptools==45.2.0
ENV PATH="$PATH:/databricks/python3/bin"
RUN pex --interpreter-constraint='CPython>=3.7' boto3 -o boto3.pex -m boto3
EOF
Sending build context to Docker daemon 3.072kB
Step 1/9 : FROM databricksruntime/python-virtualenv:latest
---> 6d14d1c4da8d
Step 2/9 : RUN apt-get update && apt-get install -y --allow-unauthenticated apt-transport-https ca-certificates curl gnupg2 software-properties-common curl software-properties-common build-essential
---> Running in 085a999e997e
Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB]
...
Processing triggers for dbus (1.10.6-1ubuntu3.3) ...
Removing intermediate container 085a999e997e
---> b39aead9ebae
Step 3/9 : RUN add-apt-repository ppa:deadsnakes/ppa
---> Running in a985ceb3f4b6
This PPA contains more recent Python versions packaged for Ubuntu.
...
gpg: key 6A755776: public key "Launchpad PPA for deadsnakes" imported
gpg: Total number processed: 1
gpg: imported: 1 (RSA: 1)
OK
Removing intermediate container a985ceb3f4b6
---> 654f182d6ef2
Step 4/9 : RUN apt update
---> Running in 25d3cbe0c6d1
...
76 packages can be upgraded. Run 'apt list --upgradable' to see them.
Removing intermediate container 25d3cbe0c6d1
---> 31e7a456c83b
Step 5/9 : RUN apt install --yes python3.7 python3.7-dev
---> Running in e7601428f9c3
...
Setting up python3.7-dev (3.7.9-1+xenial1) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Removing intermediate container e7601428f9c3
---> 2a03e6c57939
Step 6/9 : RUN virtualenv -p python3.7 --system-site-packages /databricks/python3
---> Running in 8bfcdf409659
Using base prefix '/usr'
New python executable in /databricks/python3/bin/python3.7
...
Running virtualenv with interpreter /usr/bin/python3.7
Removing intermediate container 8bfcdf409659
---> 2e2296a1ba42
Step 7/9 : RUN /databricks/python3/bin/pip3.7 install pex setuptools==45.2.0
---> Running in bee547a895cb
Collecting pex
Downloading pex-2.1.16-py2.py3-none-any.whl (2.3 MB)
...
Successfully installed pex-2.1.16 setuptools-45.2.0
Removing intermediate container bee547a895cb
---> 320f252b829d
Step 8/9 : ENV PATH="/home/jsirois/.rvm/gems/ruby-2.6.0/bin:/home/jsirois/.rvm/gems/ruby-2.6.0@global/bin:/home/jsirois/.rvm/rubies/ruby-2.6.0/bin:/home/jsirois/.cargo/bin:/home/jsirois/bin:/home/jsirois/.local/bin:/home/jsirois/.nvm/versions/node/v8.11.3/bin:/home/jsirois/.poetry/bin:/opt/google-cloud-sdk/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/var/lib/snapd/snap/bin:/home/jsirois/.bash.d/bin:/home/jsirois/go/bin:/home/jsirois/.gem/ruby/2.5.0/bin:/home/jsirois/.rvm/bin:/home/jsirois/.bash.d/bin:/databricks/python3/bin"
---> Running in 8887cb7aa912
Removing intermediate container 8887cb7aa912
---> ea3611e65fa8
Step 9/9 : RUN pex --interpreter-constraint='CPython>=3.7' boto3 -o boto3.pex -m boto3
---> Running in e9ff82a8f6ff
Removing intermediate container e9ff82a8f6ff
---> 22d0b02808bd
Successfully built 22d0b02808bd
And the pex seems ok:
$ docker run --rm -it -ePEX_INTERPRETER=1 22d0b02808bd ./boto3.pex
Python 3.7.9 (default, Aug 18 2020, 06:24:24)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import boto3
>>>
now exiting InteractiveConsole...
@jperkelens or @stuhood does the original bug report still repro for you? Probably most importantly, does it repro after appropriate docker image prune
ing (fresh image builds)?
@jsirois : Yea, I repro with the exact commands from the description via:
docker image prune
docker run \
-v "$(pwd):/workdir" \
-w '/workdir' \
--rm \
-it \
"databricksruntime/python-virtualenv:latest" \
/bin/bash
# ... run each of the commands from the description
The docker-image-prune reports zero bytes pruned though, so I might be on a different copy of the databricksruntime/python-virtualenv:latest
tag...? My docker foo is weak.
EDIT: @jsirois : but also, FWIW, @jperkelens was able to resolve his issue by removing the --system-site-packages
flag, so he is no longer blocked on this.
If docker output says ---> Using cache
, the step is cached:
Step 2/9 : RUN apt-get update && apt-get install -y --allow-unauthenticated apt-transport-https ca-certificates curl gnupg2 software-properties-common curl software-properties-common build-essential
---> Using cache
---> b39aead9ebae
a different copy of the
databricksruntime/python-virtualenv:latest
tag
They have not changed the image for 13 months, so that's unlikely it, but the image incluides a RUN apt update
which can pick up new packages. So if that update step is cached, you'd pick up older packages than me. It could be that an upgraded package has the "fix".
but also, FWIW, @jperkelens was able to resolve his issue by removing the --system-site-packages flag, so he is no longer blocked on this.
Yeah - thanks. I'd still like to squash the issue if possible.
So if that update step is cached, you'd pick up older packages than me. It could be that an upgraded package has the "fix".
It wasn't... all of the steps ran. Probably (at least) because I was actually copy pasting them into a live shell one by one, rather than running them via a Dockerfile.
Hrm. Not being able to repro is ... maddening. If either of you have time could you run this on your image substituting your image id for 22d0b02808bd
?:
$ docker run --rm -it 22d0b02808bd apt list --installed | grep python
dh-python/now 2.20151103ubuntu1.1 all [installed,upgradable to: 2.20151103ubuntu1.2]
libpython3-stdlib/xenial,now 3.5.1-3 amd64 [installed,automatic]
libpython3.5-minimal/now 3.5.2-2ubuntu0~16.04.5 amd64 [installed,upgradable to: 3.5.2-2ubuntu0~16.04.11]
libpython3.5-stdlib/now 3.5.2-2ubuntu0~16.04.5 amd64 [installed,upgradable to: 3.5.2-2ubuntu0~16.04.11]
libpython3.7/xenial,now 3.7.9-1+xenial1 amd64 [installed,automatic]
libpython3.7-dev/xenial,now 3.7.9-1+xenial1 amd64 [installed,automatic]
libpython3.7-minimal/xenial,now 3.7.9-1+xenial1 amd64 [installed,automatic]
libpython3.7-stdlib/xenial,now 3.7.9-1+xenial1 amd64 [installed,automatic]
python-apt-common/xenial-updates,now 1.1.0~beta1ubuntu0.16.04.9 all [installed,automatic]
python-pip-whl/xenial-updates,now 8.1.1-2ubuntu0.4 all [installed,automatic]
python3/xenial,now 3.5.1-3 amd64 [installed]
python3-apt/xenial-updates,now 1.1.0~beta1ubuntu0.16.04.9 amd64 [installed,automatic]
python3-dbus/xenial,now 1.2.0-3 amd64 [installed,automatic]
python3-gi/xenial,now 3.20.0-0ubuntu1 amd64 [installed,automatic]
python3-minimal/xenial,now 3.5.1-3 amd64 [installed,automatic]
python3-pkg-resources/xenial,now 20.7.0-1 all [installed,automatic]
python3-pycurl/xenial,now 7.43.0-1ubuntu1 amd64 [installed,automatic]
python3-software-properties/xenial-updates,xenial-security,now 0.96.20.10 all [installed,automatic]
python3-virtualenv/xenial-updates,now 15.0.1+ds-3ubuntu1 all [installed,automatic]
python3.5/now 3.5.2-2ubuntu0~16.04.5 amd64 [installed,upgradable to: 3.5.2-2ubuntu0~16.04.11]
python3.5-minimal/now 3.5.2-2ubuntu0~16.04.5 amd64 [installed,upgradable to: 3.5.2-2ubuntu0~16.04.11]
python3.7/xenial,now 3.7.9-1+xenial1 amd64 [installed]
python3.7-dev/xenial,now 3.7.9-1+xenial1 amd64 [installed]
python3.7-distutils/xenial,now 3.7.9-1+xenial1 all [installed,automatic]
python3.7-lib2to3/xenial,now 3.7.9-1+xenial1 all [installed,automatic]
python3.7-minimal/xenial,now 3.7.9-1+xenial1 amd64 [installed,automatic]
$ docker image ls databricksruntime/python-virtualenv:latest
REPOSITORY TAG IMAGE ID CREATED SIZE
databricksruntime/python-virtualenv latest 6d14d1c4da8d 18 months ago 778MB
As mentioned above, I have not been using a Dockerfile
: rather, using docker run
followed by pasting each command, so the apt-list
bit is maybe not interesting:
$ docker run --rm -it 6d14d1c4da8d apt list --installed | grep python
dh-python/now 2.20151103ubuntu1.1 all [installed,local]
libpython3-stdlib/now 3.5.1-3 amd64 [installed,local]
libpython3.5-minimal/now 3.5.2-2ubuntu0~16.04.5 amd64 [installed,local]
libpython3.5-stdlib/now 3.5.2-2ubuntu0~16.04.5 amd64 [installed,local]
python-pip-whl/now 8.1.1-2ubuntu0.4 all [installed,local]
python3/now 3.5.1-3 amd64 [installed,local]
python3-minimal/now 3.5.1-3 amd64 [installed,local]
python3-pkg-resources/now 20.7.0-1 all [installed,local]
python3-virtualenv/now 15.0.1+ds-3ubuntu1 all [installed,local]
python3.5/now 3.5.2-2ubuntu0~16.04.5 amd64 [installed,local]
python3.5-minimal/now 3.5.2-2ubuntu0~16.04.5 amd64 [installed,local]
Thanks - the means by which the command are run in the image - by hand or by Dockefile RUN, are immaterial, so the listing is useful. But now I see you ran this against the base image and not the commands you subsequently ran - gotcha. Yeah, not so useful. Thanks - I'll try to debug this by other means.
OK, so focusing on --system-site-packages
(thanks @stuhood), virtualenv implements this via a custom site.py
. That site.py
calculates the underlying system site packages when the venv interpreter starts up and adds all those paths to sys.path
via direct sys.path
manipulation. As such, there is no way for the Pex runtime to detect these sys.path
s should be scrubbed without adding a hunk of code that knows an awful lot about virtualenv's implementation details. In short, this means Pex cannot trust sys.path
. It currently does when running Pip PEXed internally:
https://github.com/pantsbuild/pex/blob/2783d304daf43dc1d59753c7e5fc2599b16acd01/pex/pip.py#L35-L54
I can instead construct PYTHONPATH from the paths of the vendored setuptools and wheel directly, the main difference being this will pre-pend those paths to the sys.path of spawned Pip subprocesses. Right now they come in the order they are in sys.path which may put --system-site-packages
before the pexed up setuptools and wheel packages.
Noting the abive fix does not solve the general hermeticity problem here, just the specific problem for the Pip PEX used at PEX buildtime.
For example:
root@4185a16f2609:/# pex -v setuptools
pex: Building pex :: Resolving distributions (['setuptools']) :: Resolving for:
...
pex: Activating PEX virtual environment from /tmp/tmp4xggexvl: 28.6ms
pex: No entry point specified, dropping into interpreter
pex: Bootstrap complete, performing final sys.path modifications...
pex: PYTHONPATH contains:
pex: /tmp/tmp4xggexvl
pex: * /databricks/python3/lib/python37.zip
pex: /databricks/python3/lib/python3.7
pex: /databricks/python3/lib/python3.7/lib-dynload
pex: /usr/lib/python3.7
pex: /usr/local/lib/python3.7/dist-packages
pex: /usr/lib/python3/dist-packages
pex: /tmp/tmp4xggexvl/.deps/setuptools-50.3.0-py3-none-any.whl
pex: /tmp/tmp4xggexvl/.bootstrap
pex: * - paths that do not exist or will be imported via zipimport
Python 3.7.9 (default, Aug 18 2020, 06:24:24)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
Here the --system-site-packages
added by virtualenv to sys.path
directly in site.py
and unscrubbed by Pex are:
pex: /usr/local/lib/python3.7/dist-packages
pex: /usr/lib/python3/dist-packages
Besides not being scrubbed, these come before /tmp/tmp4xggexvl/.deps/setuptools-50.3.0-py3-none-any.whl
in the sys.path.In this particular builtime case, we're burned by pkg_resources
(Note pkg_resources
should come from setuptools
but there is no setuptools
package in this listing! - odd but not germain):
root@4185a16f2609:/# ls -1 /usr/lib/python3/dist-packages
__pycache__
_dbus_bindings.cpython-35m-x86_64-linux-gnu.so
_dbus_glib_bindings.cpython-35m-x86_64-linux-gnu.so
apt
apt_inst.cpython-35m-x86_64-linux-gnu.so
apt_pkg.cpython-35m-x86_64-linux-gnu.so
aptsources
curl
dbus
debconf.py
gi
lsb_release.py
pkg_resources
pycurl-7.43.0.egg-info
pycurl.cpython-35m-x86_64-linux-gnu.so
pygobject-3.20.0.egg-info
pygtkcompat
python_apt-1.1.0.b1_ubuntu0.16.04.9.egg-info
softwareproperties
unattended_upgrades-0.1.egg-info
virtualenv-15.0.1.egg-info
virtualenv.py
It looks like one way to solve this is to invert how we calculate which sys.path
entries to scrub. Instead of calculating site-libraries, user-site-libraries and extras to scrub:
https://github.com/pantsbuild/pex/blob/0a29161e3b24a1b7a884ab3acd5f2f411bac7336/pex/pex.py#L309-L315
We could calculate just the core libraries and scrub everything else via:
python -sSE -c 'import sys; print("\n".join(sys.path))'
That solves the case above and seems as robust as you can possibly be since the answer is delegated to the python interpreter in question.
Checking back in here. Although I still cannot repro the failure you get using the commands above I can repro the leak by inspection of sys.path in a PEX built in the databaricks image. I've done some work over in #1128 that makes Pex both aware of virtual environments and equips it with the capability to bust out of a virtual environment. I'll use that to fix this. The approach above using ~ python -sSE -c 'import sys; print("\n".join(sys.path))'
was not robust across all interpreter installations (those that include bits of the stdlib in ~site-packages :/).
Having issues creating pex files in an image based on
databricksruntime/python-virtualenv
.Dockerfile:
Command run in container:
pex --interpreter-constraint='CPython>=3.7' boto3 -o boto3.pex -m boto3
Portion of failure StackTrace: