Interference with system python packages via /usr/lib/python3/

[neumond]

description

Current deadsnakes fresh python packages interfere with system python via shared site packages. They load some packages from /usr/lib/python3/, which causes mysterious errors like this one https://gist.github.com/neumond/17c1b711127fb74f1e2994dc719785c9

These paths added by site.py module, they're Ubuntu-specific, you can check in origin Python sources:

https://github.com/python/cpython/blob/master/Lib/site.py#L336-L339

and shipped version from deadsnakes/ubuntu packages:

def getsitepackages(prefixes=None):
    ...
        if os.sep == '/':
            if 'VIRTUAL_ENV' in os.environ or sys.base_prefix != sys.prefix:
                sitepackages.append(os.path.join(prefix, "lib",
                                                 "python" + sys.version[:3],
                                                 "site-packages"))
            sitepackages.append(os.path.join(prefix, "local/lib",
                                             "python" + sys.version[:3],
                                             "dist-packages"))
            sitepackages.append(os.path.join(prefix, "lib",
                                             "python3",
                                             "dist-packages"))
            # this one is deprecated for Debian
            sitepackages.append(os.path.join(prefix, "lib",
                                             "python" + sys.version[:3],
                                             "dist-packages"))
        else:
            sitepackages.append(prefix)
            sitepackages.append(os.path.join(prefix, "lib", "site-packages"))

I've made script to fix Python installation, but it should be better to fix in packages.

# tested only under Ubuntu Xenial
# change to -i.bak if you want backup of old file
sudo sed -i '
/^[ \t][ \t]*sitepackages\.append(os\.path\.join(prefix,[ \t]*"lib",[ \t]*/ {
    N
    N
    s/sitepackages\.append(os\.path\.join(prefix,[ \t]*"lib",[ \t]*\n[ \t]*"python3",\n[ \t]*"dist-packages"))/# Disabled shared python3 directory by sed script/
}
' /usr/lib/python3.7/site.py
[ -f /usr/lib/python3.7/__pycache__/site.cpython-37.pyc ] && sudo python3.7 -m compileall /usr/lib/python3.7/site.py

NOTE: applying this fix requires you to install at least setuptools manually.

os information

lsb_release -a

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.5 LTS
Release:    16.04
Codename:   xenial

uname -a

Linux neumond-Latitude-7490 4.4.0-140-generic #166-Ubuntu SMP Wed Nov 14 20:09:47 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[asottile]

This is ~kind of working as designed.

From the deadsnakes homepage:

Python modules in the official Ubuntu repositories are packaged to work with the Python interpreters from the official repositories. Accordingly, they generally won't work with the Python interpreters from this PPA. As an exception, pure-Python modules for Python 3 will work, but any compiled extension modules won't.

(emphasis mine)

It looks like you're using sudo easy_install which shouldn't really ever be run -- you should use pip if anything, and probably use a virtualenv. Debugging this myself, the file it's choking on:

    *******************************************************************************
    build/bdist.linux-x86_64/egg/__pycache__/ptr.cpython-37.pyc
    *******************************************************************************

and that's when it is invoking setup_requires here (setup_requires downloads eggs, and generally just doesn't work well, python setup.py test should be considered deprecated and you may want to tell the pyrsistent people that they should switch to tox / virtualenv / anything other than setup.py test)

this doesn't actually have anything to do with interference, it would work fine with a newer setuptools version (this was a bug in setuptools):

root@0bea0a282bf7:/# python3.7 -m pip freeze --all | grep setuptools
You are using pip version 8.1.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
setuptools==20.7.0

root@0bea0a282bf7:/# python3.7 -m pip install setuptools --upgrade
Collecting setuptools
  Using cached https://files.pythonhosted.org/packages/37/06/754589caf971b0d2d48f151c2586f62902d93dc908e2fd9b9b9f6aa3c9dd/setuptools-40.6.3-py2.py3-none-any.whl
Installing collected packages: setuptools
  Found existing installation: setuptools 20.7.0
    Not uninstalling setuptools at /usr/lib/python3/dist-packages, outside environment /usr
Successfully installed setuptools-40.6.3
You are using pip version 8.1.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
root@0bea0a282bf7:/# python3.7 -m pip install pyrsistent
Collecting pyrsistent
  Using cached https://files.pythonhosted.org/packages/60/dd/64ff98c6a4eedbdec7f4308198a74a8c0efac9f13e198f473d42053b4140/pyrsistent-0.14.7.tar.gz
Collecting six (from pyrsistent)
  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Building wheels for collected packages: pyrsistent
  Running setup.py bdist_wheel for pyrsistent ... error
  Complete output from command /usr/bin/python3.7 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-m9w9jo12/pyrsistent/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d /tmp/tmp_rep15udpip-wheel- --python-tag cp37:
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help

  error: invalid command 'bdist_wheel'

  ----------------------------------------
  Failed building wheel for pyrsistent
  Running setup.py clean for pyrsistent
Failed to build pyrsistent
Installing collected packages: six, pyrsistent
  Running setup.py install for pyrsistent ... done
Successfully installed pyrsistent-0.14.7 six-1.12.0
You are using pip version 8.1.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

(ignore the soft-fail on wheel, I didn't bother installing wheel before trying this)

---

to confirm this isn't a deadsnakes issue, create a python3.7 virtualenv and install setuptools==20.7.0, you'll see the same issue:

root@0bea0a282bf7:/# . venv/bin/activate
(venv) root@0bea0a282bf7:/# pip install setuptools==20.7.0
Collecting setuptools==20.7.0
  Downloading https://files.pythonhosted.org/packages/73/73/57576f40b4d87a4f8d0fba656469d05559910337d852bc513ff34f47bdc8/setuptools-20.7.0-py2.py3-none-any.whl (508kB)
    100% |████████████████████████████████| 512kB 742kB/s 
Installing collected packages: setuptools
  Found existing installation: setuptools 40.6.3
    Uninstalling setuptools-40.6.3:
      Successfully uninstalled setuptools-40.6.3
Successfully installed setuptools-20.7.0
(venv) root@0bea0a282bf7:/# pip freeze --all
pip==18.1
pkg-resources==0.0.0
setuptools==20.7.0
wheel==0.32.3
(venv) root@0bea0a282bf7:/# pip install pyrsistent
Collecting pyrsistent
  Using cached https://files.pythonhosted.org/packages/60/dd/64ff98c6a4eedbdec7f4308198a74a8c0efac9f13e198f473d42053b4140/pyrsistent-0.14.7.tar.gz
    Complete output from command python setup.py egg_info:
    /usr/lib/python3.7/distutils/dist.py:274: UserWarning: Unknown distribution option: 'python_requires'
      warnings.warn(msg)
    warning: no previously-included files found matching '*.nix'
    warning: no previously-included files found matching 'appveyor.yml'
    warning: no previously-included files found matching '.travis.yaml'
    warning: no previously-included files found matching '.pre-commit-config.yaml'

    Installed /tmp/easy_install-vx8umdqu/pytest-runner-4.2/.eggs/setuptools_scm-3.1.0-py3.7.egg
    /usr/lib/python3.7/distutils/dist.py:274: UserWarning: Unknown distribution option: 'python_requires'
      warnings.warn(msg)
    zip_safe flag not set; analyzing archive contents...
    Traceback (most recent call last):
      File "/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 154, in save_modules
        yield saved
      File "/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
        yield
      File "/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 243, in run_setup
        DirectorySandbox(setup_dir).run(runner)
      File "/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 273, in run
        return func()
      File "/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 242, in runner
        _execfile(setup_script, ns)
      File "/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 46, in _execfile
        exec(code, globals, locals)
      File "/tmp/easy_install-vx8umdqu/pytest-runner-4.2/setup.py", line 76, in <module>
        'Programming Language :: Python :: 3.7',
      File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/lib/python3.7/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 209, in run
        os.path.join(archive_root, 'EGG-INFO'), self.zip_safe()
      File "/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 245, in zip_safe
        return analyze_egg(self.bdist_dir, self.stubs)
      File "/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 355, in analyze_egg
        safe = scan_module(egg_dir, base, name, stubs) and safe
      File "/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 392, in scan_module
        code = marshal.load(f)
    ValueError: bad marshal data (unknown type code)

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-qj748ds4/pyrsistent/setup.py", line 87, in <module>
        package_data={'pyrsistent': ['py.typed', '__init__.pyi', 'typing.pyi']},
      File "/usr/lib/python3.7/distutils/core.py", line 108, in setup
        _setup_distribution = dist = klass(attrs)
      File "/venv/lib/python3.7/site-packages/setuptools/dist.py", line 269, in __init__
        self.fetch_build_eggs(attrs['setup_requires'])
      File "/venv/lib/python3.7/site-packages/setuptools/dist.py", line 313, in fetch_build_eggs
        replace_conflicting=True,
      File "/venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 826, in resolve
        dist = best[req.key] = env.best_match(req, ws, installer)
      File "/venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1092, in best_match
        return self.obtain(req, installer)
      File "/venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1104, in obtain
        return installer(requirement)
      File "/venv/lib/python3.7/site-packages/setuptools/dist.py", line 380, in fetch_build_egg
        return cmd.easy_install(req)
      File "/venv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 640, in easy_install
        return self.install_item(spec, dist.location, tmpdir, deps)
      File "/venv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 670, in install_item
        dists = self.install_eggs(spec, download, tmpdir)
      File "/venv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 850, in install_eggs
        return self.build_and_install(setup_script, setup_base)
      File "/venv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 1078, in build_and_install
        self.run_setup(setup_script, setup_base, args)
      File "/venv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 1064, in run_setup
        run_setup(setup_script, args)
      File "/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 246, in run_setup
        raise
      File "/usr/lib/python3.7/contextlib.py", line 130, in __exit__
        self.gen.throw(type, value, traceback)
      File "/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
        yield
      File "/usr/lib/python3.7/contextlib.py", line 130, in __exit__
        self.gen.throw(type, value, traceback)
      File "/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 166, in save_modules
        saved_exc.resume()
      File "/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 141, in resume
        six.reraise(type, exc, self._tb)
      File "/venv/lib/python3.7/site-packages/pkg_resources/_vendor/six.py", line 685, in reraise
        raise value.with_traceback(tb)
      File "/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 154, in save_modules
        yield saved
      File "/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
        yield
      File "/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 243, in run_setup
        DirectorySandbox(setup_dir).run(runner)
      File "/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 273, in run
        return func()
      File "/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 242, in runner
        _execfile(setup_script, ns)
      File "/venv/lib/python3.7/site-packages/setuptools/sandbox.py", line 46, in _execfile
        exec(code, globals, locals)
      File "/tmp/easy_install-vx8umdqu/pytest-runner-4.2/setup.py", line 76, in <module>
        'Programming Language :: Python :: 3.7',
      File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/lib/python3.7/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 209, in run
        os.path.join(archive_root, 'EGG-INFO'), self.zip_safe()
      File "/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 245, in zip_safe
        return analyze_egg(self.bdist_dir, self.stubs)
      File "/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 355, in analyze_egg
        safe = scan_module(egg_dir, base, name, stubs) and safe
      File "/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 392, in scan_module
        code = marshal.load(f)
    ValueError: bad marshal data (unknown type code)

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-qj748ds4/pyrsistent/

---

since this isn't a problem with the deadsnakes packaging, and more of a "the version of setuptools that ships with xenial is old" problem I'm going to close this. Thanks for the issue nonetheless!

[neumond]

This is very questionable benefit, I feel much more comfortable having isolated installations. As you correctly noted, pure python packages can work, but compiled can't:

>>> import PIL  # no error!
>>> from PIL import Image
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/PIL/Image.py", line 66, in <module>
    from PIL import _imaging as core
ImportError: cannot import name '_imaging' from 'PIL' (/usr/lib/python3/dist-packages/PIL/__init__.py)

So I can have half-working/half-importable libraries which just delay ImportErrors/other errors (python 3.5 is different to 3.7). Suppose I'm making some timer, which should launch something important after several hours. I forget to install an override package, and instead immediate failure, timer starts, but after several hours fails to do the job. And what I get in return? Just several megabytes less of python files in system only in rare case when you use pure python packages of old versions?

Virtualenvs are not convenient for everything, I'm using lots of stuff installed system-wide (in /usr/local/), per user (pip install .. --user) and with virtualenvs, and that was a complete surprise to discover this shared python3 folder, which you can't even disable easily.

[neumond]

As for pyrsistent I agree this is just old setuptools issue not related to deadsnakes. I was trying to install poetry like this: pip install poetry --user.

[asottile]

• your original failure has nothing to do with the shared folder but that you expect old setuptools to work
• installing with sudo pip (or installing system wide python libraries without the system package manager) is asking for trouble
• virtualenvs are great, you should use them more -- if your issue is "getting things on PATH" symlink the binaries into /usr/local/bin or ~/bin or prefix your PATH (ENV PATH=/venv/bin:$PATH)

in this case it's not our decision but debian's -- consider the upstream python3.7 package (which ships on ubuntu bionic (python3.6)):

root@20ef5fff8b12:/# apt update && apt install python3.7 --no-install-recommends
...
root@20ef5fff8b12:/# python3.7 -c 'import sys; print(sys.path)'
['', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/usr/local/lib/python3.7/dist-packages', '/usr/lib/python3/dist-packages']

diverging from upstream would be a much bigger surprise

[neumond]

System wide is safe enough if you run pip under non-root user. Looks like debian python package policy didn't distinguish python 3 minor versions at all. So that's debian's fault lol.

[asottile]

System wide is safe enough if you run pip under non-root user.

not sure how you're installing into a system directory as a non-root user :P but if it involves chown ... /usr/local then yikes