pip should not execute arbitrary code from the Internet

[glyph]

When you 'pip install' something, it fetches the code from the internet, and then executes it. If you follow the advice of many projects and 'sudo pip install' something, pip executes that code from the internet as root.

pip does not do TLS certificate verification, nor does it do package signature verification, nor does it even do DNSSEC. There is no assurance whatsoever that the code being installed came from the intended source. The archetypical hipster hacker doing a 'pip install django' over some cafe's wifi will be pwned within seconds if the DNS for pypi.python.org happens to be spoofed.

I believe that this might be addressed by https://github.com/pypa/pip/pull/402 but that deals with a bunch of other issues as well, and I felt there should be a report somewhere about this somewhat well-known deficiency in pip's download and update procedures.

[westurner]

Though, I must point out -- as others have on this ticket -- that the following are separate issues:

Documentation links from here, reddit (2), [...] :

PEPs

• PEP 0001: PEP Purpose and Guidelines (2000)
• PEP 0241: Metadata for Python Software Packages (2001)
• PEP 0301: Package Index and Metadata for Distutils (2002)
• PEP 0314: Metadata for Python Software Packages 1.1 (2003)
• PEP 0345: Metadata for Python Software Packages 1.2 (2005)
• PEP 0386: Changing the version comparison module in Distutils (2010)
• PEP 0440: Version Identification and Dependency Specification (2012) [DRAFT]
• PEP 0426: Metadata for Python Software Packages 2.0 (2012) [DRAFT]

JSON metadata (pymeta.json, pymeta-dependencies.json) is generated from setup.py ... http://hg.python.org/peps/file/default/pep-0426/pymeta-schema.json

• PEP 0381: Mirroring infrastructure for PyPi (2009)
• PEP 0427: The Wheel Binary Package Format 1.0 (2012) (readthedocs)

Python

• http://docs.python.org/2/distutils/
• http://docs.python.org/3/distutils/
• https://pypi.python.org/pypi/stdeb

PIP

• http://www.pip-installer.org/en/latest/
• http://www.pip-installer.org/en/latest/usage.html#pip-freeze
• http://www.pip-installer.org/en/latest/usage.html#pip-wheel

Setuptools

• http://pythonhosted.org/setuptools/
• http://pythonhosted.org/setuptools/merge.html

Distlib

• http://pythonhosted.org/distlib/
• http://pythonhosted.org/distlib/tutorial.html#signing-a-distribution
• http://pythonhosted.org/distlib/tutorial.html#verifying-signatures
• http://pythonhosted.org/distlib/tutorial.html#using-the-wheel-api
• http://pythonhosted.org/distlib/tutorial.html#using-the-metadata-and-markers-apis
• http://pythonhosted.org/distlib/tutorial.html#getting-hold-of-root-certificates

Narrative Documentation

• https://python-guide.readthedocs.org/en/latest/scenarios/admin.html
• https://python-guide.readthedocs.org/en/latest/scenarios/ci.html
• https://python-guide.readthedocs.org/en/latest/shipping/packaging.html
• https://python-guide.readthedocs.org/en/latest/shipping/freezing.html
• https://python-packaging-user-guide.readthedocs.org/en/latest/
• http://guide.python-distribute.org/index.html -- The Hitchhiker’s Guide to Packaging
• http://pythonhosted.org/an_example_pypi_project/ -- An Example PyPi Project (with Documentation)
• https://github.com/ipython/ipython/wiki/A-gallery-of-interesting-IPython-Notebooks#-reproducible-academic-publications
• http://scipy-lectures.github.io/intro/language/reusing_code.html

Glossaries

• https://python-packaging-user-guide.readthedocs.org/en/latest/glossary.html
• http://guide.python-distribute.org/glossary.html
• http://docs.repoze.org/compoze/glossary.html
• http://sphinx-doc.org/ext/intersphinx.html

Reddit threads:

• http://www.reddit.com/r/Python/comments/1bx3vj/how_are_python_apps_deployed_to_production/
• http://www.reddit.com/r/Python/comments/1hhi5o/current_python_package_management_frameworks
• http://www.reddit.com/r/Python/comments/1ew4l5/im_giving_a_demo_of_python_to_a_bunch_of_java/#ca4gf55

tag: https://en.wikipedia.org/wiki/DevOps

[westurner]

https://python-packaging-user-guide.readthedocs.org/en/latest/packaging_tutorial.html#create-your-first-release

Source Repository GPG

• http://stackoverflow.com/questions/10077996/sign-git-commits-with-gpg
• http://stackoverflow.com/questions/11556184/whats-the-purpose-of-signing-changesets-in-mercurial

Python Package GPG (./<package>.asc)

• http://pythonhosted.org/distlib/tutorial.html#signing-a-distribution
• http://pythonhosted.org/distlib/tutorial.html#verifying-signatures

For any archive downloaded from an index, you can retrieve any signature by just appending .asc to the path portion of the download URL for the archive, and downloading that.

Python Wheel JWS S/MIME (PEP 427)

• https://tools.ietf.org/html/draft-ietf-jose-json-web-signature-11
• http://www.python.org/dev/peps/pep-0427/#signed-wheel-files
• https://bitbucket.org/dholth/wheel/src/tip/wheel/signatures/__init__.py
• https://distlib.readthedocs.org/en/latest/internals.html#the-wheel-api

Index Mirror DSA (PEP 381)

• http://www.python.org/dev/peps/pep-0381/#mirror-authenticity

[Cryptographic] Hash Functions

• https://en.wikipedia.org/wiki/Cryptographic_hash_function#Verifying_the_integrity_of_files_or_messages
• https://en.wikipedia.org/wiki/Cryptographic_hash_function#Hash_functions_based_on_block_ciphers
• https://en.wikipedia.org/wiki/Category:Cryptographic_hash_functions
• https://en.wikipedia.org/wiki/Hash_function_security_summary
• https://en.wikipedia.org/wiki/Category:Broken_hash_functions
  • https://en.wikipedia.org/wiki/MD5
  • https://en.wikipedia.org/wiki/SHA-1
• http://pythonhosted.org/passlib/lib/passlib.hash.html#unix-modular-crypt-hashes
• http://pythonhosted.org/passlib/modular_crypt_format.html

x-post to #1035

[dstufft]

I'm going to close this ticket as it has no clear goal. Pip no longer downloads things from PyPI without TLS.

If there are specific deficiencies with what pip offers per item tickets should be opened for each one.

[radiosilence]

Also could I point out that doing sudo pip install is something you should never do? Either install --local, user a virtualenv, or use a system package manager for installing things systemwide.

[dstufft]

Using sudo pip install is fine. Not all systems with sudo even have a package manager or often times the system package manager is very outdated or missing that item all together.

pip vs OS packages is a user choice with various trade offs to both sides of the argument.

[radiosilence]

If your system doesn't have a package manager, why not use a virtualenv?

If you are using a system with any kind of package manager, you should never sudo pip install, because then you are messing with a file structure that is supposed to be managed with the package manager, and say a system component requires a specific version of something, you are going to get file conflicts and all kinds of hell.

If a system-wide package requires a python component, the dependency should be resolved with the package manager. If not, virtualenv.

I don't think it is a personal choice, because it's a terrible habit and I've seen many systems get screwed up due to it by people who are making a "user choice" and have no idea the implications of what they are doing. If you are using, say, OSX for development, use a virtualenv, if you are creating a distributable package, then use brew (or whatever) to install Python dependencies (and if brew doesn't have the package, change that by making a brew package).

For instance - I want to run uWSGI as a system-wide daemon. The version of uWSGI in my package manager (let's say Debian wheezy) is totally out of date. So, I create a virtualenv in /opt/uwsgi and install it there, then have my init script reference /opt/uwsgi/bin/uwsgi, and don't fuck up my system.

It's just being responsible.

[dstufft]

If my system doesn't have a package manager how am I supposed to install virtualenv ;) Also needing to activate a virtualenv for using command line tools is ugly.

I don't know what Linux systems you use, but in my preferred Debian based ones sudo pip install installs to /usr/local which, as far as I understand the FHS, is outside of the domain of the system package manager and is intended for just such use.

[radiosilence]

Ok, makes sense - though I would explicitly check your distro does this first. But then, what's the problem with doing pip install --user and then adding ~/.local/bin to your $PATH?

[dstufft]

It's only available to my user :) Get's annoying when I'm bouncing between different accounts for different things.

[westurner]

AFAIK, sudo pip install (like every other unchecked source of executable code) should not be run as root on an actual system. Debian mitigates around this with fakeroot and virtual containers for building packages. Files built are signed and checksummed.

If pip dropped privs to the minimum it needs to install into a path in PATH and sys.path, it would still be a risk to execute sudo pip install.

Could someone indicate to me how live.sysinternals.com is more or less of a risk than just 'sudo rm *'?

Don't open (executable) email attachments. Don't sudo pip install.

[westurner]

I like virtualenv and virtualenvwrapper alot. I like not having write permission to scripts that I execute often (e.g. --user).

Does pip support --prefix? Does pip do snapshots/backups prior to installation?

[merwok]

how am I supposed to install virtualenv

You don’t have to: just download virtualenv.py and run it. From that first venv, you can get pip, venvwrapper, etc. Yes, it’s per-user.

needing to activate a virtualenv for using command line tools is ugly.

I add the bin directory of the bootstrap venv I describe above to my PATH.

[radiosilence]

Also, perhaps one could avoid depending on pip --user (which I find irritating) by running virtualenv ~/.local

[westurner]

Also, perhaps one could avoid depending on pip --user (which I find irritating) by running virtualenv ~/.local

For the sake of consistency, a mkvirtualenv local and a symlink to $VIRTUAL_ENV/local make it easier to work with virtualenvwrapper.

[merwok]

I’m not sure consistency is an argument here. The venvs that I manage with virtualenvwrapper are used for my Python projects and can be deleted or re-created at all time, but the global/bootstrap venv I created in ~/.usr (or ~/local in your example) is not throw-away: I depend on having things installed in there (mostly scripts/programs, not libs).

[westurner]

This is way OT. To each their own. I should have been more clear. I believe consistency to be an argument here because I like to consistently apply the same tools and processes to managing virtual environments. Commands like cpvirtualenv, cdvirtualenv, and cdsitepackages are not present with a (special-cased) plain-old virtualenv.

When I backup a virtualenv, I usually only need the pip freeze and contents of ./src. When I backup a homedir, I usually don't want to copy compiled, version-specific objects into a backup archive. (So I install scripts/programs as a dotfiles egg with a ./scripts folder and/or [console_scripts] entrypoints in setup.py.)

[nejucomo]

I consider the discussion about user permissions and install locations when running pip to be orthogonal to this ticket, which is about TLS verification. So, I created #1169 to capture that orthogonal issue.