Default to --user

[dstufft]

When pip is installed on a system that has an OS Python install there is currently a problem where pip install foo will throw an error because it doesn't have file permissions. This causes people to instead run sudo pip install foo which globally installs to the system Python. This creates an issue where people are using pip to manage system level packages when they should likely be using the system package manager.

So my intention is that pip should default to --user however there are a few sticking points with this:

• How does this interact with Windows? Does this make sense there?
• How does this interact with altinstall'd Pythons? Specially such as are installed with tools like https://github.com/yyuu/pyenv
• What do we do for when people do invoke pip as root? Installing into /root/.local/ does not seem very useful.
• What does this mean for get-pip and the pypa install instructions?
• ~/.local/bin is not on many people's $PATH, is there anything that can be done about this?
• --user installs lack precedence to global easy_install'd packages, which can be quite unexpected.

There are a number of issues that are relevant here: #624 #1443 #1153 #1500 #1051

/cc @ncoghlan

[qwcode]

What does this mean for the pypa install instructions?

currently the mention of root or Administrator access is in a footnote, so the instructions would fundamentally stay the same (assuming get-pip automatically defaulted to --user as well). the change would be to explain that they will end up with installs that only apply to their current user.

regardless of the --user change, the current instructions need to explain that some distributions like debian, are disabling ensurepip, so they shouldn't go looking for it as a way to get pip.

[dstufft]

Long term I'm pretty sure they are going to keep it enabled, they are just figuring out how to do it.

[qwcode]

although this issue is described as being about where pip should manage packages by default, almost more important (to me at least) is that this is indirectly about where get-pip will place pip by default.

[qwcode]

How does this interact with altinstall'd Pythons? Specially such as are installed with tools like https://github.com/yyuu/pyenv

the user scheme is .local/lib/pythonX.Y/site-packages, so if you're managing multiple pythons with the same major/minor version, you could end up with a mess I guess.

for tools like that, that manage real pythons, pinning pip to global mode would make sense for those.

[dstufft]

A relevant issue on the redhat/fedora bug tracker https://bugzilla.redhat.com/show_bug.cgi?id=662034#c10

[dstufft]

It looks like for Fedora/RedHat PATH=$PATH:$HOME/.local/bin:$HOME/bin is already in their /etc/skel/.bash_profile. This would make stuff installed with --user still show up by default, at least for bash users (default shell). Maybe other distros already have this?

[pfmoore]

There's nothing specially different about Windows that I know of. But we've only just got the standard Python directory onto people's PATH, I don't expect that getting a per-user scripts directory on is going to be particularly easy - and there may be technical difficulties as well, I'm not sure putting a user-level environment variable like %LOCALAPPDATA% into the system-level %PATH% even works :-(

I'm against rushing this. I would prefer waiting till we've had more experience with user installs and have managed to iron out the issues first. I know that's a chicken and egg problem, but I don't see rushing into a change as helping.

Also, with my Windows hat on, I'd have to say that making it harder for Unix users who haven't yet worked out that careless use of sudo is bad to do stupid things, isn't really that good a justification...

I'm assuming that this would only apply when running pip from a system Python.

[dstufft]

Well the different thing on Windows is there isn't a system provided Python like there is on *nix that also comes with system provided Python packages :)

I'm not particularly wanting to rush this either. I just wanted to start the discussion.

This doesn't really affect people who type sudo pip install as I think installing something as root should still go to the system site packages by default. This would be to guide people towards using --user when running as an unprivileged user. Right now what you get is that pip install as a regular user just bombs out with a permission error. Even if we improve that message to bomb out and suggest --user that's still not the greatest UX.

In talking to one of the Fedora people, I think the way it may make to do this is to implement a permissions check. If I have permission to write to the site-packages directory then I am a privledged user and pip installs to the global Python, If I do not then I am an unprivileged user and it installs to --user.

[ncoghlan]

As far as Windows goes, we usually duck the problem because we install Python to a non-privileged directory by default, so pip doesn't trigger UAC when installing globally. If someone changes that to install into Program Files instead, then UAC will trigger when they run pip (which is also OK). I'm not sure what happens if they select a per-user install in the installer.

As Paul notes, PATH on Windows doesn't include the user scripts directory yet (just the global one), so that's definitely worth taking into account.

I don't see any major barriers on the POSIX side though. How does this UX idea sound: for wheel based installs, check for permissions on the installation target directory, and if the user doesn't have write permissions, put up a prompt asking if they would like to do a --user install instead? A new "--global" or "--system" flag would force the "no" answer.

And then at some point in the future, we could skip the prompt and simply assume --user if the permissions weren't right for a global install.

[dstufft]

A prompt (except in the case of --no-input) probably makes sense for the transition step. I think checking permissions should make things work for Windows too since by default the location is in a spot where people have permissions on and so the only time they'd hit this would be if they are using a systems admin provided Python that explicitly didn't give them permission.

Essentially this makes our failure mode much better.

[qwcode]

making it harder for Unix users who haven't yet worked out that careless use of sudo is bad to do stupid things, isn't really that good a justification..

sudo and security is really a secondary issue IMO.
this is primarily about preventing the conflict between OS packaging and pip in the system python.
Currently, linux users are often placed in an impossible situation.

1. OS Mandate: 'Use the OS pkg mgr; Don't infect your system with roque pip-installed packages (including a get-pip'd pip)'
2. Reality: "The OS packages (including pip) are too old, and I can't upgrade to the experimental release of the distro, just to get upgrades of some package. I'm going rogue...."

[qwcode]

another baby step is to document (assuming we test it) that you can get-pip.py --user, and have the PUG mention --user as a possibility for the install of virtualenv (and wheel, and twine too, if not in a virtualenv)

[ncoghlan]

FWIW, distro vendors also consider the status quo a problem and are looking at various ways to improve the tools for doing selective upgrades without impacting core OS components. Still worth us tackling from the upstream side though, since those efforts are at various stages of maturity and we want a consistent cross-platform solution for end users.

[pfmoore]

I have no problem with changing things to help co-operate with distros on Unix; if defaulting to --user on Unix and not on Windows is acceptable, that's fine with me. I just don't see switching from a bad experience on Unix to a bad experience on Windows as a good trade-off. And I'm yet to be convinced that --user on Windows is ready for prime time.

Note that triggering UAC when running pip is bad on Windows, because what it actually means is that pip won't run except in an elevated console window. It does not mean that pip users get a nice prompt which they can say "OK" to, unfortunately, that's only how GUI apps work...

[dstufft]

@pfmoore What do you think the user experience of Windows should be when you pip install something and you don't have permissions to write to the directory?

[dstufft]

FWIW I'm totally OK with making this optional depending on Windows or not. I'm just wondering if the check of "Do I have permissions to write to the site-packages folder" check won't achieve that anyways in 99% of installs, and if installing to --user would be a better experience in the fraction of cases where you don't have write permissions to the site-packages directory.

In other words, if we check permissions the proposal isn't replacing global install with a user install, it's replacing a permission error with a --user install.

[qwcode]

the proposal isn't replacing global install with a user install it's replacing a permission error with a --user install

ok, but recall that the permissions check idea (#1048) won't the be the easiest thing

[dstufft]

It'll be pretty easy in the Wheel case, and we can get the 99.9% case covered with sdists, it should only be a problem in setup.py's that do really funky things.

[pfmoore]

What do you think the user experience of Windows should be when you pip install something and you don't have permissions to write to the directory?

Well, being able to write to the site-packages directory is so rare on Windows that I'm not sure how relevant the question is in practice. But if it happened, I would say that an error saying "system site-packages is not writeable - did you mean to use --user?" would be the sensible approach.

Actually, I think that would be better on Unix as well. Switching the behaviour depending on writeability is bizarre, and it's not clear to me if using --user for a personally-built Python in a writeable directory is the right approach (but I have precisely zero experience of such a setup, so I don't have anything to back that opinion up).

Suggestion: Start with an error suggesting --user as above. Once people are using --user more commonly, and we have more experience with it, then let's consider switching the default.

[ncoghlan]

The approach Paul suggests is roughly what I had in mind, but with a yes/no prompt rather than bailing out with an error message. There are pros & cons to those two approaches (mostly relating to how they fail when invoked from another script).

[rkuska]

Would it be possible to add to Python something like sys.localprefix (I see python-dev here)? Sys.localprefix would be default to sys.prefix. Distros could define it same way as they define prefix (e.g. prefix = '/usr', localprefix='/usr/local'). Pip and easy_install would use sys.localprefix as a location to install to if used as sudo, if they don't have sufficient rights fall back to --user.

[bkabrda]

We've done something like this with "gem install" in Fedora 17 and Ruby devels were generally very satisfied about it, so I thought I'd share:

• "gem install foo" installs "foo" into ~/somedir if uid != 0
• "gem install foo" installs "foo" into /usr/local/somedir if uid == 0
• "yum install rubygem-foo" installs RPM-packaged "foo" into /usr/somedir

I think that choosing install dir based on user privileges would be very confusing for people; "pip install foo" should work the same way for all non-root users. If you consider majority of users, they'll want to "pip install" into their home and "sudo pip install" into a system-wide dir. Therefore I think it's the best approach to do this based on uid as shown above - and for those users who are not root but want to install into a system-wide dir, there is always the --target option.

[Ivoz]

@bkabrda that's an easy choice to make as long as you're not worried about adding scripts to $PATH. What have you done there?

[bkabrda]

@Ivoz as noted by @dstufft, Fedora already has $HOME/.local/bin on PATH, so everything worked out of the box (speaking of scripts).

[ncoghlan]

Slavek's suggestion sounds good to me for POSIX systems. It should also work nicely with containers, since stuff running in a container can be told to think it is uid 0, even though it's something else entirely from outside the container.

For Windows, I'm less sure what the right answer is. We don't have quite the same problem with getting into an argument with the system package manager, although it does exist to some degree (pip install vs MSI installers), and have the additional complication that even in Python 3.4, the installer's optional PATH modifications only add the global Scripts directory, not the per-user one.

Windows also has weirdness based on whether Python is installed to the default location (uncontrolled) or into Program Files (UAC privilege escalation needed to make changes).

[felixonmars]

The "localprefix" idea suggested by @rkuska looks nice to me, simple and easy. Also I really like the approach @bkabrda suggests, especially for distros like Arch that make python 3.x the default python.

One more note is, Arch has disabled ensurepip in the 3.4.0 release just like Debian (mostly because we want to provide latest versions of setuptools and pip, while the bundled versions may be outdated for ages), so it would be nice to have a tip for it.

@lvoz For Arch we didn't add any path under $HOME to PATH, but I still think it fine. We already have wiki entry for Ruby that tells the user to add it.

[ncoghlan]

It would be nice if rather than just disabling it, the Arch and Debian developers would help Slavek work on his patch to have ensurepip reconstruct a wheel from the system versions and install that into virtual environments.

[bkabrda]

Just FYI, we already have working Python 3.4 RPM builds in Fedora's Copr build system (testing, not advisable to install them if you don't want to break anything). If you want to know more about how we approach this, have a look at the code etc, see my discussion with Barry Warsaw [2] at debian-python ML. (Our patches still need some love before we propose them upstream, but everything works ok for us downstream ATM)

[1] http://copr-fe.cloud.fedoraproject.org/coprs/bkabrda/python-3.4/ [2] https://lists.debian.org/debian-python/2014/03/msg00046.html

[felixonmars]

Sorry, but AFAIK Arch never supported wheel, nor do we want to make ensurepip invoke package manager (correct me if I understand this approach wrong, though).

For ensurepip, it would be a nice idea to warn the user not to install pip using ensurepip though, and that's the most I can think of for now.

[bkabrda]

Our ensurepip will never invoke package manager. In short, Fedora's ensurepip will rely on system packaged setuptools and pip to be always present (our python3 package will depend on them, which will create a dependency loop, but we can handle that) and when user will create new virtualenv, it will repack system setuptools and pip into wheels and install them into the new virtualenv. We're starting to get a bit off-topic, so I guess we should continue this discussion someplace else...

[dstufft]

@felixonmars Does that mean on Arch venv is broken the same as it is on Debian?

[felixonmars]

OK, thanks for the explanation, I think I got the idea. I'll see what I can do and follow up on debian-python for this.

@dstufft Yes, the bundled version got installed instead of the system version.

[dstufft]

@bkabrda So that solves the problem if you're invoking pip with a system Python, we have to special case virtualenv/venv in that case (which isn't the end of the world). But it also means that we're assuming all Pythons are system Pythons, even ones installed by the user into their homedir such as with https://github.com/yyuu/pyenv (which is how I run Python actually). So I'm not sure, that's why I thought of the permission check.

[qwcode]

ditto what @dstufft said, in using a tool like pyenv to juggle non-root pythons, it would be pretty odd for root to be the key to do a global install (where "global" != "system")

[qwcode]

We've done something like this with "gem install" in Fedora 17

you mean the rpm of gem for fedora 17 has this as a distro hack?, or gem itself has this logic?

[qwcode]

one thought is that pip should provide a "system friendly mode" that can be turned on by distros that are packaging pip, or when users know they are installing pip themselves into a system-managed python. otherwise, pip does it's normal global-by-default logic.

[dstufft]

Ah @felixonmars I just saw https://projects.archlinux.org/svntogit/packages.git/tree/trunk/PKGBUILD?h=packages/python , if all you're doing is calling --without-ensurepip that's fine. That leaves ensurepip module intact for the venv module. The patches Ubuntu (Debian?) currently has in place actually calls rm -rf on the ensurepip package itself, making it unavailable to be called within a venv.

[merwok]

I think defaulting to user installs was proposed by Nick two years ago, or maybe someone else longer ago, but the thread went into many directions and was not fruitful. Main obstacle was backward compat IIRC, but these days sysadmins and pip users may be more used to be careful with virtualenv/pip/setuptools updates, for better or worse. I don’t remember if that discussion happened on distutils-sig or pypa-dev.

[qwcode]

it was catalog-sig: https://mail.python.org/pipermail/catalog-sig/2013-February/004773.html

[felixonmars]

@dstufft Oh, I see... But I still feel wrong to have a bundled (maybe old) version installed while there are packages installed system-wise :)

[dstufft]

@felixonmars well the problem is you can't install a system package into a virtualenv afaik :) (And even if you could there are different concerns with what you want in a system package vs inside a virtualenv)

[felixonmars]

@dstufft That's where Slavek's rewheel kicks in, but that's a bit off-topic :)

[dstufft]

Yea, check the last bit of https://lists.fedoraproject.org/pipermail/python-devel/2014-March/000583.html :(

[felixonmars]

@dstufft Thanks!

Not sure if I should continue the discussion there, because that shouldn't be a problem since we didn't unbundle pip in Arch, and rewheel should generate usable wheel :)

[bkabrda]

@dstufft you're right that we would probably need to specialcase virtualenv and possibly even pyenv, which seems ugly. I have to admit I'm not that familiar with pyenv, so I can't think of any "proper" solution for that right now. I'll try to have a closer look at it and think of something.

@qwcode for "gem install" on Fedora, it's a downstream patch. TBH I didn't touch Fedora's Rubygems for long, but the last time I checked, it was downstream only.

[rkuska]

I have posted link to this issue to related bug in python [1]. I would like to move this discussion to python core, to have a configurable location for local installs of not only pip but also easy_install and python setup.py install, after that pip could use (e.g.) sys.localprefix dir if uid==0 and install to user dir if uid!=0 as @bkabrda mentioned.

For the time being I am fine with pip installing to user dir by default.

[1] http://bugs.python.org/issue1298835

[rkuska]

As Nick pointed out I moved the discussion from issue1298835 to pypa-dev [1]. [1] https://groups.google.com/forum/#!topic/pypa-dev/r6qsAmJl9t0

[hickford]

Default to --user

Good idea!

pip install foo will throw an error because it doesn't have file permissions. This causes people to instead run sudo pip install foo which globally installs to the system Python

It's worse than that. Many people (such as users of a university computer network) don't have sudo privileges. When pip fails with a permission error, they will either:

1. Email their administrator and ask them to install the package. That's slow and tedious for everyone.
2. Give up and do without the package.

Only if they are particularly experienced or well-informed will they try pip install --user.

If we end up deciding against 'default to --user', we should consider the less disruptive alternatives:

1. Fallback to --user (if install fails due to perimissions errors)
2. (If install fails due to perimissions errors), encourage the user (in large friendly letters) to try --user

[astrojuanlu]

I actually have been using --user in all my installations for a while now, but generally people are not aware of this option. I agree it's annoying to see the permissions error every single time you forget the option. I noticed user installations are not compatible with conda though (conda/conda#448).

[zooba]

I'd like to revive this, as people are going to run into it on Windows far more often now that 3.5 defaults to installing into Program Files (which requires administrator privileges to modify).

I like the "fallback to --user on permission error" option best for installs, with "default to --user and fallback to system" on uninstalls and maybe add a --system or --system-only option to prevent the fallback. Users deliberately running pip with admin permissions will get a system install as expected, and currently users without permissions would get an error, so I don't believe there's any back-compat issues here.

My aim is for an uninformed user to be able to do pip install spam; python -c "import spam" without errors (though possibly a warning when retrying with --user).