pypa / packaging.python.org

Python Packaging User Guide
http://packaging.python.org
1.43k stars 923 forks source link

Recommend a default cross platform virtual environment management tool #118

Closed ncoghlan closed 7 years ago

ncoghlan commented 9 years ago

(Note: the issue scope broadened considerably over the course of the subsequent discussion. When reading the comments below, note that the early ones were responding to the original issue title and description, which are preserved below)

As you accumulate more virtual environments, it's useful to have a tool to help manage them (creating new ones, listing the ones that already exist, activating them, deactivating them, etc). virtualenvwrapper, pyenv, vex, pew - these kinds of things.

However, as with packaging tools themselves, new users are unlikely to have the knowledge they need to evaluate the trade-offs between the different tools that are available. In particular, the differences between creating a subshell (vex, pew), modifying the current shell session (virtualenvwrapper), and permanently modifying user configuration settings (pyenv) have different trade-offs in terms of the amount of assumed knowledge regarding the operation of the system shell, as well as the chances of inadvertently impacting the operation of other Python applications on the system.

Considering just those four tools, and their complexity and potential for side effects, I'd assess them as follows:

In that context, I'd suggest that vex would be an appropriate recommendation as a minimal "getting started with environment management" solution. If a given user eventually decides they want something more configurable, or more closely integrated with their preferred shell environment, then they may want to explore pew, virtualenvwrapper or pyenv, and the packaging guide could point them in those directions (with some discussion of the associated trade-offs).

Outside the Python package management space, conda then takes the step up to being a full cross-platform platform, managing the Python runtime and arbitrary external dependencies, rather than just Python virtual environments. However, as an all-in-one solution that bundles environment management, dependency management and binary distribution, conda's operating in a different category from the more single-purpose tools being discussed here.


Original issue title: Mention "vex" somewhere (at least in the "Other projects" section)

Original issue description:

https://pypi.python.org/pypi/vex is a nice alternative to things like virtualenvwrapper that I discovered via the PyCon New Zealand lightning talks.

It works by creating new subshells, rather than trying to alter the existing one. So deactivation is just a matter of exiting from the subshell.

This is really nice, because it's shell independent, and provides all the niceties of an integrated command line application, rather than a collection of shell functions.

ncoghlan commented 9 years ago

As far as "why vex, and not the other helpers?", it's mainly because it's much simpler. It's strictly less powerful than virtualenvwrapper, but still covers all the basics of setting up, activating and deleting environments.

https://pypi.python.org/pypi/pew is another shell independent alternative to virtualenvwrapper, although it aims to match the full virtualenvwrapper feature set much more closely.

Both vex and pew are also pure Python, unlike other POSIX only approaches like pyenv (which I don't consider particularly interesting, as they don't work in native Windows shells)

dstufft commented 9 years ago

Note that the subshell approach is fundamentally dependent on what you do in your shell RC files. This isn't a reason in itself to discourage use but maybe it should be called out? I tried something similar to vex myself and ended up trashing it because it more or less is incompatible with pyenv because pyenv modifies the $PATH in the RC files to put itself at the start of $PATH. vex does call this out in it's PyPI description (although it's wrong for some shells, I'm not sure about bash but for zsh I was unable to use any file to modify the $PATH in a way that didn't follow into a subshell) it seems to make it sound like the user is doing something wrong (it recommends putting ~/.local/bin last in the $PATH, where I would say it should go before /usr/bin as I generally install things there as an override to what the system has installed).

ncoghlan commented 9 years ago

I'd put the blame for that on pyenv (and its rbenv progenitor) being broken by design - it should have the warning, rather than vex.

dstufft commented 9 years ago

Huh? The problem is it adds things to the start of $PATH in the RC files, which is the only way to do that if you want the pyenv installed stuff to take precedence over system installed. The exact same problem occurs if you put $HOME/.local/bin on the start of $PATH.

ncoghlan commented 9 years ago

Yes, the problem is in overriding persistently and globally the default state of the system. pyenv is the tool making global changes, not vex, thus pyenv is the tool that needs the warning.

ncoghlan commented 9 years ago

Compare that to virtualenvwrapper which has no impact on the system if nothing is activated, and even when something is activated, it only affects that particular shell session. Ditto for vex.

dstufft commented 9 years ago

To be clear, if my memory serves I could reproduce the problem with an otherwise empty ZSH install and doing echo "PATH=$HOME/.local/bin" > ~/.zshrc or echo "PATH=$HOME/.local/bin" > ~/.zprofile. There was zero way I could find to put something ahead of the system /usr/bin that didn't also break the way vex works.

dstufft commented 9 years ago

That approach is essentially saying "you're wrong for wanting pip install --user foo to override the system install of foo".

ncoghlan commented 9 years ago

That sounds like a real bug report against vex then.

ncoghlan commented 9 years ago

Although your examples there are taking /usr/bin off the PATH, not putting the local directory before it.

dstufft commented 9 years ago

Er, I just mistyped that, I really meant echo "PATH=$HOME/.local/bin:$PATH" > ~/.zshrc

ncoghlan commented 9 years ago

Although, unless there's some odd difference between inhering PATH from the current shell and via a startup file:

[ncoghlan@lancre py3k]$ PATH=~/.local/bin:$PATH vex misc which python /home/ncoghlan/.virtualenvs/misc/bin/python [ncoghlan@lancre py3k]$ PATH=~/.local/bin:$PATH vex misc bash -c 'which python' /home/ncoghlan/.virtualenvs/misc/bin/python

dstufft commented 9 years ago

Both vex and pew seem to be aware, I was mostly just raising awareness here that such an issue exists and that we might want to call it out (I don't know if we should, was just mentioning it incase we should!).

dstufft commented 9 years ago

Yes there is a difference, because the startup files get run after inheritance inside the subshell.

ncoghlan commented 9 years ago

Ah, roger - so the startup file will override the settings from parent shell. Although wouldn't you also need a "python" or other conflicting application in the local bin directory to actually encounter a problem?

I still think the problem there is with the approach of overriding system executables persistently and globally. And yes, with the benefit of an additional 6 years experience, I do consider the fact that Python allows overriding system packages via --user somewhat dubious - note that we don't allow that to override standard library packages. Putting user site packages last so you can only add new packages, and then requiring virtual environments for overrides would be my preferred solution today.

dstufft commented 9 years ago

Yes, like for example pip itself :(

I think we'll have to disagree on the overriding bit heh, I love being able to override and i wish that Python put the stdlib last not first ;)

ncoghlan commented 9 years ago

It's the system administrator/platform maintainer world view version the application developer/data analyst world view. Former want consistency for ease of support, latter want selective upgrades for access to specific components.

Getting virtual environments to a point where they can override the stdlib might make sense - and may even be feasible once PEP 432 is done and getpath.c (in its current form) is nothing more than a memory.

pfmoore commented 9 years ago

There are some bugs with vex on Windows. There was https://github.com/sashahart/vex/issues/20 which appears to be fixed, but I just tried to test now using vex installed in a virtualenv (which is not exactly a normal configuration, but whatever) and got a bug which appears to be related to the fact that virtualenv on Windows uses Scripts rather than bin. I haven't reported this because I don't have the time to follow up by testing vex properly on Windows, I'm afraid. Also, the comment on the commit that fixes issue 20 (https://github.com/sashahart/vex/commit/b99f2ae1e9cf2ec1e29cba4cf9c3d0b7714841df) doesn't inspire me with confidence that Windows is a fully-supported platform :-(

I'd prefer it if we're cautious about recommending tools without being clear on their possible limitations.

For what it's worth, by the way, https://github.com/pfmoore/ve is my reimplementation of the ideas in vex using Powershell, for Windows... (Disclaimer - it doesn't have much in the way of tests, but I use it myself regularly).

ncoghlan commented 9 years ago

Would it be possible to offer to help set up Appveyor for vex? Yes, writing solutions just for ourselves is often easier, but platform specific solutions are a pain from an educational perspective (that's the big problem with pyenv as well).

pfmoore commented 9 years ago

I'm afraid I don't have the time to offer help for vex, but I'd be happy if someone did and as a result vex gained full Windows support (it's often problematic to even tell if projects support Windows, because they don't think to make an explicit statement - particularly if it's a case of "there's no reason I know of that Windows shouldn't work, but I don't use it so I can't tell", which is basically where vex is AIUI). Sorry, I'm not trying to be anti-vex here (I like the idea a lot), just urging caution on what the PUG says if we can't be sure a solution is cross-platform.

My project is not particularly "just for me", it's just that there's no particularly obvious channels to announce or publish Powershell utilities, so nobody but me knows about it :-)

ncoghlan commented 9 years ago

Sorry, didn't mean to come across as snarky - it's actually irritation that transferred from a conversation with a bunch of folks that were gushing about pyenv, blithely unconcerned with the fact that it only works in a POSIX compatible shell, and hence poses a potentially big hurdle for Windows users.

I found out about vex shortly after that, and was hoping it would be a nice, cross-platform, genuinely shell independent approach to virtual environment management (it's also nice that vex aims to solve just that problem, rather than aiming to branch out to solve the Python runtime management problem as well).

I've also now looked more closely at pew, and matching the full virtualenvwrapper feature set does definitely give it more power. However, the simplicity of vex still makes it quite attractice as a possible initial recommendation, and it doesn't look like pew has Windows based CI set up either.

pfmoore commented 9 years ago

No problem - there's an entirely separate issue here, which is really more for distutils-sig if anywhere, about how projects accurately and honestly express their platform compatibility (given that generally, the Python ecosystem is remarkably platform-neutral). As well as "Supports X" there is "I see no reason this wouldn't work on X but I can't check", "I am actively looking for people to assist with X support", "I want this to work on X but won't compromise on certain principles that may or may not align with how X works", "I think X is stupid, I'll accept patches but I'm not interested in trying to work with X's idiosyncracies" and many other flavours.

All of these are entirely acceptable viewpoints, but they strongly affect usability for people using platform X (both end users who can't help with compatibility patches, and X specialists who don't want to always end up being the provider of support for X on every project they use).

For here, I think we need to only promote tools that are at least as cross-platform as the core tools (pip, setuptools, etc). We can maybe have a "platform-specific or specialised tools" section for things that don't meet that standard, but I'd like to avoid complaints of "I tried <project> and it didn't work". (My mention of my project was essentially a variation of that, by way of example - "I tried <project1> and it didn't work, why didn't you mention <project2> instead/as well?").

sashahart commented 9 years ago

If anyone doesn't want to promote vex that's fine with me, I just made the tool I needed and shared it. But it feels kind of bad to understand from this discussion that I have been somehow dishonest about platform compatibility, and to see other misunderstandings about what vex is doing or not, when a question could have been asked instead. I don't think I have given a lot of cause to assume bad faith. As far as I can tell there has been a workaround for the Windows issue pfmoore filed a ticket about for a while now.

And "Windows users should probably be used to specifying full paths to python" is strictly true, I said this as someone who has been using python command line tools in Windows for some years! If you start to grep through any code base or pile of emails, you are going to find people complaining. Also, you might be surprised at the cussing that happens in a kitchen. Why read into that?

I do examine proposals critically and discuss them rather than just doing whatever anyone says, because I want to control the rate of cruft increase; because I made this thing, I have a certain vision for it (without which a new project is pointless) and a certain awareness of its internal constraints; and because I will be living in this code, trying to refine it and fielding complaints about it for some years to come. I think it is not necessarily helpful for the project itself when people apply pressure to route around that critical process in order to get their way, and it's certainly discouraging.

To be clear, I actually have nothing against Windows (except as an instance of that pernicious general category "software") and I do use it occasionally with vex. As is often the case of Linux users, I have used Windows for many years and have some awareness of it. I just don't have trivial access to a box with the necessary setup in a reproducible environment, it is always an ordeal. I'm certainly happy to work on Windows in general, though.

If we're honest, most programs are not bug-free on any platform. Additionally, if we're honest, command-line Python on Windows has certain inherent frictions and PATH is near the center of that cluster. There are some things developers of Python programs just can't do about platform brokenness - Windows users may feel that mentioning these issues is picking on them right up until they have to deal with the same issue directly. And I suspect even exemplars like pip and setuptools struggle with Windows issues and can't do certain things.

What I'm against is broken "fixes" - approaches which seem to solve a problem, but are inadvisable for other reasons. For example, we can differ about this, but I still don't think it makes sense for every project which runs executables on Windows to manually reimplement the PATH search according to its own ad hoc rules. It might seem to fix something for you, and it might get me off the hook temporarily if I stop caring about the quality of my output, but there are also negative externalities.

vex doesn't try to do very much and I don't foresee that vex will add any more major features, either, so there is no chance of a feature arms race with pyenv or pew which both have different project scopes. I think some people just like vex and some people just like pew or virtualenvwrapper or whatever, according to taste. As someone who has strong opinions about tools I appreciate being able to get the flavor I want (and sympathetically, I appreciate others being able to get the flavors they want) even if that means there are multiple things in the ecosystem.

pfmoore commented 9 years ago

Um. @sashahart, I need to apologise. There's a lot of things getting mixed up in this discussion, probably because I am typing without thinking sufficiently beforehand. I'll read and digest your response properly before commenting, but I do want to make one thing clear immediately - I never meant anyone to think that there's any dishonesty about vex's compatibility. Any misunderstanding or misinterpretation was my mistake, and I apologise unreservedly if I implied otherwise or caused you any offense.

(PS - this is getting very offtopic for this issue - if anyone would prefer that we take this conversation offline, please say so)

ncoghlan commented 9 years ago

Firstly, thanks @sashahart for writing vex, and for being willing to weigh in here! vex and pew are by far the frontrunners in Windows compatible virtual environment management, just by being written in Python rather than a platform specific shell language (conda also has excellent Windows support, but is well and truly into the space of providing a full "cross-platform platform", rather than focusing on the simpler task of managing Python environments).

As far as the issue goes, I'm happy to broaden the scope to match the "Provide a default recommendation for a cross platform environment management tool" discussion that emerged. That kind of "If in doubt, start here" kind of recommendation is one of the most valuable aspects of packaging.python.org - making folks aware that alternatives exist, but letting them postpone exploring them if the default recommendation is sufficient for their needs.

ncoghlan commented 9 years ago

Issue title & description have been updated to match the scope of the discussion. I've preserved the original title and description to help the comments above make more sense.

r1chardj0n3s commented 9 years ago

+1 on @ncoghlan's comment about the scope and us providing useful recommendations.

Ivoz commented 9 years ago

@ncoghlan you might be confused by the name (which I wouldn't blame you for), but pyenv does not actually manage python virtual environments, like the other tools you are comparing here.

It manages python installations, on *nixes. So you shouldn't be comparing it to virtualenvwrapper et al, they're accomplishing different things. Indeed, pyenv also has a compatibility wrapper for virtualenvwrapper to allow both to work nicely in concert.

By the by, conda also muddies the "traditional" waters by allowing one to manage the python version ( / installation) used inside the environment it creates; whereas for the other tools managing environments and python versions would be regarded as two distinct processes.

ncoghlan commented 9 years ago

@Ivoz Yeah, that's a fair point. conda's actually designed as a full cross-platform platform - it's useful to think of it as a user space distro (i.e. something more in the Nix vein than any of the Python specific tools)

berdario commented 9 years ago

Uh, I just discovered that Nick Coghlan uses (or attempted to use) my tool and I felt all giggly and proud :3

Anyhow, as you probably know, since October I've added Appveyor testing to pew. While that's good news (no random breakages while making changes for linux!), the situation is not perfect. For example I haven't been able to use virtualenv-clone (and thus pew cp) on Windows recently (but I distinctly remember it working when I first started pew): all the related tests are currently skipped.

I actually was not aware of the existence of a single pew user on Windows (and before october maybe there were 0 users indeed), so: please leave some feedback! (like, installation experience... is chocolatey good enough? issues... any comments)

That said, I like the idea of promoting some development tools (be it pew, vex or any other). It seems to me that some newcomers to python don't get to appreciate how easy it is to reuse something from the pypi, and that might be also because virtualenv is an extra thing to install for them (ok, now we have venv in the stdlib, but that's just a first step).

Imho, something like clojure's leiningen would be the gold standard for such development tools: it scaffolds your new project, handle dependencies, install the compiler/runtime/interpreter (that's a library like any other for clojure), run tests...

Granted, python is different, since we cannot have multiple versions of the same library installed in the same store and it cannot be compiled like Clojure (lein doesn't need clojure to bootstrap: only Java).

To get the same thing with Python we need:

Btw, I'm also a Nix user (not 100% comfortable with it, yet) and that could solve all of these needs as well (except the scaffolding part), but it's probably a bit brash to bless it as the go-to tool for Python development.

(sorry if I went a bit off-topic with my comment: at first I wanted just to chime in)

ncoghlan commented 7 years ago

Eep, it's kinda scary to see how long ago I started this discussion :)

Anyway, a conversation today prompted me to revisit it in light of http://www.curiousefficiency.org/posts/2016/09/python-packaging-ecosystem.html, another few years of using vex regularly, and the creation of pipenv last year.

The key point is that while I still like vex for my own use, the fact that it doesn't follow the same traditional <tool> <subcommand> structure that pip uses is a problem from a learnability perspective.

The fact that pew supports operations like pew inall pip install --upgrade pip setuptools wheel, and pew add for sharing content between virtual environments are also major points in its favour.

We also know from pipenv that pew is viable as a building block for higher level more opinionated environment management tools.

So I think the way forward here would be to:

theacodes commented 7 years ago

@ncoghlan Sounds good to me - happy to review but I won't have time to author this anytime soon.

pfmoore commented 7 years ago

@ncoghlan Do we have confirmation that pew works reliably on Windows? From what I recall, I tried it some time ago (probably way back when you originally started this discussion) and found a number of bugs. When I reported them the pew developers were happy to accept patches but didn't have much in-house expertise on Windows. Having said that, I can't find those issues and the github project owner seems different, so it's possible I'm thinking of a different project. There seems to be specific support for powershell in pew, which suggests they are covering Windows specifically.

I'm happy to recommend a tool, but IMO it's important that we're comfortable with its cross-platform support. I'll try to get a chance to check out pew again, but life's pretty hectic at the moment, so I can't promise.

pfmoore commented 7 years ago

Apologies - please ignore the above comment, I was getting confused between pew and pex. I've got no reason to believe there's any issue with the Windows support of pew.

ncoghlan commented 7 years ago

@pfmoore As far as I know, the one known-missing piece on Windows is the lack of pew install support for Python runtime management (since the underlying pythonz library is *nix only, those features don't work on Windows).

So Windows folks wanting that kind of capability are either still going to need to a full cross-platform platform like conda, or else figure out what would be needed to enable pew install in Windows (e.g. resolving the limitation in pythonz, or perhaps finding a way to use the pre-built conda binaries).

However, all the virtual environment managements features work (and are tested on Appveyor).

pfmoore commented 7 years ago

Cool - sorry for the noise.

ncoghlan commented 7 years ago

Considering my answer to https://twitter.com/ncoghlan_dev/status/900547395095625728, another notable absence from the current tooling recommendations is the fact that we don't point anyone towards pip-tools or pipenv either, and frankly, if you're doing professional software development with Python (as opposed to ad hoc scripting), you should probably be using either conda or one of those two options.

I filed https://github.com/pypa/python-packaging-user-guide/issues/367 for that.

ncoghlan commented 7 years ago

I'm going to close this as:

  1. 369 added a new tutorial that will recommend pipenv as the starting point for new users, rather than raw pip

  2. pipenv not only integrates with pew for virtual environment management, but also integrates with pyenv for language runtime management

That still leaves Windows users without an automated option for managing the language runtimes themselves (short of switching to something like conda), but that's not going to be resolved here in PyPUG, since the problem is with the tooling not existing for that (as far as we're aware), rather than our failing to guide new users through a plethora of competing options.

So probably the best place to discuss that now would be on the pipenv repo, with actual Windows users and folks like @zooba considering whether it makes the most sense to try to add Windows support to pyenv itself, or to instead create a Windows-specific backend, and have pipenv use that. (I'm not going to file that issue though, since my Windows machine is pretty much solely for games these days)