Windows: vex myenv pip install ... does not install in virtualenv

[schlamar]

If I run vex myenv pip install ... it does not install the packages in the virtualenv. It installs them in the main site-packages. A shell starts correctly, though.

Running Windows 7 x64 and Python 2.7.

[sashahart]

First of all I'll note this isn't happening on Linux, so it's likely a Windows-specific issue.

Getting a clean Windows environment usable for reproducing issues requires an inordinate amount of time on my part so it could be a while before I can personally reproduce this and find a cause. In the interim, especially if you want a solution quite soon, you may wish to investigate a little on your own.

You could verify that vex is setting VIRTUAL_ENV appropriately. If that's not happening then nothing is going to work as intended in vex anyway. e.g.:

vex myenv python -c "import os; print(os.environ['VIRTUAL_ENV'])"

It is also possible that pip has some behavior difference across platforms that causes it not to correctly honor the virtualenv in WIndows.

Typically pip will install packages to a location used by the python that was used to run pip. So this problem could be produced if the wrong python was in use. We can check which python runs as 'python' inside the env.

python -c "import sys; print(sys.executable)"
vex myenv python -c "import sys; print(sys.executable)"

After your other ticket concerning how python is not found using PATH, here is another ticket which has a good chance of relating to PATH somehow. I wonder if something is up with PATH on your system specifically? It might also be useful to see the PATH that is being searched (to find 'python') both inside and outside your virtualenv

python -c "import os; print(os.environ['PATH'])"
vex myenv python -c "import os; print(os.environ['PATH'])"

If you decide you want to help track this down then posting output from commands like these might be a good place to start.

[schlamar]

Ah, vex is not using the interpreter from the virtualenv. The issue might be that the interpreter is installed to ENV/Scripts/python instead of ENV/bin/python on Windows.

$ vex myenv python -c "import sys; print(sys.executable)"
c:\Python27\python.exe

The correct one should be:

$ vex myenv bash -c "python -c 'import sys; print(sys.executable)'"
c:\Users\schlaich\.virtualenvs\myenv\Scripts\python.exe

$PATH looks good.

$ vex myenv python -c "import os; print(os.environ['PATH'])"
C:\Users\schlaich\.virtualenvs\myenv\Scripts;c:\Users\schlaich\bin;...

[sashahart]

I don't know why you skipped the test of what VIRTUAL_ENV is getting set to. I put that first because it is the single most important detail to know about here.

vex was never directly using the interpreter from the virtualenv and it wasn't supposed to. vex isn't launching the interpreter it is run with, it is launching whatever process you tell it to whether that is python or not. The way virtualenv activate works is by setting environment variables. The way vex works is by setting environment variables. The difference is that vex is setting them in a new process. Other than that, the mechanism of operation should not be different. It has always been up to other things to use the environment variables appropriately. This cannot be an adequate explanation of whatever problems you are seeing.

'vex myenv bash' is not equivalent to 'vex myenv python'. The former is not a correction of the latter. 'vex myenv python' should be working if virtualenv is working. I am interested in the ultimate causes of your bugs. But I am completely unconcerned with what happens when you run bash on Windows because the number of people who run bash on Windows is probably not statistically distinguishable from 0.

I will look at this on a new, sane, bash-free Windows environment when I next get the chance. Until then I'm not interested in kludges that come without an adequate exploration of underlying causes, against a Windows environment that may not be sane.

[schlamar]

VIRTUAL_ENV is correct. Bash does not matter. The cause of this issue is http://bugs.python.org/issue15451 (modified PATH is ignored in subprocess.Popen).

Alternative solution to #16 would be to manually search the (new) PATH for the executable. But this is IMO much more unreadable/complicated than #16.

[sashahart]

Not using bash on Windows does matter because you are testing using software with different behavior from what almost anyone will use on Windows, which will differ from CMD.EXE or PowerShell around many things similar to PATH. At a more basic level, I already asked you once not to report bugs using bash.

If VIRTUAL_ENV is correct, and PATH isn't being honored, then vex has done its job as much as virtualenv activate does its job. I encourage you to try using virtualenv/activate and virtualenvwrapper in the exact same environment.

I am not going to rush into doing bad things to the vex code that I will regret later. shell=True was a mistake in the Popen API and it's bad news. Switching to shell=True is a kludge with many other implications. If there is a bug in subprocess.Popen, and it isn't going to be fixed upstream, there isn't really a good alternative, so the realistic fix will be to do away with the subprocess and invoke virtualenv implementation after an import. Which I've been planning to do, against a clean environment, when I have the free time.