tox package broken

[RonnyPfannschmidt]

Issue description

when running the nix-os shipped tox on one of my packages, it breaks

$ tox -r
GLOB sdist-make: /home/ronny/Projects/tools/Sentaku/setup.py
py27 recreate: /home/ronny/Projects/tools/Sentaku/.tox/py27
py27 installdeps: pytest, attrs, cached_property
py27 inst: /home/ronny/Projects/tools/Sentaku/.tox/dist/sentaku-0.1.dev27+n05ce7c3.zip
py27 installed: You are using pip version 7.1.2, however version 8.1.1 is available.,You should consider upgrading via the 'pip install --upgrade pip' command.,attrs==15.2.0,cached-property==1.3.0,pluggy==0.3.1,py==1.4.31,pytest==2.9.1,sentaku==0.1.dev27+n05ce7c3,tox==2.3.1,virtualenv==13.1.2,wheel==0.24.0
py27 runtests: PYTHONHASHSEED='1866977423'
py27 runtests: commands[0] | py.test
Traceback (most recent call last):
  File ".tox/py27/bin/py.test", line 7, in <module>
    from pytest import main
  File "/home/ronny/Projects/tools/Sentaku/.tox/py27/lib/python2.7/site-packages/pytest.py", line 21, in <module>
    from _pytest.config import (
  File "/home/ronny/Projects/tools/Sentaku/.tox/py27/lib/python2.7/site-packages/_pytest/config.py", line 8, in <module>
    import py
ImportError: No module named py
ERROR: InvocationError: '/home/ronny/Projects/tools/Sentaku/.tox/py27/bin/py.test'
py35 recreate: /home/ronny/Projects/tools/Sentaku/.tox/py35
py35 installdeps: pytest, attrs, cached_property
py35 inst: /home/ronny/Projects/tools/Sentaku/.tox/dist/sentaku-0.1.dev27+n05ce7c3.zip
py35 installed: You are using pip version 7.1.2, however version 8.1.1 is available.,You should consider upgrading via the 'pip install --upgrade pip' command.,attrs==15.2.0,cached-property==1.3.0,pluggy==0.3.1,py==1.4.31,pytest==2.9.1,sentaku==0.1.dev27+n05ce7c3,tox==2.3.1,virtualenv==13.1.2,wheel==0.24.0
py35 runtests: PYTHONHASHSEED='1866977423'
py35 runtests: commands[0] | py.test
Traceback (most recent call last):
  File ".tox/py35/bin/py.test", line 7, in <module>
    from pytest import main
  File "/home/ronny/Projects/tools/Sentaku/.tox/py35/lib/python3.5/site-packages/pytest.py", line 21, in <module>
    from _pytest.config import (
  File "/home/ronny/Projects/tools/Sentaku/.tox/py35/lib/python3.5/site-packages/_pytest/config.py", line 8, in <module>
    import py
ImportError: No module named 'py'
ERROR: InvocationError: '/home/ronny/Projects/tools/Sentaku/.tox/py35/bin/py.test'
codechecks recreate: /home/ronny/Projects/tools/Sentaku/.tox/codechecks
codechecks installdeps: flake8
codechecks installed: You are using pip version 7.1.2, however version 8.1.1 is available.,You should consider upgrading via the 'pip install --upgrade pip' command.,flake8==2.5.4,mccabe==0.4.0,pep8==1.7.0,pluggy==0.3.1,py==1.4.31,pyflakes==1.0.0,tox==2.3.1,virtualenv==13.1.2,wheel==0.24.0
codechecks runtests: PYTHONHASHSEED='1866977423'
codechecks runtests: commands[0] | flake8 src testing
_____________________________________________________________________________________________________ summary _____________________________________________________________________________________________________
ERROR:   py27: commands failed
ERROR:   py35: commands failed
  codechecks: commands succeeded

Steps to reproduce

clone https://github.com/RonnyPfannschmidt/sentaku get tox run tox on the project

Technical details

the missing py package is a python dependency of the pytest package

$ nixos-version
16.03.678.2597f52 (Emu)

$ nix-env --version
nix-env (Nix) 1.11.2

$ nix-instantiate --eval '<nixpkgs>' -A lib.nixpkgsVersion
"16.03.678.2597f52"

* System: (NixOS: `nixos-version`, Ubuntu/Fedora: `lsb_release -a`, ...)
* Nix version: (run `nix-env --version`)
* Nixpkgs version: (run `nix-instantiate --eval '<nixpkgs>' -A lib.nixpkgsVersion`)

[domenkozar]

Quite busy these days, maybe @FRidh has an idea. Otherwise I'll peek at it in following days.

[FRidh]

I don't use Tox myself but I guess TOX_TESTENV_PASSENV should help you. http://tox.readthedocs.io/en/latest/config.html?highlight=tox_testenv_passenv

Right now PYTHONPATH contains the paths to all installed dependencies, which includes tox and thus py. I suppose passing to TOX_TESTENV_PASSENV a PYTHONPATH should do the trick?

In this case I pass the current PYTHONPATH

TOX_TESTENV_PASSENV="PYTHONPATH" tox

and that seems to work. I guess though it would be better if PYTHONPATH was unset so it would install py it self, but it doesn't seem to do that.

[FRidh]

and I was using nix-shell with nix-shell -p python27Packages.tox python35Packages.python.

[RonnyPfannschmidt]

shouldnt tox in nix be updated to do that then?

i can verify that the TOX_TESTENV_PASSENV="PYTHONPATH" works

but in normal tox usage thats not how it should go

[FRidh]

If we think this approach is general enough and should work for basically all cases, then yes. Like I said, I never use tox myself so I am not sure whether this is a good approach or not. We can at least document it (see also #7368).

[RonnyPfannschmidt]

after taking a look at the wrapper, im reasonably certain tox should not passenv the pythonpath since it makes new virtualenvs that should just work

my understanding is, that tox in nix breaks because pip invocations will see tox pythonpath instead of the real virtualenv pythonpath

as far as i can tell this is impossible to fix without changing completely how nix composes the pythonpath

[RonnyPfannschmidt]

however a tox bugfix can be done, namely applying passenv to pip invocations

[FRidh]

@RonnyPfannschmidt you asked whether it is possible to create Python envs without setting PYTHONPATH, but e.g. using pth files. We used to have that until https://github.com/NixOS/nixpkgs/pull/11086/, if I remember correctly, with the recursivePthLoader.

The recursivePthLoader [1] iterates over sys.path and adds directories specified in the pth files to the end of PYTHONPATH. It's actually still there, only being used in virtualenv, although I don't see just yet how it is used there.

virtualenv does seem to work correctly with regards to pip. When I create a virtualenv it installs pip 7.1.2 and is being used. I think this is the behaviour we also want for tox, to use the pip that is installed in the virtualenv.

[1] https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/python-modules/recursive-pth-loader/default.nix

[RonnyPfannschmidt]

but the pythonpath exposed by the tox executable in nix pushed a pylib install into the pip invocation but then the test call (where tox does more isolation) suddenly those packages are not there, thus the builds break

the basci issue is, that pythonpath is a rather bad mechanism to transfer python packages as per system pacakges, and pth files generated work better

wheels are bascially just pythons variant of .nar tht doe get unpacked into a python site package folder

when usign them in nix, nix should ensure that the executable scripts and the python path gets constructed in a non-obstrusive manner (i.e. without smearing into the environment variables)

[RonnyPfannschmidt]

ok, a writedown of what happens and why the interaction tox<>virtualenv breaks on nix

1. tox binary starts via wrapper, pythonpath is set
2. tox runs pip without isolation, pip sees nix packages due to the tox virtualenv packages and will not "reinstall" them
3. tox ruins test commands with environment isolation, the nix pacakges used by tox are not there, things break

in order to properly fix, the tox script, should instead of being a shell wrapper, use python, add directly to site and not touch the environment

also all "python executable" packages that do not invoke sub-processes should probably do the same

[domenkozar]

I think this is an issue for upstream. Why doesn't tox isolate PYTHONPATH?

[RonnyPfannschmidt]

because on a normal system that is not necessary per se but it certainly shows an issue with nix as well, a virtuelanv created by nix should be immune forwards invocation by a nix based command

[domenkozar]

virtualenv and tox don't isolate PYTHONPATH. How is this Nix fault? :)

[RonnyPfannschmidt]

nix creates a wrapper that changes it ^^

[domenkozar]

Nix exposes libraries to python via PYTHONPATH so tox can actually import packages. This is 101 how Python works. Nothing magic or unexpected. This is really an issue for upstream.

[RonnyPfannschmidt]

the basic implication is, that any nix installed python tool that invokes things in my path can break them in case of virtualenv unless all python tools use environment isolation in all cases

also this implies that if a python3 tool invokes a python2 tool we can get funny effects unless all tools learn very specific isolation thats based on how nix splits and constructs python installations

[RonnyPfannschmidt]

17749 fixed this for the general case