Documentation issue or unclear bug with sitepackages feature

mbukatov commented 7 years ago

Description

When sitepackages feature is enabled in tox test enviroment (either via sitepackages = true or --sitepackages command line option), test code can't access modules installed in tox virtual enviroment, only modules from system site packages are available.

This contradicts my expectations based on the description of the feature:

Set to True if you want to create virtual environments that also have access to globally installed packages. default: False, meaning that virtualenvs will be created without inheriting the global site packages.

Moreover it makes this feature unusable for unit tests written in pytest.

OS and pip list output

I'm using tox (from distribution package) on Fedora 25:

$ rpm -q python3-tox
python3-tox-2.3.1-4.fc25.noarch
$ tox --version
2.3.1 imported from /usr/lib/python3.5/site-packages/tox/__init__.py

Full pip list output is too long (about 300 items), so here are just related modules:

$ pip list | egrep '(tox|virtualenv|^py\ |pluggy)'
pluggy (0.3.1)
py (1.4.32)
tox (2.3.1)
virtualenv (15.0.3)
virtualenv-clone (0.2.6)
virtualenvwrapper (4.7.1)

Setup for reproducer

This is a minimal reproducer setup, tox configuration used doesn't make much sense in this case as the only purpose is to show where the problem is.

Let's create a simple python project:

$ ls reproducer
const.py  __init__.py
$ cat reproducer/const.py 
ONE = 1
TWO = 2

With setup.py file:

$ cat setup.py 
from setuptools import setup, find_packages
setup(name='reproducer', version='0.0.1', packages=find_packages(exclude=['tests']))

With unit tests written in pytest framework:

$ cat tests/test_reproducer.py 
import reproducer.const

def test_one():
    assert reproducer.const.ONE == 1

def test_two():
    assert reproducer.const.TWO == 2

Then we add tox.ini file to connect the pieces together:

$ cat tox.ini 
[testenv]
deps = pytest
sitepackages = true
commands = py.test tests

So that we can run the unit tests via tox.

Here is a repository with such setup: https://github.com/mbukatov/tox-sitepackages-reproducer

Actual Results

When I run tox, it doesn't work:

$ tox
GLOB sdist-make: /home/martin/projects/tox-sitepackages-reproducer/setup.py
python create: /home/martin/projects/tox-sitepackages-reproducer/.tox/python
python installdeps: pytest
python inst: /home/martin/projects/tox-sitepackages-reproducer/.tox/dist/reproducer-0.0.1.zip
python installed: alabaster==0.7.9,apipkg==1.4,appdirs==1.4.1,args==0.1.0,astroid==1.4.5,Babel==2.3.4,beautifulsoup4==4.5.3,blivet==2.1.6,certifi==2015.4.28,chardet==2.3.0,click==6.7,clint==0.5.1,colorama==0.3.7,copr==1.76,copr-cli==1.58,coverage==4.2,cssselect==0.9.2,cupshelpers==1.0,cycler==0.10.0,decorator==4.0.11,distro==1.0.2,dnf-etckeeper==0.0.0,dnspython==1.15.0,docutils==0.13.1,execnet==1.4.1,file-magic==0.3.0,flake8==2.5.5,fros==1.1,funcsigs==1.0.2,git-review==1.25.0,html5lib==0.999,humanize==0.5.1,imagesize==0.7.1,iniparse==0.4,initial-setup==0.3.43,IPy==0.81,ipython==3.2.1,isort==4.2.5,javapackages==4.7.0,jenkins-job-builder==1.4.0,Jinja2==2.8.1,jsonpointer==1.10,jsonschema==2.5.1,langtable==0.0.36,lazy-object-proxy==1.2.1,livereload==2.4.1,logilab-common==0.63.2,lxml==3.7.2,Markdown==2.6.8,MarkupSafe==0.23,marshmallow==2.10.5,matplotlib==1.5.2rc2,mccabe==0.2.1,mistune==0.7.3,mkdocs==0.16.1,mkdocs-bootstrap==0.1.1,mkdocs-bootswatch==0.4.0,mkdocs-material==0.2.1,mock==2.0.0,multi-key-dict==2.0.3,ntplib==0.3.3,numpy==1.11.2,ordered-set==2.0.0,packaging==16.8,path.py==5.2,pbr==1.10.0,pep8==1.6.2,pexpect==4.2.1,pid==2.0.1,Pillow==3.4.2,pkginfo==1.3.2,pluggy==0.3.1,plumbum==1.6.0,progress==1.2,ptyprocess==0.5.1,pwquality==1.3.0,py==1.4.32,pycrypto==2.6.1,pycups==1.9.72,pycurl==7.43.0,pyenchant==1.6.8,pyflakes==1.2.3,Pygments==2.1.3,pygobject==3.22.0,pygpgme==0.3,pyinotify==0.9.6,PyIscsi==1.0,pykickstart==2.32,pylint==1.6.4,pyp2rpm==3.2.1,pyparsing==2.1.10,pyparted==3.10.7,PySocks==1.5.6,pytest==2.9.2,pytest-ansible-playbook==0.3.0,pytest-cache==1.0,pytest-capturelog==0.7,pytest-flakes==1.0.1,pytest-pep8==1.0.6,python-augeas==0.5.0,python-bugzilla==1.2.2,python-dateutil==2.6.0,python-dmidecode==3.12.2,python-etcd==0.4.3,python-jenkins==0.4.12,python-meh==0.43,pytz==2016.6.1,pyudev==0.21.0,PyXB==1.2.4,PyYAML==3.11,pyzmq==15.3.0,rdflib==4.1.2,reproducer==0.0.1,requests==2.10.0,requests-file==1.4,requests-ftp==0.3.1,requests-toolbelt==0.7.0,rpm-python==4.13.0,rpmconf==1.0.18,rpyc==3.3.0,selenium==2.53.5,seobject==0.1,sepolicy==1.1,simplegeneric==0.8.1,simplejson==3.10.0,six==1.10.0,slip==0.6.4,slip.dbus==0.6.4,snowballstemmer==1.2.1,sosreport==3.2.0a1,spec2scl==1.1.3,Sphinx==1.4.8,sphinx-rtd-theme==0.1.9,SSSDConfig==1.14.2,systemd-python==232,tornado==4.4.2,tox==2.3.1,twine==1.7.4,urllib3==1.15.1,virtualenv==15.0.3,virtualenv-api==2.1.9,wrapt==1.10.8,yamllint==1.6.0
python runtests: PYTHONHASHSEED='822135775'
python runtests: commands[0] | py.test tests
WARNING:test command found but not installed in testenv
  cmd: /usr/bin/py.test
  env: /home/martin/projects/tox-sitepackages-reproducer/.tox/python
Maybe you forgot to specify a dependency? See also the whitelist_externals envconfig setting.
==================================================================== test session starts =====================================================================
platform linux2 -- Python 2.7.13, pytest-2.9.2, py-1.4.32, pluggy-0.3.1
rootdir: /home/martin/projects/tox-sitepackages-reproducer, inifile: 
plugins: flakes-1.0.1, pep8-1.0.6
collected 0 items / 1 errors 

=========================================================================== ERRORS ===========================================================================
_________________________________________________________ ERROR collecting tests/test_reproducer.py __________________________________________________________
tests/test_reproducer.py:1: in <module>
    import reproducer.const
E   ImportError: No module named reproducer.const
================================================================== 1 error in 0.01 seconds ===================================================================
ERROR: InvocationError: '/usr/bin/py.test tests'
__________________________________________________________________________ summary ___________________________________________________________________________
ERROR:   python: commands failed

Note that test code executed by pytest can't access python modules which are not installed (either in system site packages, user site packages or virtualenv).

What happened? Test code can't import reproducer.const module, even though it's properly installed in tox virtualenv:

$ source .tox/python/bin/activate
$ python
Python 3.5.2 (default, Sep 14 2016, 11:28:32) 
[GCC 6.2.1 20160901 (Red Hat 6.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import reproducer.const
>>> reproducer.const
<module 'reproducer.const' from '/home/martin/projects/tox-sitepackages-reproducer/reproducer/const.py'>

Also note that system site packages feature has been passed to virtualenv properly, as I can import python module from system site packages without any problem:

>>> import bugzilla
>>> bugzilla
<module 'bugzilla' from '/usr/lib/python3.5/site-packages/bugzilla/__init__.py'>

This means that virtualenv has been created correctly and the problem is in virtualenv usage.

Expected resuts

Tox executes tests without any problem, so that test code can access both python modules installed in virtualenv and system site packages.

Additional details

When I drop sitepackages = true from tox.ini and recreate virtual environment, it works:

$ tox --recreate
GLOB sdist-make: /home/martin/projects/tox-sitepackages-reproducer/setup.py
python recreate: /home/martin/projects/tox-sitepackages-reproducer/.tox/python
python installdeps: pytest
python inst: /home/martin/projects/tox-sitepackages-reproducer/.tox/dist/reproducer-0.0.1.zip
python installed: appdirs==1.4.1,packaging==16.8,py==1.4.32,pyparsing==2.1.10,pytest==3.0.6,reproducer==0.0.1,six==1.10.0
python runtests: PYTHONHASHSEED='3386328494'
python runtests: commands[0] | py.test tests
==================================================================== test session starts =====================================================================
platform linux -- Python 3.5.2, pytest-3.0.6, py-1.4.32, pluggy-0.4.0
rootdir: /home/martin/projects/tox-sitepackages-reproducer, inifile: 
collected 2 items 

tests/test_reproducer.py ..

================================================================== 2 passed in 0.01 seconds ==================================================================
__________________________________________________________________________ summary ___________________________________________________________________________
  python: commands succeeded
  congratulations :)

This shows that issue demonstrated above is caused by sitepackages = true and not by something else.

mbukatov commented 7 years ago

This issue may be related to https://github.com/tox-dev/tox/issues/80

obestwalter commented 7 years ago

@mbukatov thanks for the detailed report.

mbukatov commented 7 years ago

Looking at the tox output (see my report above), I noticed a possible problem in my setup:

WARNING:test command found but not installed in testenv
  cmd: /usr/bin/py.test
  env: /home/martin/projects/tox-sitepackages-reproducer/.tox/python
Maybe you forgot to specify a dependency? See also the whitelist_externals envconfig setting.

So I changed the way pytest is invoked like this:

-commands = py.test tests
+commands = python -m pytest tests

Retried:

$ tox --recreate --sitepackages
GLOB sdist-make: /home/martin/projects/tox-sitepackages-reproducer/setup.py
python recreate: /home/martin/projects/tox-sitepackages-reproducer/.tox/python
python installdeps: pytest
python inst: /home/martin/projects/tox-sitepackages-reproducer/.tox/dist/reproducer-0.0.1.zip
python installed: alabaster==0.7.9,apipkg==1.4,appdirs==1.4.2,args==0.1.0,astroid==1.4.5,Babel==2.3.4,beautifulsoup4==4.5.3,blivet==2.1.6,certifi==2015.4.28,chardet==2.3.0,click==6.7,clint==0.5.1,colorama==0.3.7,copr==1.76,copr-cli==1.58,coverage==4.2,cssselect==0.9.2,cupshelpers==1.0,cycler==0.10.0,decorator==4.0.11,distro==1.0.2,dnf-etckeeper==0.0.0,dnspython==1.15.0,docutils==0.13.1,execnet==1.4.1,file-magic==0.3.0,flake8==2.5.5,fros==1.1,funcsigs==1.0.2,git-review==1.25.0,html5lib==0.999,humanize==0.5.1,imagesize==0.7.1,iniparse==0.4,initial-setup==0.3.43,IPy==0.81,ipython==3.2.1,isort==4.2.5,javapackages==4.7.0,jenkins-job-builder==1.4.0,Jinja2==2.8.1,jsonpointer==1.10,jsonschema==2.5.1,langtable==0.0.36,lazy-object-proxy==1.2.1,livereload==2.4.1,logilab-common==0.63.2,lxml==3.7.2,Markdown==2.6.8,MarkupSafe==0.23,marshmallow==2.10.5,matplotlib==1.5.2rc2,mccabe==0.2.1,mistune==0.7.3,mkdocs==0.16.1,mkdocs-bootstrap==0.1.1,mkdocs-bootswatch==0.4.0,mkdocs-material==0.2.1,mock==2.0.0,multi-key-dict==2.0.3,ntplib==0.3.3,numpy==1.11.2,ordered-set==2.0.0,packaging==16.8,path.py==5.2,pbr==1.10.0,pep8==1.6.2,pexpect==4.2.1,pid==2.0.1,Pillow==3.4.2,pkginfo==1.3.2,pluggy==0.3.1,plumbum==1.6.0,progress==1.2,ptyprocess==0.5.1,pwquality==1.3.0,py==1.4.32,pycrypto==2.6.1,pycups==1.9.72,pycurl==7.43.0,pyenchant==1.6.8,pyflakes==1.2.3,Pygments==2.1.3,pygobject==3.22.0,pygpgme==0.3,pyinotify==0.9.6,PyIscsi==1.0,pykickstart==2.32,pylint==1.6.4,pyp2rpm==3.2.1,pyparsing==2.1.10,pyparted==3.10.7,PySocks==1.5.6,pytest==2.9.2,pytest-ansible-playbook==0.3.0,pytest-cache==1.0,pytest-capturelog==0.7,pytest-flakes==1.0.1,pytest-pep8==1.0.6,python-augeas==0.5.0,python-bugzilla==1.2.2,python-dateutil==2.6.0,python-dmidecode==3.12.2,python-etcd==0.4.3,python-jenkins==0.4.12,python-meh==0.43,pytz==2016.6.1,pyudev==0.21.0,PyXB==1.2.4,PyYAML==3.11,pyzmq==15.3.0,rdflib==4.1.2,reproducer==0.0.1,requests==2.10.0,requests-file==1.4,requests-ftp==0.3.1,requests-toolbelt==0.7.0,rpm-python==4.13.0,rpmconf==1.0.18,rpyc==3.3.0,selenium==2.53.5,seobject==0.1,sepolicy==1.1,simplegeneric==0.8.1,simplejson==3.10.0,six==1.10.0,slip==0.6.4,slip.dbus==0.6.4,snowballstemmer==1.2.1,sos==3.2,spec2scl==1.1.3,Sphinx==1.4.8,sphinx-rtd-theme==0.1.9,SSSDConfig==1.14.2,systemd-python==232,tornado==4.4.2,tox==2.3.1,twine==1.7.4,urllib3==1.15.1,virtualenv==15.0.3,virtualenv-api==2.1.9,wrapt==1.10.8,yamllint==1.6.0
python runtests: PYTHONHASHSEED='3458891052'
python runtests: commands[0] | python -m pytest tests
==================================================================================================================================================== test session starts =====================================================================================================================================================
platform linux -- Python 3.5.2, pytest-2.9.2, py-1.4.32, pluggy-0.3.1
rootdir: /home/martin/projects/tox-sitepackages-reproducer, inifile: 
plugins: pep8-1.0.6, flakes-1.0.1, capturelog-0.7, ansible-playbook-0.3.0
collected 2 items 

tests/test_reproducer.py ..

======================================================================================================================================== 2 passed, 1 pytest-warnings in 0.01 seconds =========================================================================================================================================
__________________________________________________________________________________________________________________________________________________________ summary ___________________________________________________________________________________________________________________________________________________________
  python: commands succeeded
  congratulations :)

And noticed that it works.

~~Which means that this is a configuration mistake on my side and not a tox bug. I should have inspected the warning.~~

That said, maybe it would be a good idea to remove examples like that from tox documentation, because invoking pytest via py.test executable directly breaks --sitepackages feature. For example, the front page of tox project shows the same setup I initially used (and which caused this problem).

But looking at pytest documentation about invoking pytest via python -m pytest:

This is almost equivalent to invoking the command line script pytest [...] directly, except that python will also add the current directory to sys.path.

I realized that using pythom -m pytest fixes my reproducer because reproducer module was imported from current directory directly and that I need to investigate further to figure out if this is a tox issue or not.

mbukatov commented 7 years ago

To investigate further, I added 2 python modules into install_requires of setup.py: one which I have installed in system site packages from rpm package (lxml) and other which I don't have installed at all (kernelconfig), so that the first one should be imported from system site packages and the other from tox managed python virtualenv. Moreover I added 2 test cases which checks this assumption:

def test_import_lxml():
    import lxml
    assert lxml.__file__.startswith("/usr/lib64")

def test_import_kernelconfig():
    import kernelconfig
    assert "/.tox/python/lib" in kernelconfig.__file__

And it works as expected:

$ tox --recreate
GLOB sdist-make: /home/martin/projects/tox-sitepackages-reproducer/setup.py
python create: /home/martin/projects/tox-sitepackages-reproducer/.tox/python
python installdeps: pytest
python inst: /home/martin/projects/tox-sitepackages-reproducer/.tox/dist/reproducer-0.0.1.zip
python installed: alabaster==0.7.9,apipkg==1.4,appdirs==1.4.2,args==0.1.0,astroid==1.4.5,Babel==2.3.4,beautifulsoup4==4.5.3,blivet==2.1.6,certifi==2015.4.28,chardet==2.3.0,click==6.7,clint==0.5.1,colorama==0.3.7,copr==1.76,copr-cli==1.58,coverage==4.2,cssselect==0.9.2,cupshelpers==1.0,cycler==0.10.0,decorator==4.0.11,distro==1.0.2,dnf-etckeeper==0.0.0,dnspython==1.15.0,docutils==0.13.1,execnet==1.4.1,file-magic==0.3.0,flake8==2.5.5,fros==1.1,funcsigs==1.0.2,git-review==1.25.0,html5lib==0.999,humanize==0.5.1,imagesize==0.7.1,iniparse==0.4,initial-setup==0.3.43,IPy==0.81,ipython==3.2.1,isort==4.2.5,javapackages==4.7.0,jenkins-job-builder==1.4.0,Jinja2==2.8.1,jsonpointer==1.10,jsonschema==2.5.1,kernelconfig==0.3.2,langtable==0.0.36,lazy-object-proxy==1.2.1,livereload==2.4.1,logilab-common==0.63.2,lxml==3.7.2,Markdown==2.6.8,MarkupSafe==0.23,marshmallow==2.10.5,matplotlib==1.5.2rc2,mccabe==0.2.1,mistune==0.7.3,mkdocs==0.16.1,mkdocs-bootstrap==0.1.1,mkdocs-bootswatch==0.4.0,mkdocs-material==0.2.1,mock==2.0.0,multi-key-dict==2.0.3,ntplib==0.3.3,numpy==1.11.2,ordered-set==2.0.0,packaging==16.8,path.py==5.2,pbr==1.10.0,pep8==1.6.2,pexpect==4.2.1,pid==2.0.1,Pillow==3.4.2,pkginfo==1.3.2,pluggy==0.3.1,plumbum==1.6.0,progress==1.2,ptyprocess==0.5.1,pwquality==1.3.0,py==1.4.32,pycrypto==2.6.1,pycups==1.9.72,pycurl==7.43.0,pyenchant==1.6.8,pyflakes==1.2.3,Pygments==2.1.3,pygobject==3.22.0,pygpgme==0.3,pyinotify==0.9.6,PyIscsi==1.0,pykickstart==2.32,pylint==1.6.4,pyp2rpm==3.2.1,pyparsing==2.1.10,pyparted==3.10.7,PySocks==1.5.6,pytest==2.9.2,pytest-ansible-playbook==0.3.0,pytest-cache==1.0,pytest-capturelog==0.7,pytest-flakes==1.0.1,pytest-pep8==1.0.6,python-augeas==0.5.0,python-bugzilla==1.2.2,python-dateutil==2.6.0,python-dmidecode==3.12.2,python-etcd==0.4.3,python-jenkins==0.4.12,python-meh==0.43,pytz==2016.6.1,pyudev==0.21.0,PyXB==1.2.4,PyYAML==3.11,pyzmq==15.3.0,rdflib==4.1.2,reproducer==0.0.1,requests==2.10.0,requests-file==1.4,requests-ftp==0.3.1,requests-toolbelt==0.7.0,rpm-python==4.13.0,rpmconf==1.0.18,rpyc==3.3.0,selenium==2.53.5,seobject==0.1,sepolicy==1.1,simplegeneric==0.8.1,simplejson==3.10.0,six==1.10.0,slip==0.6.4,slip.dbus==0.6.4,snowballstemmer==1.2.1,sos==3.2,spec2scl==1.1.3,Sphinx==1.4.8,sphinx-rtd-theme==0.1.9,SSSDConfig==1.14.2,systemd-python==232,tornado==4.4.2,tox==2.3.1,twine==1.7.4,urllib3==1.15.1,virtualenv==15.0.3,virtualenv-api==2.1.9,wrapt==1.10.8,yamllint==1.6.0
python runtests: PYTHONHASHSEED='3836285494'
python runtests: commands[0] | python -m pytest tests
====================================================== test session starts =======================================================
platform linux -- Python 3.5.2, pytest-2.9.2, py-1.4.32, pluggy-0.3.1
rootdir: /home/martin/projects/tox-sitepackages-reproducer, inifile: 
plugins: pep8-1.0.6, flakes-1.0.1, capturelog-0.7, ansible-playbook-0.3.0
collected 4 items 

tests/test_reproducer.py ....

========================================== 4 passed, 1 pytest-warnings in 0.02 seconds ===========================================
____________________________________________________________ summary _____________________________________________________________
  python: commands succeeded
  congratulations :)

Which means that this is rather a configuration mistake on my side and not a tox bug after all. I should have inspected the warning.

mbukatov commented 7 years ago

Would it make sense to mention this problem (that invoking pytest by command like executable breaks sitapackages feature) in description of sitepackages feature or in py.test and tox section?

obestwalter commented 7 years ago

@mbukatov many thanks for the effort you put into investiagating this. Very appreciated.

Would it make sense to mention this problem (that invoking pytest by command like executable breaks sitapackages feature) in description of sitepackages feature or in py.test and tox section?

I have to admit that I still don't quite understand the problem. I will play around with your reproducer to gather an understanding what needs to be either documented or fixed. I will rename the issue and mark it accirdingly to discuss this here further.

obestwalter commented 7 years ago

I closed https://github.com/tox-dev/tox/issues/80 as the paste was gone and the OP did not answer anymore (which is understandable after more than four years :)). If this is is still a problem that hits users in a few corner cases, we can track this here.

funkyfuture commented 7 years ago

thanks @mbukatov, this is certainly worth to be elaborated in the docs. i spent several hours to deal with a complicated dependency situation and ended up with not using tox. this is exactly what i wanted to do, but 'obviously' the sitepackages option seemed to behave weird.

kenahoo commented 7 years ago

I've encountered what I think is the same problem, I opened a Stack Overflow question about it:

https://stackoverflow.com/questions/44145898/tox-and-lib-and-lib64-and-site-packages

In short, the sitepackages setting seems to behave exactly opposite from what the docs say it does.

obestwalter commented 7 years ago

@kenahoo huh - sitepackages is something I never needed yet, so couldn't say. If that is really the case, we have a bug that would be very likely easy to fix. Could anyone using this chip in?

Xeite commented 6 years ago

Any progress for this issue? Hope to fix it up to make faster tox based container image.

obestwalter commented 6 years ago

No, not that I know of.

tbondarchuk commented 6 years ago

Seems to be working OK for me on the docker container with pytest preinstalled via pip and invoked via python -m pytest:

[tox]
envlist = py36
usedevelop = True

[testenv]
sitepackages = True
commands = python -m pytest

obestwalter commented 5 years ago

I just read through this thing again and I had missed a few important bits. So it is clear now that this is a configuration problem and not a bug :man_facepalming:

This is the important hint:

WARNING:test command found but not installed in testenv
  cmd: /usr/bin/py.test
  env: /home/martin/projects/tox-sitepackages-reproducer/.tox/python
Maybe you forgot to specify a dependency? See also the whitelist_externals envconfig setting.

This means that pytest is also installed globally and the the wrong pytest is being used.

@mbukatov

Which means that this is rather a configuration mistake on my side and not a tox bug after all. I should have inspected the warning.

I had completely overread that you had come to that conclusion yourself the first time around - sorry :)

The recommendation by @aliusmiles and others here is spot on to use the python -m <tool> invocation form (because this will use {envpython}) or {envbindir}/<tool> might work if the tool does not implement __main__.py to make sure that the tools from the virutalenv are sued rather then from the global environment.

This is a perfect example why sitepackages should be avoided if at all possible. If it is not possible the surrounding environment should at least not have tools installed that are used in venvs as part of tests, etc. (e.g. pytest, coverage, ...).

I opened #1128 now to clarify this and if that is merged we can close this I think.

update: if implemented #1127 will avoid the problem altogether by enforcing whitelisting for all commands that are not in the current environment.

tox-dev / tox