Installed package shadowed by project package in some cases (compiled extensions?)

[jlorieau]

First, I want to thank you for making and sharing a very useful tool.

From my testing, I've found that the python package installed by tox in each virtual environment is likely not used for the tests. Before describing the problem, I will say that I believe this bug only effectively impacts python programs that depend on compiled extensions or those that require special post-processing during installation.

Most python software is packaged as follows:

project/project/__init__.py
project/project/tests
project/docs
project/setup.py

Alternatively, the tests folder may be in the root folder. Tests are typically discovered and run from the root project folder.

When tox runs, it creates a series of virtual environments under .tox and installs project into each. If the project includes extensions, installation typically includes the compilation of the extensions. However, it appears that tox then runs the tests from the root project folder. In this case, the local copy of the package module obscures the installed copy, and it will be loaded in python scripts (import project) instead of the package installed in the virtual environment. I made this conclusion because the project appears to be successfully installed in each virtual environment in the .tox directory, yet the extensions have to be built in place in the root project folder for the tests to run. Essentially, the following has to be added to the tox.ini (using nosetests, for example):

commands=
  python setup.py clean --all build_ext --force --inplace
  nosetests

I'm reporting this issue because testing against the local copy of the package may introduce unexpected side-effects for other users requiring post-processing of their python packages. I can confirm that this is a problem in python 2.7 and python 3.6--even though loading of relative and absolute paths should not be an issue in the later.

My expectation, as a user, is that tox use the installed package to run the tests.

I couldn't find a simple fix to the problem--though I suspect that one exists--and I suggest at least adding a special note in the documentation about this behavior.

[RonnyPfannschmidt]

well, you are running into expected python behaviour tht tox cannot change

one of the reasons why py.test suggests externlized trees is the import mechanism of python that just makes it necessary to pull the package in the worktree into the pythonpath if you want to run the tests in the worktree

unittest using packages work around this by invoking the testrunners exclusively using the pythonpath

[jlorieau]

Thanks for the note. I couldn't find a note on this in the documentation, and I suggest adding one. All of the python packages I looked at that use tox may be subject to this problem, including major projects like django, numpy and scipy.

Following your recommendation to look up py.test, they referenced the following blog, which describes the issue nicely.

https://blog.ionelmc.ro/2014/05/25/python-packaging/#the-structure

[gaborbernat]

Well what you're suggesting is not definitely the expected behavior, as you many not want to package your tests, for example. If you don't want to install the package in the current venv just pass skip_install = true for that.

[obestwalter]

I don't understand the corner case described here, so could someone point to some existing project where this problem occurs? Is this a problem with nosetest?

@RonnyPfannschmidt what do you mean is expected behaviour? If a project is structured the way it was described (which is the way how the tox project is structured also) then I still expect that tox tests my installed package and not the project code otherwise I would not need tox to build my package for me and install it into the virtualenv. So could you explain what you mean?

[stlehmann]

@jlorieau I experience the same issue with a package of mine that compiles a shared library and puts it in the .tox/py36/lib/python3.6/site-packages/mypackage directory of the virtual environment on installation.

When using pytest with tox the shared library could not be found. The reason was the same as yours. Pytest used my local package directory structure to import mypackage from there in favor to the installed version in .tox/py36/lib/python3.6/site-packages/mypackage. When I look at my project structure this makes perfect sense as @RonnyPfannschmidt said. So when starting pytest from the project root directory the interpreter will always import the local project as local packages take priority to packages listed in PYTHONPATH.

mypackage/
    mypackage/
        __init__.py
    tests
    setup.py
    tox.ini

This can be fixed by changing the current directory to tests. This way the package can't be accessed from the current directory anymore. Then the interpreter searches the path and finds our installed one. However for some reason this only works when using the discover command instead of pytest. Would like to know why...

[testenv]
command = discover
deps = discover
changedir = tests

[RonnyPfannschmidt]

@obestwalter tox is very explicitly not structured like the example

in the example py.test has no choice but to cause a shadow, in tox we took great care to have the tests end up with a distinct sys.path entry

[obestwalter]

Are we talking about the same thing?

I am referring to:

Most python software is packaged as follows:

project/project/__init__.py
project/project/tests
project/docs
project/setup.py

Alternatively, the tests folder may be in the root folder. Tests are typically discovered and run from the root project folder.

Please note the mention of the "alternative" layout (which to me is actually the standard, which is recommended almost everywhere nowadays and the standard cookiecutter pypackage layout). Tox is using exactly this "alternative" layout. If I understad @jlorieau thinks both layout styles cause the same kind of problem. There is also Testing & Packaging that seems to identify the same problem (if I understand it correctly - I really have a bit of trouble to get my head around it in my post EuroPython daze :)). I guess I have to make some experiments and find out exactly how all this plays together as all my current knowledge about this is my own experience where I never had any problems with this (but maybe because in my projects it doesn't matter what I test against - I dunno).

[obestwalter]

Digging around in the issues I found this which I closed on the basis of the proposed workaround of using changedir (admittedly without really understanding what was going on). I have to do some homework in that area, I guess.

[jlorieau]

I'm writing with an update on my findings. My project (with compiled extensions) has unittests/pytests and doctests. The unittests/pytests do not import the local copy after refactoring these to a directory above the source code.

project/project/__init__.py
project/tests
project/docs
project/setup.py

Previously, my tests directories were within each submodule of the project/project directory. I did not need to place the 'changedir` option to get this to work.

However, I still run into this problem with my doctests. For doctests, I have to compile my extensions for the Python2 or 3 interpreters I am running tox against because it is loading the local copy of my project.

From my understanding, this problem occurs since python gives precedence to the local directory in loading packages, and the local project folder has the same name as the one installed in the tox virtualenv. This problem is partly rectified by placing the tests directory in the root project folder. To avoid importing the project altogether, the source folder (project/project) can be renamed (project/src or project/_project). Tox itself doesn't implement this solution, but pytest does.

I was thinking about this problem, and I believe a complete solution is fairly straightforward. I would create a symbolic link to the project/project and project/tests directories to some temporary folder(s) that do not have a name clash in the tox virtualenv, then change to that directory and discover those tests.

[obestwalter]

Hi @jlorieau, thanks for shedding more light on this. Maybe that proposed src layout (see blog posts) is the better way after all to avoid these problems in all possible scenarios.

[capdaha]

I've encountered the same problem with Tox and Pytest and found out that it is not an issue with Tox, but the way Pytest works. Here is the link to the conventions for Python test discovery by Pytest and two common test layouts. It suggests two test layouts for a project which solve the issue: 1) Tests outside of a module and without init files.

setup.py
mypkg/
    __init__.py
    app.py
    view.py
tests/
    test_app.py
    test_view.py
    ...

2) A src layout with init files in tests as @jlorieau's solution.

setup.py
src/
    mypkg/
        __init__.py
        app.py
        view.py
tests/
    __init__.py
    foo/
        __init__.py
        test_view.py
    bar/
        __init__.py
        test_view.py

With these layouts you can just run your local tests.

The third solution is to install tests with a package and run installed tests - execute Pytest with pyargs argument: pytest --pyargs mypkg.

[gaborbernat]

This is sadly outside of the control of tox, the src layout can help.