Allow tests to be run out of the source tree

[stefanor]

Debian has autopkgtests that can run package test suites against the installed version of the package to prove that it was installed successfully. They also help to prove that the dependencies remain compatible.

Upstream unit tests are usually run by running the tests outside the source tree. This is trivial when tests are in a separate module.

Fissix's tests are within the fissix module. Many imports would work correctly outside source tree, but relative imports between the test and source packages need to be made explicit.

Would you be willing to carry this patch to allow Debian to run your unit tests more frequently? In future merges from lib2to3, I'd imagine more care would need to be taken to keep the imports separate.

Closes: #30.

[sten0]

Thank you @stefanor ! This will close: #30

[graingert]

In future merges from lib2to3, I'd imagine more care would need to be taken to keep the imports separate.

lib2to3 is deprecated and I imagine won't have further changes

[graingert]

Fissix's tests are within the fissix module.

this is how they work for lib2to3 in cpython already - what does debian do here to make them work?

[stefanor]

what does debian do here to make them work?

The entire python standard library test suite is packaged up as a binary package libpython3.9-testsuite, but that's very unusual. Test suites aren't usually packaged separately, as there isn't much reason for users to want to install them.

For your run of the mill python libraries: Most of the time, tests are in a separate module, and can be tested against the installed library, without any hassle. Copying them out of the source, so that they can be put on sys.path without the library from the source, is often all that's needed.

When tests are within the package module's namespace, sometimes they're just bundled into the library's binary package, if the overhead is negligible.

[amyreese]

These tests are included in the published sdists and wheels. Is there a reason to not just include those in the .deb and run the tests directly from the installed location?

[amyreese]

Assuming they get installed into the system library path, I'm thinking something along the lines of python -m pytest <path/to/site-packages/fissix/tests> should work. It does from a virtualenv:

(venv39) jreese@mordin ~/scratch ‹4› » python -m pytest venv39/lib/python3.9/site-packages/fissix/tests
================================================================================ test session starts ================================================================================
platform darwin -- Python 3.9.1rc1+, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /Users/jreese/scratch
collected 665 items

venv39/lib/python3.9/site-packages/fissix/tests/test_all_fixers.py .                                                                                                          [  0%]
venv39/lib/python3.9/site-packages/fissix/tests/test_fixers.py .............................................................................................................. [ 16%]
............................................................................................................................................................................. [ 42%]
............................................................................................................................................................................. [ 68%]
..........                                                                                                                                                                    [ 70%]
venv39/lib/python3.9/site-packages/fissix/tests/test_main.py xxx.                                                                                                             [ 70%]
venv39/lib/python3.9/site-packages/fissix/tests/test_parser.py ..x...............................................................................................             [ 85%]
venv39/lib/python3.9/site-packages/fissix/tests/test_pytree.py ....................................                                                                           [ 90%]
venv39/lib/python3.9/site-packages/fissix/tests/test_refactor.py ....................                                                                                         [ 93%]
venv39/lib/python3.9/site-packages/fissix/tests/test_util.py ........................................                                                                         [100%]

========================================================================== 661 passed, 4 xfailed in 11.60s ==========================================================================

[amyreese]

To be clear, I'm not against this PR, just trying to understand the desire for running tests in a separate location rather than including them with the installed package, similar to what would be available if someone did pip install fissix.

[stefanor]

These tests are included in the published sdists and wheels. Is there a reason to not just include those in the .deb and run the tests directly from the installed location?

Yes, that's an option. It doubles the size of the package, but at 500k that's not a critical problem.

@sten0: I don't know what your thoughts on that are, you seemed to be trying to avoid that.

[sten0]

When fissix's test suite is included in the binary package, installation of that packages fails during the byte-compilation step ("BOM BOOM!" and other errors). I forget what the other errors were, but it looked to me like it would not be appropriate to install these files in terms of long-term maintenance burden. Beyond that I was mentored to strict standards by a member of the Debian Policy Team who impressed on me the position that test suites should never be installed as part of the main package--no matter the size.

@jreese, given that the self-tests are do not seem to be intended for installation (and are not part of the PyPi sdist) [edit: only tests/test_smoke.py is missing from the sdist], and cause[s] errors during installation of the .deb package, I believe @stefanor's approach is the correct way forward and that it presents a lower risk of long-term maintenance burden than working to keep the self-tests in a state that successfully installs.