jaraco
commented
4 years ago Thanks Paul for writing up this proposal.
While I agree in principle - that the code under test should mirror the supported user's environment as closely as possible, as developers we make lots of compromises that violate that expectation:
- installing to a virtualenv instead of to a system or user environment
- installing with test dependencies installed
- running with tests installed (many packages exclude tests from the installed environment)
- relying on a small subset of possible dependencies
- running as editable-installed
I acknowledge that testing as editable-installed is an imperfect representation of the target environment. It is, however, a very close approximation and has at least these advantages that I appreciate:
- One copy of the code means there's never any confusion about which copy of the code is running.
- Edits can be made and run in real-time, perhaps in a fast loop watching for failing tests to pass.
- There's an elegant simplicity to having the code appear in the repository as it runs in production (especially for pure-Python packages).
- It's an easier transition to having a simple code layout instead of adding another level of indirection. Adding a
srcdirectory to every project means the "getting started" begins with "create three directories, one for your repo, one for your source, and one for your package". - A simple code layout allows one to inspect simple packages without virtualenvs or other machinery (e.g.
python -c "import six"). - The current, simple approach is the one employed by hundreds of other packages I maintain. It would be a burden to transition a portion of those (divergent approaches) or all of them (lots of flux).
I'm also concerned with the disadvantages of the proposal. Both approaches for (OP 2) involve several changes to the codebase and add debt. They require more non-default settings and limit the options for overriding. For example, the "run from another directory" approach has these disadvantages:
- It's more difficult to replicate interactive the behaviors the test environment exhibits (change cwd, tempdir).
- Overriding the default target is no longer possible through posargs.
Regarding (OP 1), I know in the past I've run into bugs in tox for which the best known workaround was to use an editable install. One was tox-dev/tox#467 but the more serious one was tox-dev/tox#373 where the long-term resolution (pytest-dev/pytest#2042) is still imperfect and runs doctests on a different copy of the source than the code under test (another manifestation of "which copy of the code matters").
Some tests in the test suite already honor the proposed intention (e.g. test_integration, test_distutils_adoption), exercising critical behavior that would be masked by an editable install. Most of the tests, however, are exercised at a scope (unit, functional) that's unaffected by an editable install. Moreover, adopting test-as-installed still doesn't address some of the biggest gaps this project has for testing (capturing real-world scenarios like upgrades and source-only bootstrapping).
All of this experience leads me to believe that the costs of this change as proposed would vastly outweigh the benefits.
If running all tests as installed is important, I suggest to consider another approach - create a separate tox environment or other isolated build process that's dedicated to running tests as installed. This opt-in feature could be ignored by most workflows but invoked in select CI environments to provide the necessary protections. Perhaps this feature could run in dedicated docker containers or VMs so as to more accurately represent execution outside of a virtualenv as well.
Could we start with such an approach and evaluate the benefit before considering impacting the default developer workflows?
pganssle
joergent
abravalheri
kloczek
webknjaz
Right now, we are testing an editable install of
setuptoolsand possibly relying on the fact that the tests are run from the repository root, which is a problem. We should instead test our projects as installed, meaning we should:usedevelop=Trueflag fromtox.ini.src/layout and/or run the tests from another directory per the suggestions in this blog post.This is especially important after the resolution to #2259, which makes a pretty significant change to the way our project works when it is installed normally as opposed to in an editable install. Considering that we're basically considering editable installs of
setuptoolsto be an unsupported configuration, we should definitely be testing only installed configurations.