Remove test command and tests_require

[jaraco]

In this comment, @graingert proposes that we may be able to completely remove support for tests_require instead of transitioning that tooling from easy_install to pip install.

While he didn't directly propose removal, the effort would only benefit that ticket if the install functionality were completely removed, so let's explore what that will entail.

While I agree that tox is an excellent, powerful, robust solution, it's more heavy than tests_require, requiring that the user have tox installed in advance. As a more thorough solution, it also is subject to bugs and constraints that a simpler test runner is not. There are several advantages to setup.py test and tests_require over tox:

• Doesn't have any prerequisites (except the implicit one on Setuptools, which is generally assumed).
• As a result of not having any prerequisites, invocation is often a simple invocation (setup.py test) instead of multiple steps (e.g. pip install --user tox; python -m tox).
• setup.py test allows for invocation under a number of different Python versions naturally (i.e. python3.3 setup.py test or py -3.3 setup.py test) whereas tox offers "run under the python in which tox is installed" or "run for explicitly-declared python versions".
• As setup.py test doesn't use virtualenv, it's not subject to virtualenv bugs or other constraints imposed by virtualenv (such as version 14 dropping support for Python 3.2.).
• As setup.py test doesn't rely on pip, it's not subject to the bugs of pip (such as issues with --editable installs or namespace packages) or other constraints imposed by pip (such as dropping support for Python 3.2).

I consider these advantages small and easy enough to overcome, especially now that many of these issues have been resolved in setuptools, pip, and virtualenv. If we can get to a place that tox can broadly supplant the uses cases of setup.py test and pytest-runner (and thus tests_require) in practice, then yes, deprecating and removing it would be in order. Given the amount of activity and bugs I see around these tools, I'd asses they're still in active use.

Before flagging this functionality as deprecated, I'd like to survey the community about the possibility to see if there are use cases that would prove difficult to support with tox.

@graingert, would you be interested in being the champion for this effort (removing tests_require), starting with the outreach on distutils-sig and then implementing the deprecation/removal changes?

[FRidh]

Perhaps its relevant here. After discussing with @RonnyPfannschmidt we added the following to the Tox docs.

Integrating tox with setup.py test is as of October 2016 discouraged as it breaks packaging/testing approaches as used by downstream distributions which expect setup.py test to run tests with the invocation interpreter rather than setting up many virtualenvs and installing packages. If you need to define setup.py test you can better see about integrating your eventual test runner with it, here is an example of setup.py test integration with pytest. As the python eco-system rather moves away from using setup.py as a tool entry point it’s maybe best to not go for any setup.py test integration.

As a downstream we prefer to have one command that tests the package for the interpreter that is available and not any other versions.

[jaraco]

I've discovered that tox -e py or tox -e python will test against the python in which tox is installed, and in fact that's the default behavior if no envlist is provided. I use this usage by default across dozens of projects and find it quite useful.

[FRidh]

Yes, I agree, as long as no other envs are defined, then there's no problem.

[techtonik]

tox is good for CI, but too unwieldy for development. There should be an easy and intuitive way to run single test. Then this way can be combined with filesystem watcher of your choice.

[graingert]

@techtonik tox works totally fine for dev, you can use {posargs} to pass test flags to pytest: tox -e py36 -- --cov -s etc

[techtonik]

@graingert how much time does it take on you system to execute a single test with tox. On my it takes more than 5 seconds, which is not acceptable. Also, tox doesn't plan to add output redirection, which makes debugging hard.

https://github.com/tox-dev/tox/issues/405 https://github.com/tox-dev/tox/issues/73

[RonnyPfannschmidt]

@jaraco instead of test_requires, i'd like to see something that allows pip install -e .[test]

[mcepl]

I don't like the idea of replacing simple test calling with tox with my openSUSE Python maintenance hat on. We strive hard to achieve reproducible builds in openSUSE and thus we strongly prefer packages with small dependency trees. osc dependson openSUSE:Factory python-tox standard x86_64|grep python-|wc -l gives me 33 dependent packages (there are some false positives) and that seems too many for tool which should end up as a dependency of setuptools (or be on the same level in the dependency tree). We would probably have to eliminate all python setup.py test calls with something more primitive (e.g., python -munittest discover if you don't plan to eliminate that as well), but that seems counterproductive.

[pganssle]

I don't like the idea of replacing simple test calling with tox with my openSUSE Python maintenance hat on. We strive hard to achieve reproducible builds in openSUSE and thus we strongly prefer packages with small dependency trees.

To be clear, tox is not installed in the environment that actually runs the tests (unless you specifically want that). It manages virtual environments that have exactly the dependencies you want installed and nothing more. I suspect that this will give you fewer dependencies during the run, not more.

We would probably have to eliminate all python setup.py test calls with something more primitive (e.g., python -munittest discover if you don't plan to eliminate that as well), but that seems counterproductive.

Eliminate where or how? For your own projects or for projects you are packaging? tox is similar to make test in that each project can specify its own test commands. python -munittest discover will not work for a large number of packages, which intend for you to use pytest, and presumably also intend to have the dependencies that tox installs.

[mcepl]

Eliminate where or how? For your own projects or for projects you are packaging? tox is similar to make test in that each project can specify its own test commands. python -munittest discover will not work for a large number of packages, which intend for you to use pytest, and presumably also intend to have the dependencies that tox installs.

Well, the problem with whole virtual environments tox promotes is that it is completely useless in the Linux packaging world: we have all builds working in their own chrooted completely isolated environments so whatever issues virtenv is trying to solve are completely irrelevant for us. Also, we have different mechanisms (full blown isolated builds in separate virtual machines or something of that calibre) for testing with different versions of Python or something.

So, we usually end up with digging into tox.ini and ripping out only unittest (or pytest, it doesn't matter) line and using only that.

[pganssle]

@mcepl Well, in that case I think then tox is not actually necessary, since tox is just doing the part that you are doing manually anyway (e.g. setting up the dependencies and environment, then running the tests).

One nice thing about the very declarative nature of tox is that it would probably not be terribly difficult for you to write a script that parses tox.ini and creates your preferred type of build file from it (subject to adjustment since you'd need to map the names of PyPI dependencies to your own system).

In any case, fewer and fewer projects are actually using setup.py test because of the problem of dependency management, so likely deprecating setup.py will create less work for you, since you wouldn't need to support both projects using tox and projects using setup.py test (and extracting tests_require from a non-declarative setup.py file, etc).

[mcepl]

@mcepl Well, in that case I think then tox is not actually necessary, since tox is just doing the part that you are doing manually anyway (e.g. setting up the dependencies and environment, then running the tests).

That's exactly what I’ve meant, so general feeling that more and more projects rely on tox more heavily makes me a bit uneasy. And yes, setup.py test is probably just more a calling convention than anything useful. We usually rather use pytest test runner directly (even for nose- and unittest- based test suites).

[native-api]

tox is designed to build and test the package in all Python versions that it officially supports. While setup.py test is supposed to run tests, in the running Python version, assuming the build has already been done.

So, they are doing massively different things. To get comparable experience, a user needs to somehow figure out which tox environment specified ini tox.ini corresponds to the running Python (if there even is such an environment -- tox only supports official CPython releases AFAICS) and pass it to tox -e.

[pganssle]

@native-api None of these things are true. While tox is very useful for parametrizing over different Python versions, it is trivial to use the current interpreter, it's even mentioned in this issue: tox -e py.

It certainly supports pypy in the normal parametrized list of environments.

Yes there are differences between tox and setup.py test, because tox does what setup.py test should do. It installs your dependencies in an isolated environment and then runs them.

[native-api]

@pganssle None of these things are true.

+1 for the explanations. It's not in the reference documentation so I had no way to know this. (The whole purpose of a reference is to be a complete list of the project's official guarantees.)

[FRidh]

In Nixpkgs the default Python builder (that expects setuptools-based projects) runs python setup.py test. Fewer projects implement it nowadays so we typically have to override it. Hopefully soon we can switch the default builder to use pyproject. In that case, we don't have any default command for the test phase, because nothing has been defined and nothing common exists, although it is tempting to just run pytest by default (bootstrapping is a bit more difficult then though).

I can understand the motivation for using Tox, but I suggest that at this point it makes more sense to standardize by going through a PEP. PEP 517/518 is a big improvement, and getting testing included there as well is where we should be going.

[techtonik]

Looks like tox is toxic, because it can not be used without adding a vendor lockin.

Quick <1s test runs for TTD in IDEs that support test-on-edit and using tox for a slow, heavy and thorough CI testing should not be mutually exclusive options.

[RonnyPfannschmidt]

its not ..

[pganssle]

I can understand the motivation for using Tox, but I suggest that at this point it makes more sense to standardize by going through a PEP. PEP 517/518 is a big improvement, and getting testing included there as well is where we should be going.

You are welcome to write a draft PEP for this. I suspect that such a thing would be welcomed. I think the reason there has been no clear spec for this is that it's not very common for end users to run a project's tests - usually the people who run the tests are the developers of the package themselves and certain downstream redistributors. The package developers usually have to manually configure CI to run their tests and they don't have to do this at scale, so a standard is less necessary. The redistributors tend to be testing their whole package ecosystem, which means that something defined in terms of the PyPI package ecosystem will still need manual intervention, reducing the need for any sort of standard.

Still, if you and other downstream redistributors of Python packages would like to propose such a standard, I recommend doing so. I think the packaging category or the users category on the Python discourse would be a good place to start. I think you will get a better response if you come in with a concrete proposal rather than creating a general "brainstorming" topic, but that is just my experience with posting such things.

[FRidh]

Discussion on Python Packaging discourse about a section in pyproject.toml. https://discuss.python.org/t/proposal-for-tests-entry-point-in-pyproject-toml/2077

[andy-maier]

I have difficulties following the thought that tox would be the right standard tool to run tests. I like tox, but it is the tool for creating virtual environments and then to invoke a tool that runs tests.

What I am missing most is an ability to include the test files in a package (i.e. the test files, not just a specification of their additional package dependencies). In https://github.com/pypa/setuptools/issues/1684, @pganssle said:

I personally still recommend that you include your test code in your sdist, but not in the wheel.

How can that be achieved?

[McSinyx]

If I understand your question correctly, in case you have tests in a separate dir, you can include them in MANIFEST.in, e.g.

include tests/*.py

[jaraco]

I personally still recommend that you include your test code in your sdist, but not in the wheel.

How can that be achieved?

McSinyx' suggestion to include them in MANIFEST.in is one way. Another is to use a tool like setuptools_scm to always include all of your source files from your repo in your sdist. See the keyring project for an example of tests that get included with the sdist but aren't installed.

[techtonik]

Is it possible to define tests files directly in pyproject.toml? Something like:

exclude = {{ tests.files }}

[tests.files]
tests/*.py

And FWIW, the dependencies to run them:

[tests.requirements]
pytest
[tests.requirements:py2]
six

And the command to run:

[tests.command]
pytest {{ args }}

Or what would be the correct syntax?

If the data is defined this way, then:

1. packaging tool can exclude the test from packing
2. IDE can find the requirements, match them against specific Python interpreter and exec the right command

[graingert]

Yes, you can use tool.pytest.ini_options https://docs.pytest.org/en/latest/customize.html#pyproject-toml

[RonaldAJ]

Admittedly I am coming late to this discussion. But the deprecation warning escaped my attention for a long time as it precedes all the other output during testing. If I see all the tests passing I don't scroll up to search for deprecation messages.

As far as I can judge tox complicates the use of the already complicated setuptools even further. I am therefore unhappy with the deprecation of the test command, which is working fine for me. It feels to me like imposing a standard suitable for packages supported by large teams, which doesn't fit other usecases.

What makes deprication desirable? Are there large costs involved in keeping it up and running? Because there are also costs involved in adapting to this change, mainly for developers who are not experts on Python distribution and testing, and don't work in a large team.

[jaraco]

What makes deprication desirable?

See OP.

Are there large costs involved in keeping it up and running?

Yes. test depends on tests_requires which depends on easy_install which depends on old, insecure, and unsupported behavior.

[mcepl]

Could we split this ticket into two, please?

a) “Remove test command and test_require …”, and b) “in favour of tox”

I agree with the first one. But could we still keep supported “python setup.py build && some-kind-of-runner” as well (for nosetest2/pytest/python -munittest discover)?

[RonaldAJ]

What makes deprication desirable?

See OP.

The OP only states minor advantages of setup test. But I understand there are maintenance costs now.

@mcepl's suggestion of having some possibility to run the existing tests now discovered by the test command directly from setup using another method under the hood would mitigate the negative impact.

[pganssle]

Could we split this ticket into two, please?

There is no need to split the ticket into two. From our perspective, the important part is that python setup.py test goes away, and we no longer support running it. We recommend tox because that basically does a better job of solving the same problem, but you are welcome to use whatever test runner you want.

1878 was merged October 22, 2019, which was released as part of 41.5.0 on October 27, 2019.

I think we should go ahead and target the removal of setup.py test for approximately 1 year after the deprecation (which is at the end of next month).

[FRidh]

Could the removal be done in two steps? First, remove the dependency resolving part. And then later, the actual test execution part. If I am correct the main maintenance issue is the resolving part, which I absolutely agree with that should disappear.

The reason I am suggesting this is that distro's (speaking here as Nixpkgs maintainer) will provide the dependencies themselves anyway, but a significant part (I guess a third or so, rough guess) of all Python packages still use setup.py test for testing.

[RonaldAJ]

Wasting my time trying to overcome this change. I don't understand why you can't remove the dependency part while keeping the test runners. What is their relation? To me they are two entirely different things.

[RonaldAJ]

Workaround for nose2 using the ability to define your own commands. It can probably be adapted for other testsuites.

The OnSuccess nose2 plugin can be edited out, but it provides me with the ability to remove log-files generated during testing. Only imports specific to this solution are shown.

from setuptools import Command
import nose2
from nose2.events import Plugin

class OnSuccess(Plugin): 
    def wasSuccessful(self, event):
        if event.success:
            print('Success!')
        else:
            print('Failure!')

class nose2Testing(Command):
    """ Run my command.
    """
    description = 'nose2testing'

    user_options = []  # obligatory

    def initialize_options(self):
        self.onsuccesshook = OnSuccess()
        print('initialize_options')

    def finalize_options(self):
        print('finalize_options')

    def run(self):
        nose2.discover(argv=['.'], exit=False, extraHooks=[('wasSuccessful',self.onsuccesshook)])

if __name__ == "__main__":
    setup(
        ...
        cmdclass={
                'nose2': nose2Testing,
            },
    )

Now I can call setup.py:

python setup.py clean build install nose2

[jaraco]

@RonaldAJ Glad you found a workaround. Note that Setuptools also supports defining that command as a plugin, in a third-party package the way pytest runner does. You could use that to implement the nose2 command.

Still, if it were up to me, I'd try to decouple test running from building, as you're only adding constraints that may later become deprecated as well.

why you can't remove the dependency part while keeping the test runners

Setuptools is aiming to get out of the business of being a swiss-army knife of project management (as distutils was envisioned) and instead focus on providing a best-in-class build implementation (mainly the operations defined by PEP 517). The goal is to separate concerns (SCM tooling, testing, environments, installation, package indexes, distribution, building) into different, largely independent tools, coordinated by standards.

[RonaldAJ]

@jaraco I guess that good working examples of migration would help. Setuptools is hard to use for people like me who need to make changes very occasionally.

[jaraco]

That would be nice. Unfortunately, because there are such a diverse array of workflows, there's no deterministic migration. However, for the use-case you indicate above, why not simply pip install .; nose2? That should be roughly equivalent to setup.py clean build install nose2. If you need something more elaborate, then consider using a tool like tox or nox or make to facilitate the setup.