deprecate nosetests, make trial be a more widely-used unit test runner

[twisted-trac]

	@zooko reported
Trac ID	trac#4306
Type	task
Created	2010-02-23 19:40:05Z

I've been setting up continuous integration using Hudson. These instructions make it very easy to set up junit-xml integration, coverage.py integration, and other things: http://redsolo.blogspot.com/2007/11/hudson-embraces-python.html . I got that running -- it took about 10 minutes -- and then I decided to do the same thing with trial. This has taken one whole day since then and I'm still not done.

I propose that the Twisted project deprecate trial and start supporting nosetests (or py.test, or anything that a lot of other non-Twisted-hackers are also supporting).

A requirement for this would be that the Twisted test suite itself can run under this alternative runner.

I hasten to add that I like trial. It is IMO well-engineered and well-maintained. The only advantage that nosetests has is that it is more widely used and comes with a lot more howtos like http://redsolo.blogspot.com/2007/11/hudson-embraces-python.html , integrations into other tools like hudson, etc.

The advantage of declaring nosetests (or some such tool) to be the primary testing tool for Twisted would be that more people would use it and report bugs in its Twisted integration (http://code.google.com/p/python-nose/source/browse/nose/twistedtools.py ) and contribute improvements to its twisted integration. Another advantage would be that people who are considering using twisted and who already use nosetests would feel reassured that their familiar testing tools will work well with this unfamiliar library.

Thomas Herve said on IRC that there were some problems with the nose/twistedtools.py code, but he didn't give details. If I end up switching my current project from "integrate trial with all these other tools" to "integrate nose with twisted" then I'll report on this ticket how it went.

Searchable metadata

``` trac-id__4306 4306 type__task task reporter__zooko zooko priority__normal normal milestone__ branch__ branch_author__ status__closed closed resolution__wontfix wontfix component__trial trial keywords__ time__1266954005000000 1266954005000000 changetime__1267146106000000 1267146106000000 version__None None owner__ ```

[twisted-trac]

	Automation removed owner

[twisted-trac]

	@glyph set status to closed

I'm going to close this as "wontfix", because as it stands, nobody is really interested in doing this. It sounds like the original reporter doesn't even really want it any more :). In general, it would be good to get tickets like this closed quickly once we know they're not really on the roadmap, so they don't distract us when considering all open tickets.

If anyone has more compelling reasons in the future, as always, they should feel free to reopen the ticket. My position on this is roughly similar to jml's, so I'm not going to waste time restating it all. (Also, jml, as de-facto trial maintainer, pretty much gets to change my mind by fiat if he cares enough about it.)

Zooko, I think that your point about evangelizing trial outside of the Twisted community is well-made; there are other parts of Twisted, like twisted.python.filepath, that I'd like to do this with as well. But trial and filepath both have significant issues, and I think that for now we should spend more time addressing them than on marketing and evangelism. A few more sprints like the one we just had, though, and I'm sure they'll be in great shape!

[twisted-trac]

	@zooko commented

Oh wait it wasn't http://redsolo.blogspot.com/2007/11/hudson-embraces-python.html that I was following (although that one is good too) it was http://www.rhonabwy.com/wp/2009/11/04/setting-up-a-python-ci-server-with-hudson/ .

[twisted-trac]

	robertc commented

Some discussion: the glue for running twisted stuff in nose is done by having to add a decorator to every test; its not very pretty as a result. It also runs the reactor in a thread, and probably doesn't support choosing a reactor. I'm not sure it knows about cleanliness either (but perhaps that is a feature).

On 'would integration problems like being the first person to try something be avoided if trial did not exist': I don't think so. Being the first person will always happen sometimes. Many times for the same problem in fact - information isn't always shared widely enough to avoid duplicate discoveries.

The test runner breaks down into several layers:

• test class code [assertions, reactor support]
• test discovery code
• execution UI
• reporting UI [ITrialReporter or whatever it is called implementations]

For your particular use case, if trial was able to use a standard test result object with less glue, you wouldn't have needed a new release of trial - as the entire thing you needed was just a new reporting UI implementation.

I think 'be able to reuse components better' has been a theme in trial for many years now - my first conversation with jml was about this in fact :). I don't think deprecation of trial is needed to achieve that :- rather that we're starting to move out of the pure 'can run a test case, or have my test case run' problems we had initially, and are now into less commonly replaced parts of the system. They are less commonly replaced because the stdlib makes it very hard to replace them, and because interactive testing often does not need to replace them. However with machine interfaces becoming very common for testing, this is changing.

[twisted-trac]

	@jml commented

Let's be clear about the constraints:

• I'm happy to consider deprecating the trial command-line tool
• I am unwilling to deprecate the twisted.trial API, since it's the only solid way of running tests that return Deferreds and the only way of supporting our particular extensions to unittest.
• We are not going to deprecate trial in favor of another test runner until that test runner works with the Twisted test suite
• Even then, we would need a way of knowing that whatever replacement we recommended worked for other users of trial.
• Many big projects find that they need their own test runner (or at least their own plugins or wrappers) to support their particular workflows. Any new test runner would have to balance the considerations of being recommended for general use and being ideal for the development of Twisted itself.
• If a test runner needs a plugin to run Twisted tests, then either the runner is broken or twisted.trial has bugs, since trial tests look exactly like standard library unit tests to the external observer.
• If we deprecate trial, then all we can do is recommend another tool. We are not altering our testing system such that we are locked in to another tool.

With respect to nose specifically, I'm told that it is unmaintained right now, which is a mark against it. Also, based on my minimal use of either and passing acquaintance with the respective communities, I would lean toward py.test over nose as a recommended alternative.

With respect to the instance you cite, your complaint is based on something that will be fixed in the next release. Deprecating trial would be over-reacting. As I said, I'm happy to consider it, but it will be based on other reasons.

Your comment about "unfamiliar library" rings false to me, since trial's TestCase has a very similar API to the standard libraries TestCase. It will take actual data to convince me that nose is more well known than the xUnit-based standard library unittest.

[twisted-trac]

	@zooko commented

So it turns out that the recipes that I cited to set up junit-xml and coverage.py integration under hudson with nosetests don't actually work as written, and the coverage integration in nosetests didn't do what I wanted, and I ended up finding it easiest to forget about junix-xml for now and package Brian Warner's trial-coverage plugin: http://allmydata.org/trac/trialcoverage/browser

Now I have the code-coverage integration with hudson that I crave, and it only works right with trial and not with nosetests (given that I didn't spend much time at all trying to diagnose the nosetests problems before I switched back to trial).

I'm not sure what the moral of this story is. Maybe we could engage in more marketing of trial as being part of the broader Python community and not just of the Twisted community?

Some of robertc's and jml's comments about making the interfaces between (different parts of) trial and other software development components be more reusable sound very good, too.