Closed robdennis closed 10 years ago
EliAndrewC
commented
10 years ago That is pretty bad; FWIW running with py.test -m 'not functional' only takes about 20 seconds. Which is still a long time for unit tests to run, but I guess that does mean that our server tests are clearly the culprit.
So for what it's worth, I don't think that plugins will ever have to write server tests. So plugin tests will all be super fast, which is the most important thing.
robdennis
commented
10 years ago Do you show the same time for running all tests?
So much time is in the collecting phase, it's baffling
On Mon, Mar 31, 2014 at 8:15 PM, Eli Courtwright notifications@github.com wrote:
That is pretty bad; FWIW running with
py.test -m 'not functional'only takes about 20 seconds. Which is still a long time for unit tests to run, but I guess that does mean that our server tests are clearly the culprit.So for what it's worth, I don't think that plugins will ever have to write server tests. So plugin tests will all be super fast, which is the most important thing.
Reply to this email directly or view it on GitHub: https://github.com/appliedsec/sideboard/issues/44#issuecomment-39157966
EliAndrewC
commented
10 years ago Yeah, it takes about 7 minutes for me as well.
robdennis
commented
10 years ago this ticket itse;fisn't really actionable, so I'd like to propose a couple things:
time so we can passively see execution time across runs (there's more to a tox invocation than the py.test part, is my logic)efficiency, performance, or maybe test-specific prefixes for those or similarthoughts @appliedsec/sideboard-assignable ?
EliAndrewC
commented
10 years ago Options 1 and 2 seem sensible, though I'm ambivalent about how much of a priority faster unit tests are. At a glance, there are 28 other open tickets at the moment, and an off-the-cuff count on my part says that I'd consider faster tests to be a higher priority than 12 of them.
Logistically there are definitely some server tests that we could refactor to just be regular unit tests, including
The only tests that really NEED to be server tests IMO are the websocket tests, since I don't think we're meaningfully testing our websocket code otherwise.
Doing the refactoring necessary to run those other tests as basic unit tests is a huge amount of work, and my own preferences/biases put this as being less important than about half of our backlog, so I'm neither very bothered or excited by this.
robdennis
commented
10 years ago As usual, a reasoned and researched response. I'd like to restate my position and we'll see where that leaves us: "Before we expect other people to start using this, we need to prioritize having a reasonable unit testing workflow. Otherwise, we need to be comfortable with the current group of people being the only contributors"
A realistic scenario:
tox before and after making a pull request is normal.There are a couple of approaches we could take, some overlapping:
functional not run by default when using toxtoxall but the first are what I'd rate "low-to-moderate" difficulty, with 2 and 3 really being about trading ease of unit tests vs the work of making it configurable and typing more characters to functional test.
I'd say that these are the kinds of things that I'd expect to have tagged with "unit-test efficiency."
ALRIGHT. Restated. Here are the ways forward as I see them:
I think we're both comfortable with the first one, and unsure if @ftobia has an opinion.
Thoughts?
EliAndrewC
commented
10 years ago To the extent that we want Sideboard to become a successful open source project used by more than one team, the progression that I see as being most likely is this:
My own opinion is that having sane tests which run quickly is only really a prerequisite to the last item on that list. I personally wouldn't feel comfortable promoting Sideboard to the internet at large when the unit tests take 7 minutes to run only 200 tests, for basically the same reasons that you enumerated. I personally DO feel comfortable promoting Sideboard to other teams and people I know personally even when the core unit tests take that long (so long as the unit tests they write for their own plugins will run in a sane amount of time).
Personally I wouldn't really want to tackle the final step of promoting Sideboard to the larger internet until early in 2015, since that will give us time to hammer at it on persona projects and try to get some other teams to use it. Brian has indicated he'd like to try it out with Djangular after we resolve #43. I'd be happy if a couple of random internet denizens started using it before then, but I wouldn't want to open the floodgates too early.
That's the kind of progression as an open source project I've had in mind while thinking about my own priorities for which tickets are most important. Someone who wanted to release Sideboard to the open internet a lot earlier would necessarily prioritize faster tests a lot higher, for all of the reasons you mentioned. In the meantime, the specific suggestions about e.g. making tox not run functional tests by default sound sensible as a temporary measure.
robdennis
commented
10 years ago ok, I think we've established the place where we diverge:
thankfully, I'm well positioned to do the moderate amount of work necessary to make the default tox invocation a lot quicker, simply by having it do less (in the way that I described) and we can come back around to this as we near 2015.
I'm comfortable closing this, and I'll be making a follow-on ticket for the tox work (re-open or comment if I missed something)
ftobia
commented
10 years ago Sounds good to me. My preferences are more in line with Rob's, but since I'm less willing to do the necessary work, they probably shouldn't be given much weight. It's good that we're on the same page w.r.t. suggesting Sideboard to the internet at large.
this wasn't always like this. 7+ minutes to "install" and run the tests on just one interpreter.
this isn't counting actually downloading/compiling dependencies since this run had everything in it already.