Test suite is too slow!

[gavinlaking]

After an intensive couple of days hacking on this, I've come to the conclusion that the test suite is pretty slow to run. At just over 7 minutes on my MacBook (2Ghz, 4GB, SSD) I'm not getting the instant feedback I desire and think I ought to put some effort into optimising them if possible.

Any thoughts?

[wilkie]

Is that running all of the tests and is that based on the time the testing framework yields, or clock time?

Basically, we don't know enough. Maybe 7 minutes just happens to be the best we can do. Who knows. I'd say somebody should do some proper benchmarking and see what section of the tests is slowest and then figure out why. Ensure that your benchmarks are reproducible and I can give you the times I get on a much beefier machine, just for an extra data point to ensure you aren't optimizing to a single case.

Focus on either unit tests or acceptance tests. I'm sure there are areas in the unit tests that could be refactored. If it is deemed that the acceptance tests are too slow, that requires a bit more test environment tweaking to improve, thinking about it casually.

I'm not sure if anybody can get you to the instant feedback point on running all of the tests, though, especially the acceptance tests. :) 7 minutes would be a godsend for a test:all on some of the projects I've worked on. :P What did you have in mind as a target time to set as a goal for the issue?

[carols10cents]

Oh man i would love this. The unit tests could definitely be loads faster-- they're pretty much all loading rails. Even rake test:lib takes 9.5s.

:+1:

[tlatsas]

Has anyone tried using spork+minitest? There are a few projects out there that provide minitest runners for spork, like: https://github.com/semaperepelitsa/spork-minitest and https://github.com/jhsu/guard-spork-minitest.

I don't really know the state of these projects, but I use spork with rspec and it considerably speeds up the overall testing process.

[wilkie]

Maybe. We still need a baseline benchmark to know what area should be tuned. We don't know if it is the stack, or a particular set of tests that are slow.

[gavinlaking]

@wilkie Good call, I definitely think benchmarking them and concentrating on the slowest areas first will give the best rewards. I'll tinker with this, and let this be a long standing issue until we reach the developer friendly first milestone.

@tlatsas I've used spork with a couple of RSpec projects and liked the speed I must admit. I'm not totally familiar with Minitest myself so I wasn't aware it was possible, or rather hadn't looked into it yet. I think that is a step I'd like to maybe add later (or make it optional as some people may not like the extra dependency/complication). Our main priority (with this issue) is to have a look at the tests and see which ones are slow and take it from there I think.

A couple of breakfast friendly examples:

rake test Run options: --seed 10523 Finished tests in 442.996347s, 0.9955 tests/s, 1.9007 assertions/s. 441 tests, 842 assertions, 0 failures, 0 errors, 6 skips real 8m7.054s user 7m41.428s sys 0m6.910s

I expect wall clock to be slightly higher since we're loading the framework, doing a bunch of IO, buttering toast etc etc.

[steveklabnik]

Spork is more trouble than it's worth.

[Valarissa]

Have to agree with @steveklabnik on this one. Spork only fixes a symptom, that being the loading of rails.

[carols10cents]

Some more granular data points as of 23a3c05c, and I'm only reporting the real time because that's all I care about as a dev waiting for the tests to finish, haha. This is on my macbook air with ruby 1.9.3-p327:

$ time rake test:decorators
real    0m27.177s

$ time rake test:lib
real    0m12.433s

$ time rake test:models
real    1m41.507s

$ time rake test:acceptance
real    7m42.950s

I just joined this email list: http://www.fastrailstests.com/ and I'm looking forward to implementing the techniques as I get them, I think it'll be neat to have rstat.us' codebase to show off the effects of the tips!