Automated unit tests

[balupton]

The time expense of running the unit tests, which requires all the different combinations of adapters and browsers to be manually started is the ultimate history.js bottleneck.

If we were able to automate this testing, so for instance we run make test and it will in the background automatically, start, run, and evaluate all the different test combinations then that would solve the bottleneck.

Once that solution is implemented, and that ultimate history.js bottleneck is eliminated, then we are left with one other ultimate history.js bottleneck - acceptance and testing of pull requests.

History.js receives a ton of pull requests, but the time expense to test them all is incredibly expensive. Recently Travis CI announced pull request support for their continuous integration system, which would mean whenever someone submits a pull request (providing the automated testing is already done) then Travis CI would test it automatically for us - before even looking at the pull request. AMAZING!

This means that accepting a pull request will become as simple as checking that the tests passed (1 minute), then releasing a new version (5 minutes).

This would be the holy grail of community driven development and maintenance, and would initiate the rebirth of History.js.

Now then. Does anyone have any idea how we can automate the browser testing so that it can be started from the command line? (while keeping support for history.js?). That seems to be the hardest bit, but if we can solve that, by golly... that would be a tremendous feat.

[ginader]

PhantomJS should allow you to do that (for webkit): http://phantomjs.org/ Also have a look at TestSwarm: https://github.com/jquery/testswarm

[Delapouite]

There's also http://browserling.com/ made by substack

[balupton]

Thanks guys :)

Based on your suggestions, I've come up with the following criteria that will need to be satisfied to meet our custom use case:

• Needs to be able to test code in multiple web browsers (not just their engines, as the History.js often deals with browser specific bugs - e.g. safari and chrome both have different problems) (eliminates phantomjs)
• Needs to be able to run without any human interaction (eliminates testswarm)
• Needs to provide an API so we can trigger the tests and get the results back automatically (eliminates browserling)
• Needs to be cheap enough (or rather free) to be support a project of History.js size and zero budget (eliminates testling)

Testling seems like the best fit so far, but is too expensive - I'll email them and see if they are willing to do a custom plan for us. Though if we can find other service that ticks all the criteria, that would be amazing!

Well done guys, let's keep searching!

[ginader]

did you see this article? http://jquery.org/updates/2012/05/04/status-update-34/ The part I found particularly interesting was: https://github.com/clarkbox/testswarm-browserstack

Have a look!

[balupton]

Wow... that could work! SWEET!.

[ginader]

:-)

[Daniel15]

You could try JsTestDriver by Google - I've used it with Jasmine in the past, but it also works with QUnit. It lets you run a "test server" that browsers connect to. You tell the server to run the tests, and it tells all the connected browsers to run them. It can start up the browsers when you start the tests, and close them when the tests are complete.

[ginader]

it seems like the Modernizr team has a similar approach: http://www.thecssninja.com/talks/debug_lazy_dev/#/

[ginader]

did you do any investigation?

[balupton]

There is now http://ryanseddon.github.com/bunyip/ which seems to wrap most of this up quite nicely.