Integration tests - Githubissues

addyosmani commented 12 years ago

Following on from:

I've always wanted TodoMVC to incude a set of integration tests for all applications to ensure that they're behaving consistently (and are also not breaking expected functionality when fixes come in).

The challenge here has always been that we have a large (and growing) set of applications (including Labs, over 24), some of which do not have the best documentation and would require significant additional effort to get those tests in place. With both @sindresorhus and I currently spending most of our time on this project on rewrites/labs it can be a challenge.

I do think we need tests (and ideally we'd have them already) but we have to balance getting a version 1.0 out with functionally stable apps sooner vs. a few months of lead time to get tests written for all apps.

My personal opinion is that we can either:

Release 1.0 as planned (with consistency updates for all applications) in a month without tests
Release the above as 1.0pre and spend time writing tests, releasing a 1.0 RC a few months later
Spend the extra time now writing tests, make sure the apps clear and then get 1.0 out in a few months

My personal priority is getting functional examples out sooner that developers can use to make decisions about which frameworks to use, even though I completely agree with @cowboy and @kangax about this approach putting the cart before the horse. It's a tricky one because this project offers examples meant for review and learning vs. something going into a large production system.

Thoughts/ideas on the above are welcome.

sindresorhus commented 12 years ago

I completely agree, but I don't think we should delay the 1.0 release for this.

kangax commented 12 years ago

How you guys want to distribute your time is totally up to you. Main thing is that we all agree that tests are needed and will be added as soon as possible.

addyosmani commented 12 years ago

I think that's fair enough, @kangax :)

sindresorhus commented 12 years ago

Any recommendation on what testing framework to use and how and what to test is more than welcome.

kangax commented 12 years ago

I used QUnit (+jQuery.simulate) to test backbone-based app recently. It worked pretty well, including all the drag-drop interactions.

But I keep hearing QUnit sucks and Jasmine, Mocha are all the rage these days. Aside from being able to run tests from command line (in Mocha), I'm not yet sure what the benefits of those are comparing to QUnit.

sindresorhus commented 12 years ago

Cool, thanks for the input :)

QUnit can btw also be run in the command line using PhantomJS. I've done it on a couple of projects and it works prefectly. It also almost passes the jQuery testsuite.

stas commented 12 years ago

+1 for mocha. I wrote ember.js tests with it and chai.js.

It will require phantom-nodify to run with phantom, but the runner can be patched easily for compatibility. I also wrote a require.js implementation for phantom-nodify, so that should be also covered.

Anyway, big :+1: for bringing this up!

IgorMinar commented 12 years ago

PhantomJS can't verify if the app works in all required browsers. I suggest that you take a look at Testacular instead.

However, I can't quite imagine how you want to write a single integration-level test suite that would work for all implementations. By definition integration tests are implementation specific.

What you need is an end to end test. And for those Selenium is likely the best solution.

stas commented 12 years ago

@IgorMinar not sure I agree about your last comment on #200.

The purpose of having a common test suite (integration test), is to avoid manual verification if an app passes the specs. It should pass in most (if not all) of the cases because:

template is the same, for every app and should be used by every app developer
most of the stuff there is jQuery (selectors and some event triggers), which should again work in every modern browser

If anyone feels like he wants to write own unit tests, that's fine, but first it should pass our specs, pragmatically. Also, if someone wants to use another framework/runner, that's fine, just provide the updated specs.

IgorMinar commented 12 years ago

I absolutely do want to have automated test suite. I'm just saying that "integration" level test suite is not going to cut it.

Integration tests are a test suite that focus on testing interactions between various components of an application and not necessarily the application as whole.

End to end tests pretend to be a user interacting with the application. These tests don't care about the internal implementation, internal state of the application or how the individual components are assembled. These tests look something like "When I click on textbox 'newTask', enter text 'new task 1' and hit enter, does the browser add a new visual element for the task with text 'new task 1'".

template is the same, for every app and should be used by every app developer

that's not true. look at the ember app, angularjs or knockout app, these frameworks are declarative and use templates for binding declarations.

most of the stuff there is jQuery (selectors and some event triggers), which should again work in every modern browser

angularjs app doesn't use jquery at all

Again, to be truly implementation agnostic, you want to build an end to end test suite.

sindresorhus commented 12 years ago

Ok, how about this. We do a Selenium test-suite which every new implementation must pass. In addition we have a test-suite template which a framework can choose to use (the one by @stas) or they can roll their own?

IgorMinar commented 12 years ago

@sindresorhus I like that

sindresorhus commented 12 years ago

@addyosmani @stas Sounds good?

stas commented 12 years ago

I don't think Selenium is needed since CSS selectors (which I already mentioned are present in every implementation template) can be used without it. jQuery sugaring can be replaced with some vanilla js code to achieve that.

I also don't think it's team's job to build "end to end" test suite. Most of the app specs, are trivial, and require "action and effect" approach. And again, if someone needs a full blown test suite... patches welcome.

Still my 2 cents, that you can ignore.

IgorMinar commented 12 years ago

@stas

I don't think Selenium is needed since CSS selectors (which I already mentioned are present in every implementation template) can be used without it. jQuery sugaring can be replaced with some vanilla js code to achieve that.

So you want to hack together a poor man's test runner instead of using an industry standard solution? Sorry but I don't think it's worth the effort.

I also don't think it's team's job to build "end to end" test suite. Most of the app specs, are trivial, and require "action and effect" approach.

It's in "team's" interest to be able to verify that all implementations pass all of the requirements. Implementors can read the spec and implement it to the dot (or not!), but the burden of verifying that the app is working according to the spec is onto the person merging the implementation into the upstream repo.

stas commented 12 years ago

I doubt that Selenium is still an industry standard. I have used Selenium before, and from all tools available, I found it one of the most slowest and bringing a lot of overhead (from provisioning point of view, to actually running it). Selenium is one of the tools I found better to avoid, like many others do.

I don't want to keep up the discussion, I think my feedback is pretty clear. It's @addyosmani and @sindresorhus who should make a decision.

IgorMinar commented 12 years ago

@stas so what's the alternative to Selenium? (home-grown test frameworks don't count)

addyosmani commented 12 years ago

Granted there will be a grace period between releases I think we'll have ample time to evaluate a number of options (including Selenium and others) for our needs. Let's wait until the evaluation has been done to continue that particular discussion :) On Jul 3, 2012 7:47 AM, "Igor Minar" < reply@reply.github.com> wrote:

@stas so what's the alternative to Selenium? (home-grown test frameworks don't count)

Reply to this email directly or view it on GitHub: https://github.com/addyosmani/todomvc/issues/134#issuecomment-6727163

stas commented 12 years ago

@igorminar Once again, I don't understand where you are trying to push this. As I see this issue, the solution is not what framework or tool will be used to solve it, but to offer people a starting point in tdd/bdd. This project is mostly for educational purposes and tools (if any) should be used to help developer pick up concepts easily.

Citing project tag line:

A common learning application for popular JavaScript MV* frameworks

Please, if there are no other questions related to the project, I would like us stop here. Once again, thanks for the feedback.

marcenuc commented 12 years ago

I think there's a little misunderstanding that is going on here. @stas and @IgorMinar are talking about two really different things. From the README:

... a project which offers the same Todo application implemented using MVC concepts in most of the popular JavaScript MV* frameworks of today. Solutions look and feel the same, ...

That's why you must use exactly the same end-to-end tests for all the framewoks, as @IgorMinar says. Otherwise you can't state that the look and feel is the same for all implementations. This is needed by the project owners to ensure that what stated in the README holds true. You can't do this with integration tests.

Integration tests belong to the implementations, and can be different for each implementation. End-to-end tests belong to the project as a whole, and must be shared between implementations.

Then the question is: what tool can be used to do end-to-end testing, which works with all implementations?

sindresorhus commented 12 years ago

I've split this into multiple issues. Feedback welcome :)

Unit testing: #254

End-to-end testing: #255

tastejs / todomvc

Integration tests #134