Webdriver integration for integration/e2e tests

[dignifiedquire]

So I have been thinking about this for quite a while now and want to get some more input if this is a good idea to integrate into Karma.

The idea is to integrate the webdriver protocol into Karma in a way that you can run integration tests as easily as you run your unit tests through Karma right now.

Here is a little example of what I imagine:

// karma.conf.js
browsers = ['chrome', 'firefox'];
files = [
  { pattern: 'tests/e2e/webdriver_test.js', webdriver: true}
];

// webdriver_test.js
var assert = require('assert');

// This gives you the browser that is currently run
// Initialization and connection is all done by Karma
var browser = karma.getBrowser();

browser.on('status', function(info) {
  console.log(info);
});

browser.on('command', function(meth, path, data) {
  console.log(' > ' + meth, path, data || '');
});

browser.on('ready', function() {
  browser.get("http://admc.io/wd/test-pages/guinea-pig.html", function() {
    browser.title(function(err, title) {
      assert.ok(~title.indexOf('I am a page title - Sauce Labs'), 'Wrong title!');
      browser.elementById('i am a link', function(err, el) {
        browser.clickElement(el, function() {
          browser.eval("window.location.href", function(err, href) {
            assert.ok(~href.indexOf('guinea-pig2'));
            browser.quit();
          });
        });
      });
    });
  });
});

[joelmoss]

Why is webdriver needed? Shouldn't this be possible using just phantomjs?

[dignifiedquire]

It's true that phantomjs has the ability to be controlled in this way, but if you want to do real world testing you need real browsers not just phantomjs. Webdriver gives you a unified api to control all major browsers, this way you can write your tests once and run them in all browsers.

[joelmoss]

Yes I know, but in most cases PhantomJs is sufficient as it covers webkit based browsers. I think it would be good to get e2e tests running with Phantomjs first.

[dignifiedquire]

Ah okay I see what you mean. The pro for directly doing webdriver is that we can then get phantomjs + all the other browsers. But if it turns out that integrating phantomjs on its own is much simpler this might be an option for now.

[joelmoss]

Yeah, I'd be more than happy for that to start with.

[alindsay55661]

+1 for webdriver. It doesn't make sense to just test on phantomjs, that is only webkit which will soon mean only Safari.

[alindsay55661]

BTW, if you do implement wd then I can move all our saucelab tests into Karma which would be awesome.

[necolas]

Webdriver integration would be great!

[dignifiedquire]

@all happy to hear you like it. @vojtajina To make this happen we would need to differentiate between files that are run through webdriver and files that get loaded into karma. As we are in the middle of changing the configuration file maybe we could at least plan for this change. Let me know what you think.

[dignifiedquire]

@joelmoss I looked into it but the best way to support running tests in phantomjs turns out to be using webdriver, so I'll go the route of directly doing webdriver.

[vojtajina]

Having a webdriver launcher (especially webdriver launcher for browserstack and saucelabs) would be great. I think it should be very simple to implement. That should enable anyone with saucelab account to test on any browser they support.

Now, what @Dignifiedquire is proposing here. I don't understand why we need this. Why do you want to merge unit tests and e2e tests ? Karma does not buy you anything - I would recommend people to just use WebDriver for writing e2e tests, Karma for unit tests. If you wanna have a single task for it, use grunt. Am I missing something ?

[dignifiedquire]

I don't understand why we need this. Why do you want to merge unit tests and e2e tests ?

The idea is not to merge the tests but rather to leverage all the capabilities of karma to use it not also as a unit test runner but as test runner to run e2e tests. Setting everything up for e2e tests can be a pretty big and shitty job and I think that when we would integrate it, it would make live for us and a whole lot of other people a lot easier. Take for instance the serving of all your test files and application files. We already taught that karma, so if we could just execute e2e tests on these this would simplify many things in testing.

[vojtajina]

I believe webdriver is the way to go with e2e testing and it does start browsers (through the drivers). As for serving the test files, I think in most of the cases, you need "your" web server for e2e testing, as it probably does some other stuff than just serving static files...

So if I was doing e2e testing, I would use webdriver and merge karma + webdriver on grunt (or something similar) level.

Just my $0.02, I'm not stopping anybody from doing anything ;-)

On Sat, Jul 13, 2013 at 7:00 AM, Friedel Ziegelmayer < notifications@github.com> wrote:

I don't understand why we need this. Why do you want to merge unit tests and e2e tests ?

The idea is not to merge the tests but rather to leverage all the capabilities of karma to use it not also as a unit test runner but as test runner to run e2e tests. Setting everything up for e2e tests can be a pretty big and shitty job and I think that when we would integrate it, it would make live for us and a whole lot of other people a lot easier. Take for instance the serving of all your test files and application files. We already taught that karma, so if we could just execute e2e tests on these this would simplify many things in testing.

— Reply to this email directly or view it on GitHubhttps://github.com/karma-runner/karma/issues/413#issuecomment-20920271 .

[bitcity]

Came around searching for webdriver to see if Karma can be used for the exact purpose described here. It would be awesome to use Karma for E2E testing with either WebdriverIO or wd

P.S. I find the API of WebdriverIO very well documented : http://webdriver.io/api.html

[mightyiam]

Intern combines unit with e2e ("functional") tests in one framework. It doesn't do some cool stuff that Karma does. It doesn't launch browsers and it doesn't watch and rerun tests on changed files, which seem basic goodies to us Karma users.

[jtalon21]

+1 for this

[FinalAngel]

that would be awesome +1

[zeitos]

+1

[ghost]

Actually it is possible to do e2e testing in an iframe, so no webdriver or phantomjs needed. I just tested it:

var express = require("express");
var app = express();

app.get("/", function(req, res){
  res.send("hello world");
});

app.listen(3333);

// ...

var client = new E2EClient("http://localhost:3333");

describe("test", function() {
    it("does some tests on the remote web application", function(next) {
        client.load("/", function (contentWindow, path){
            expect(path).to.be("/");
            expect(contentWindow.document.body.innerHTML).to.be("hello world");
            next();
        });
    });
});

I'll write an e2e lib, something similar to nightwatch based on this.