search

axemclion / grunt-saucelabs

Grunt task for running all your browser tests using Sauce Labs
MIT License
182 stars 98 forks source link
grunt grunt-task sauce-labs test test-javascript

readme

grunt-saucelabs

Build Status Selenium Test Status

Selenium Test Status

Dependency Status devDependency Status

A Grunt task for running QUnit, Jasmine, Mocha or any framework using Sauce Labs' Cloudified Browsers.

Grunt is a task-based command line build tool for JavaScript projects, based on nodejs. QUnit is a powerful, easy-to-use JavaScript unit test suite used by the jQuery, jQuery UI and jQuery Mobile projects and is capable of testing any generic JavaScript code, including itself! Jasmine is a behavior-driven development framework for testing JavaScript code. Mocha is a JavaScript test framework for running serial asynchronous tests. Sauce Labs offers browser environments on the cloud for testing code.

About the tool

The grunt-contrib-qunit task runs QUnit based test suites on PhantomJS. The saucelabs-qunit task is very similar but runs the test suites on the cloudified browser environment provided by Sauce Labs. This ensures that subject of the test runs across different browser environment. The task also uses Sauce Connect to establish a tunnel between Sauce Labs browsers and the machine running Grunt to load local pages. This is typically useful for testing pages on localhost that are not publicly accessible on the internet. The saucelabs-jasmine runs Jasmine tests in the Sauce Labs browser. The saucelabs-jasmine task requires jasmine-1.3.0. There is also a saucelabs-mocha task that lets you run your Mocha tests on Sauce Labs cloudified browser environment.

Usage

This task is available as a node package and can be installed as npm install grunt-saucelabs. It can also be included as a devDependency in package.json in your node project.

To use the task in grunt.js, load the npmTask.

grunt.loadNpmTasks('grunt-saucelabs');

In the grunt.initConfig, add the configuration that looks like the following

var request = require('request');
...
'saucelabs-qunit': {
  all: {
    options: {
      username: 'saucelabs-user-name', // if not provided it'll default to ENV SAUCE_USERNAME (if applicable)
      key: 'saucelabs-key', // if not provided it'll default to ENV SAUCE_ACCESS_KEY (if applicable)
      urls: ['www.example.com/qunitTests', 'www.example.com/mochaTests'],
      build: process.env.CI_BUILD_NUMBER,
      testname: 'Sauce Unit Test for example.com',
      browsers: [{
        browserName: 'firefox',
        version: '19',
        platform: 'XP'
      }]
      // optionally, he `browsers` param can be a flattened array:
      // [["XP", "firefox", 19], ["XP", "chrome", 31]]
    }
  }
}

The configuration of saucelabs-jasmine, saucelabs-mocha and saucelabs-custom are exactly the same. Note the options object inside a grunt target. This was introduced in grunt-saucelabs-* version 4.0.0 to be compatible with grunt@0.4.0

Full list of parameters which can be added to a saucelabs-* task:

A typical test task running from Grunt could look like grunt.registerTask('test', ['server', 'qunit', 'saucelabs-qunit']); This starts a server and then runs the QUnit tests first on PhantomJS and then using the Sauce Labs browsers.

Exposing Test Results to the Sauce Labs API

Since this project uses the Sauce Labs js unit test API, the servers at Sauce Labs need a way to get the results of your test. Follow the instructions below to assure that the results of your tests are delivered properly.

Test result details with Jasmine

You can make Job Details pages more informative on Sauce by providing more data with each test. You will get info about each test run inside your suite directly on Sauce pages.

Jasmine detailed results

You can do that by using Jasmine JS Reporter that will let saucelabs-jasmine task provide in-depth data about each test as a JSON object.

All you need to do is to include the new jasmine-jsreporter reporter to the page running Jasmine tests by adding new script in header:

<script src="https://github.com/axemclion/grunt-saucelabs/raw/master/path/to/jasmine-jsreporter.js" type="text/javascript"></script>

and telling Jasmine to use it:

jasmineEnv.addReporter(new jasmine.JSReporter());

Test result details with QUnit

Add the following to your QUnit test specification

var log = [];
var testName;

QUnit.done(function (test_results) {
  var tests = [];
  for(var i = 0, len = log.length; i < len; i++) {
    var details = log[i];
    tests.push({
      name: details.name,
      result: details.result,
      expected: details.expected,
      actual: details.actual,
      source: details.source
    });
  }
  test_results.tests = tests;

  window.global_test_results = test_results;
});
QUnit.testStart(function(testDetails){
  QUnit.log(function(details){
    if (!details.result) {
      details.name = testDetails.name;
      log.push(details);
    }
  });
});

Test result details with mocha

Add the following to the mocha test page html. Make sure you remove any calls to mocha.checkLeaks() or add mochaResults to the list of globals.

<script>
  onload = function(){
    //mocha.checkLeaks();
    //mocha.globals(['foo']);
    var runner = mocha.run();

    var failedTests = [];
    runner.on('end', function(){
      window.mochaResults = runner.stats;
      window.mochaResults.reports = failedTests;
    });

    runner.on('fail', logFailure);

    function logFailure(test, err){

      var flattenTitles = function(test){
        var titles = [];
        while (test.parent.title){
          titles.push(test.parent.title);
          test = test.parent;
        }
        return titles.reverse();
      };

      failedTests.push({name: test.title, result: false, message: err.message, stack: err.stack, titles: flattenTitles(test) });
    };
  };
</script>

OnTestComplete callback

An optional parameter to the grunt task is OnTestComplete, a callback which is called at the end of every test, before results are logged to the console. You can use this callback to intercept results from SauceLabs and re-report the results (or use the information for your own purposes)

Receives two arguments (result, callback). result is the javascript object exposed to sauce labs as the results of the test. callback must be called, node-style (having arguments err, result where result is a true/false boolean which sets the test result reported to the command line)

When running the tests for this project, we need to test the case where a test fails on Sauce. In this case, we want to record a test Failure as a Success for us.

'saucelabs-qunit': {
  all: {
    options: {
      username: 'saucelabs-user-name', // if not provided it'll default to ENV SAUCE_USERNAME (if applicable)
      key: 'saucelabs-key', // if not provided it'll default to ENV SAUCE_ACCESS_KEY (if applicable)
      urls: ['www.example.com/qunitTests', 'www.example.com/mochaTests'],
      build: process.env.CI_BUILD_NUMBER,
      testname: 'Sauce Unit Test for example.com',
      browsers: [{
        browserName: 'firefox',
        version: '19',
        platform: 'XP'
      }],
      onTestComplete: function(result, callback) {
        // Called after a unit test is done, per page, per browser
        // 'result' param is the object returned by the test framework's reporter
        // 'callback' is a Node.js style callback function. You must invoke it after you
        // finish your work.
        // Pass a non-null value as the callback's first parameter if you want to throw an
        // exception. If your function is synchronous you can also throw exceptions
        // directly.
        // Passing true or false as the callback's second parameter passes or fails the
        // test. Passing undefined does not alter the test result. Please note that this
        // only affects the grunt task's result. You have to explicitly update the Sauce
        // Labs job's status via its REST API, if you want so.

        // The example below negates the result, and also updates the Sauce Labs job's status
        var user = process.env.SAUCE_USERNAME;
        var pass = process.env.SAUCE_ACCESS_KEY;
        request.put({
            url: ['https://saucelabs.com/rest/v1', user, 'jobs', result.job_id].join('/'),
            auth: { user: user, pass: pass },
            json: { passed: !result.passed }
        }, function (error, response, body) {
          if (error) {
            callback(error);
          } else if (response.statusCode !== 200) {
            callback(new Error('Unexpected response status'));
          } else {
            callback(null, !result.passed);
          }
        });
      }
    }
  }
}

Examples

Some projects that use this task are as follows. You can take a look at their GruntFile.js for sample code

If you have a project that uses this plugin, please add it to this list and send a pull request.

Integration with a CI system

Grunt tasks are usually run alongside a continuous integration system. For example, when using Travis, adding the following lines in the package.json ensures that the task is installed with npm install is run. Registering Sauce Labs in test task using grunt.registerTask('test', ['server', 'saucelabs-qunit']); ensures that the CI environment runs the tests using npm test. To secure the Sauce Key, the CI environment can be configured to provide the key as an environment variable instead of specifying it file. CI Environments like Travis provide ways to add secure variables in the initial configuration. The IndexedDBShim is a project that uses this plugin in a CI environment. Look at the .travis.yml and the grunt.js for usage example.

Changelog

9.0.0

8.6.3

8.6.2

8.6.1

8.6.0

8.5.0

8.4.1

8.4.0

8.3.3

8.3.2

8.3.0

8.2.2

8.2.1

8.2.0

8.1.1

8.1.0

8.0.3

8.0.2

8.0.1

7.0.0

6.0.0

5.1.3

5.1.2

5.1.1

5.1.0