search

dwyl / learn-nightwatch

:last_quarter_moon_with_face: Learn how to use Nightwatch.js to easily & automatically test your web apps in *real* web browsers.
584 stars 214 forks source link

readme

Learn Nightwatch: Complete Beginners Tutorial

Automate your acceptance tests and run them in real browsers!

nightwatch-logo-with-slogan

[![Codeship Build Status](https://img.shields.io/codeship/d9151e40-1473-0134-47e0-12348d1f3442.svg?style=flat-square)](https://codeship.com/projects/157818) [![Dependency Status](https://img.shields.io/david/dwyl/learn-nightwatch.svg?style=flat-square)](https://david-dm.org/dwyl/learn-nightwatch) [![devDependency Status](https://img.shields.io/david/dev/dwyl/learn-nightwatch.svg?style=flat-square)](https://david-dm.org/dwyl/learn-nightwatch#info=devDependencies) [![HitCount](https://hits.dwyl.com/dwyl/learn-nightwatch.svg)](https://github.com/dwyl/learn-nightwatch)

Why?

Testing what the people using your application/website will see and their ability interact with the product is (probably) the most important part of building a web app/site. You can have amazing code, a super-fast backend and gorgeous UI, but none of that matters if people are unable to use it because of a basic bug!

dilbert-internet-full

User Acceptance Testing (UAT) with a tool like Nightwatch (Selenium) lets you to run real-world scenarios in your Web App which will give you confidence that the app works in the chosen device(s)/browser(s).

What?

Automated Acceptance Testing using Real Browsers.

Nightwatch is quick to setup and the tests/scenarios are easy to write.

We exhaustively read through all the tutorials, blog posts and documentation for Nightwatch (including the mailing list & StackOverflow Q&A) and have condensed our findings into this step-by-step guide. We hope you find it useful and decide to use it for your web app/site! Please give us feedback and if you get stuck, tell us!

Background Links

Who?

Who should learn/use Nightwatch?

How?

Quick Start (5mins)

Try it on your local machine in 5 mins by following these 3 easy steps:

1. Clone

Clone the repository by copy-pasting the following command into your terminal (replace cp with copy if you're on Windows):

git clone https://github.com/dwyl/learn-nightwatch.git && cd learn-nightwatch && cp sample.env .env

Note: if you're curious what that last part is, see: https://github.com/dwyl/env2

2. Install1

Make sure you cd learn-nightwatch so that you're in the correct directory and then install the required dependencies including Selenium Server and chromedriver:

npm install

3. Run (tests)2

Run the Nightwatch tests:

npm test

You should expect to see: learn-nightwatch-console-output-success

Once you see the tests pass you are well on your way to testing with Nightwatch!

1This assumes you have node.js installed. If not, https://nodejs.org/en/download/ 2Selenium Requires Java/JDK see: Java Installation section below. (don't worry, you'll be up-and-running shortly...!) Once you have Java installed re-run the Nightwatch tests (npm test).

Step-by-Step Tutorial to end to end test on your OWN PROJECT

Now that you have had a taste for running tests with Nightwatch, let's walk through each of the steps to get this working in your project.

Installation (in detail)

1) Make sure you have Java(Runtime Environment JRE) installed

While we prefer not to run Java on our machines for security reasons Selenium is still the best way of running tests in real browsers.

You can check by typing java -version into your terminal and you should see your version number if you have Java installed.

How do I install Java? https://www.java.com/en/download/help/download_options.xml pick your Operating System and follow the instructions

 Mac OSX? (use homebrew)

If you haven't updated brew in a while, do that first:

brew update

That will install cask which is now part of Homebrew.

Now you can install Java:

brew cask install java

You should see something like this: install-java-with-homebrew-cask

See: https://stackoverflow.com/questions/24342886/how-to-install-java-8-on-mac

2) cd into your project

3) Install nightwatch

First install the nightwatch node.js module from NPM:

npm install nightwatch --save-dev

Note: while the Nightwatch docs instruct to install globally (-g), we prefer to always install devDependencies locally to the project and list them explicitly in package.json so it's clear to everyone viewing/using the project exactly which version is required to run the tests.

4) Install selenium-server and chromedriver

In order to run Browser tests Nightwatch uses Selenium. We prefer to automate the installation of Selenium using selenium-server which ensures that everyone on our team always has the latest version.

npm install selenium-server chromedriver --save-dev

5) Configuration

Once you've installed nightwatch, you will need to create a configuration file. Some Nightwatch tutorials use a nightwatch.json file; this is good for the most basic cases but if you want to use variables in your configuration we recommend using a .js file; specifically called nightwatch.conf.js. Save this file to your project directory.

You can copy over our basic configuration saved in nightwatch.conf.BASIC.js: nightwatch.conf.BASIC.js

Or copy the following into a file called nightwatch.conf.BASIC.js

require('env2')('.env'); // optionally store your Evironment Variables in .env
const seleniumServer = require("selenium-server");
const chromedriver = require("chromedriver");
const SCREENSHOT_PATH = "./screenshots/";

// we use a nightwatch.conf.js file so we can include comments and helper functions
module.exports = {
  "src_folders": [
    "test/e2e"// Where you are storing your Nightwatch e2e tests
  ],
  "output_folder": "./reports", // reports (test outcome) output by nightwatch
  "selenium": {
    "start_process": true, // tells nightwatch to start/stop the selenium process
    "server_path": seleniumServer.path,
    "host": "127.0.0.1",
    "port": 4444, // standard selenium port
    "cli_args": {
      "webdriver.chrome.driver" : chromedriver.path
    }
  },
  "test_settings": {
    "default": {
      "screenshots": {
        "enabled": true, // if you want to keep screenshots
        "path": SCREENSHOT_PATH // save screenshots here
      },
      "globals": {
        "waitForConditionTimeout": 5000 // sometimes internet is slow so wait.
      },
      "desiredCapabilities": { // use Chrome as the default browser for tests
        "browserName": "chrome",
        // uncomment the lines to run Chrome in headless mode
        // "chromeOptions" : {
        //    "args" : ["headless"]
        // }
        "chromeOptions": {
          "args" : ["--no-sandbox"],
          "w3c": false
        }
      }
    },
    "chrome": {
      "desiredCapabilities": {
        "browserName": "chrome",
        "javascriptEnabled": true // turn off to test progressive enhancement
      }
    }
  }
}

function padLeft (count) { // theregister.co.uk/2016/03/23/npm_left_pad_chaos/
  return count < 10 ? '0' + count : count.toString();
}

var FILECOUNT = 0; // "global" screenshot file count
/**
 * The default is to save screenshots to the root of your project even though
 * there is a screenshots path in the config object above! ... so we need a
 * function that returns the correct path for storing our screenshots.
 * While we're at it, we are adding some meta-data to the filename, specifically
 * the Platform/Browser where the test was run and the test (file) name.
 */
function imgpath (browser) {
  var a = browser.options.desiredCapabilities;
  var meta = [a.platform];
  meta.push(a.browserName ? a.browserName : 'any');
  meta.push(a.version ? a.version : 'any');
  meta.push(a.name); // this is the test filename so always exists.
  var metadata = meta.join('~').toLowerCase().replace(/ /g, '');
  return SCREENSHOT_PATH + metadata + '_' + padLeft(FILECOUNT++) + '_';
}

module.exports.imgpath = imgpath;
module.exports.SCREENSHOT_PATH = SCREENSHOT_PATH;

One of our favourite things about using a .js file is the ability to add comments in the file. This makes it much easier for new people to understand what's going on. We have a slightly more evolved nightwatch.conf.js (with Saucelabs) see: github.com/dwyl/learn-nightwatch/nightwatch.conf.js

6) Running config file

You will need to run the config file you created to download the Selenium driver.

node nightwatch.conf.BASIC.js

7) Create Your Nightwatch Test

Nightwatch "looks" for tests in the /test folder of your project by default; you can change this to whatever you prefer. We keep our Nightwatch tests in test/e2e.

This is the simplest test you can write for Nightwatch.

Assuming you're using the same folder structure as we are (tests in test/e2e), create a file named guineaPig.js inside that folder, containing the following code:

var config = require('../../nightwatch.conf.BASIC.js');

module.exports = { // adapted from: https://git.io/vodU0
  'Guinea Pig Assert Title': function(browser) {
    browser
      .url('https://saucelabs.com/test/guinea-pig')
      .waitForElementVisible('body')
      .assert.title('I am a page title - Sauce Labs')
      .saveScreenshot('guinea-pig-test.png')
      .end();
  }
};

See: github.com/dwyl/learn-nightwatch/test/e2e

8) Run your Test

Depending on what you named your configuration file, run it with a command resembling the following:

node_modules/.bin/nightwatch --config nightwatch.conf.BASIC.js

The camelCase filename ("guineaPig") is used to display the test's name in the console as in:

Running: Guinea Pig Assert Title

We add an entry in our package.json "scripts" section to not have to type all that each time. e.g:

"scripts": {
  "e2e": "nightwatch --config nightwatch.conf.BASIC.js"
}

Then run your tests as:

npm run e2e

If you called your config file nightwatch.conf.js you can run your tests without specifying the config file, i.e.

node_modules/.bin/nightwatch

If you see the following message while trying to run the tests: learn-nightwatch-java-not-installed

Then return to step 2 to install Java

Optional (Level Up)

Saucelabs

Most people building web apps/sites don't have easy access to several devices/browsers to test their output, if you need to test in a range of browsers/devices Saucelabs is a great option.

browser logos

In our nightwatch.conf.js we have defined saucelabs as our "default" setting.

We run our tests on saucelabs by running the following npm script/command:

npm run sauce

Which corresponds to the following complete command:

./node_modules/.bin/nightwatch -e chrome,ie11,android_s4_emulator,iphone_6_simulator

This just means "Run Nightwatch using the default configuration (Saucelabs in our case) and execute all tests in this list of browsers."

Note: you will need to have the following environment variables exported for Saucelabs to run your test:

export SAUCE_USERNAME=your-username
export SAUCE_ACCESS_KEY=your-key

If you're new to Saucelabs, checkout: github.com/dwyl/learn-saucelabs

Upload Screenshots to S3

If you decide to use Saucelabs to run your tests (in several devices/browsers), it will take screenshots for you and keep them inside Saucelabs. That's nice for people who are used to using Saucelabs, but what about the other stakeholders?

We decided to upload our screenshots to S3 and created a super-simple .html file which shows a slideshow of the images.

Example: https://isearch-ui.s3-eu-west-1.amazonaws.com/1.0.21/index.html

If you want the screenshots of tests to be uploaded to S3, you will need to have the following environment variables declared:

export AWS_S3_BUCKET=yourbucket
export AWS_REGION=eu-west-1
export AWS_ACCESS_KEY_ID=IDHERE
export AWS_SECRET_ACCESS_KEY=YOURKEY

The script we wrote to perform the uploading is: github.com/dwyl/learn-nightwatch/test/e2e/upload_screenshots_to_s3.js

The screenshots taken on Saucelabs browsers/devices are saved locally and uploaded to S3 when tests succeed.


Running your Nightwatch tests on your application being served locally

Running your Nightwatch tests on your Continuous Integration (CI)

Running your Nightwatch tests on CI is easy on CodeShip.

We usually set the required (minimum) node version in our package.json e.g:

"engines": {
  "node": "4.4.6"
},

Once you have the desired version of node installed.

Setup Commands:

# install dependencies:
npm install

Test Command:

# run tests
npm test

That's it.

Running your Nightwatch tests on Travis-CI with sauce connect

Since we are testing on the localhost we have to make sure that the server is started before the tests are run and closes after the tests finish. So we need to boot up a server to serve our content. Travis makes this easy enough via a before_script task. In the task we will just start a python simple server and give it a few seconds to boot. The ampersand at the end of the python line tells travis to run the process in the background instead of blocking the execution thread, allowing us to run tasks at the same time.

language: node_js
before_script:
  - python -m SimpleHTTPServer &
  - sleep 2
node_js:
    - "6.0"

One other way to run a server before running a test is to use the before and after methods present in nightwatch.

module.exports = {
  before: function (browser, done) {
    server = require('../server')(done) // done is a callback that executes when the server is started
  },

  after: function () {
    server.close()
  },

  'Demo test': function (browser) {
    browser
      .url('localhost:3000')   // visit the local url
      .waitForElementVisible('body'); // wait for the body to be rendered

    browser
      .assert.containsText('body','hello') // assert contains
      .saveScreenshot(conf.imgpath(browser) + 'dwyl.png')
      .end()
  }
}

The server.js can be a simple express server.

function makeServer(done) {
  var express = require('express');
  var path = require('path');
  var app = express();

  app.get('/', function (req, res) {
    res.status(200).sendFile(`index.html`, {root: path.resolve()});
  });
  var server = app.listen(3000, function () {
    var port = server.address().port;
    done()
  });
  return server;
}
module.exports = makeServer;

Note : In the above example you can see that the port is fixed. It will run fine if you are running tests on a single device. If you are running tests on multiple devices on saucelabs, this will give you an error that the port is already in use, as all the devices try to start the server on the same port (in our current approach). So we need to dynamically allot available ports to prevent this error. You can use get-port for this.

This is all we need to run a test on browser/s. Now we have set up saucelabs on travis.

To run the test on Travis-CI and use sauce connect you need to add a addon to your .travis.yml

addons:
  sauce_connect: true

The username and access_key can be optionally stored in .travis.yml or can be stored on travis-ci website as environment variables. There are various methods of storing the username and access_key of saucelabs and you can read more about them here. In our case we have preferred to save it on travis website so that our .travis.yml is simple.

Now you have to make some changes in nightwatch.conf.js

const TRAVIS_JOB_NUMBER = process.env.TRAVIS_JOB_NUMBER;
// in test_settings.default:
default: {
  launch_url: 'http://ondemand.saucelabs.com:80',

  username : process.env.SAUCE_USERNAME,
  access_key : process.env.SAUCE_ACCESS_KEY,
  ...
  desiredCapabilities: {
    build: `build-${TRAVIS_JOB_NUMBER}`,
    'tunnel-identifier': TRAVIS_JOB_NUMBER,
  },
}

See the modified final config here You can run multiple test commands i.e.

- npm run test:unit; npm run test:e2e

You can see the working code here and the corresponding test on travis here

Note-1: Tests on the PRs of forked repos will fail as the secured environment variables are not accessible to them on travis. You will receive authentication error in that case.

Note-2: Running tests on IE still seems tricky. Will have to explore more. Any help is appreciated.

Note-3: If you are receiving timeout error, maybe you are running tests on many devices. Try to adjust the time or decrease the number of devices.

Running your Nightwatch tests on CircleCi.

To run the test on circle ci you need to make some adjustments to your circle.yml Here is an Example from the circle ci docs

dependencies:
  post:
    - wget https://saucelabs.com/downloads/sc-latest-linux.tar.gz
    - tar -xzf sc-latest-linux.tar.gz

test:
  override:
    - cd sc-*-linux && ./bin/sc --user $SAUCE_USERNAME --api-key $SAUCE_ACCESS_KEY --readyfile ~/sauce_is_ready:
        background: true
    #Wait for tunnel to be ready
    - while [ ! -e ~/sauce_is_ready ]; do sleep 1; done
    - npm start
        background: true
    # Wait for app to be ready
    - curl --retry 10 --retry-delay 2 -v http://localhost:5000
    # Run selenium tests
    - npm run test:e2e
  post:
    - killall --wait sc  # wait for Sauce Connect to close the tunnel

The test override starts the selenium server for you. Once it is ready the application is started in the background. Finally when ready, the tests are started. You can run multiple test commands i.e.

- npm run test:unit; npm run test:e2e

just like in the package.json



Running a single nightwatch test

Nightwatch tests can be quite time-consuming so sometimes you may just want to run one test at a time

This can be done by giving each test a tag by adding tags: [ 'tagname' ] to the beginning of your exported test scenario. You can then run the individual test (in this case with tag 'test1') with the script: "node_modules/.bin/nightwatch --tag test1"

If you want to dynamically choose which test to run using the command line, you could create another script in your package.json e.g. "e2etag": "./node_modules/.bin/nightwatch --env local --tag"

and then in your command line you can just run npm run e2etag -- test1

tl;dr

More detail than you will probably need ... but we're keeping for completeness.

Background

Why Nightwatch instead of xyz...?

We first looked at NightmareJS, and even though it looks really good (fast), we saw the reaction non-technical people had when we mentioned it and did not want to have to explain the name to people/clients every time, so instead opted for nightwatch. If nightmare ever change their name, we could re-consider it.

Research


Setup (Detail)

Manual Selenium Install

If you prefer to install it manually that's an option.

Visit: https://www.seleniumhq.org/download/ and download the latest version.

When downloading the selenium-server-standalone-2.53.0.jar you may see a warning in your browser: download-selenium-chrome-warning Click on "keep" to save the file. Once you have it, put it in the bin directory of your project and re-name it to selenium.jar (without the version number).

StackOverflow Questions

Remind me to Respond to these:

Cons (of using Nightwatch)