rjsmith/meteor-robotframework

meteor-robotframework

This Meteor Framework package enables you to use the Robot Framework acceptance testing platform to write end-to-end tests for your Meteor applications, using the Velocity framework.

Breaking Change from v0.2.0 to v0.3.0

Version 0.3.0 and after of this package uses a new version of Velocity Core. You should remove the now-redundant packages/tests-proxy folder from your project if you are upgrading from v0.1.0 or v0.2.0.

Key features

Write application-level end-to-end UI tests (see "Test Mode #1" on Sam Hatoum's THE 7 TESTING MODES OF METEOR blog post)
Write your tests using any combination of data-driven, tabular, keyword-driven, or Behaviour Driven Development ("Gherkin", Given-When-Then) styles.
Report Robot Framework test execution results using Velocity, including any logged test execution errors (as Velocity 'Logs' entries)
Auto-trigger test execution when application or test files change
This package includes some sample tests that be added to your application.
Use Robot Framework's Selenium2Library to write complex UI tests of Meteor applications without a line of javascript!
Extend the reach of your Meteor tests by harnessing the power of Robot Framework test libraries, or write your own (in Python)

This package works by spawning a child process which executes Robot Framework's pybot script, acting on the tests contained in the tests\robotframework\suites folder in your Meteor application. The generated output.xml file is then parsed by this package and reported back to Velocity framework.

Watch the introduction video on Youtube:

Installation

To run your tests using this package on a given machine, you must first install Python, Robot Framework and additional test libraries (such as Selenium2Library or PhantomRobot).

You must also ensure you have installed the appropriate browser driver software that Selenium2Library uses to automate the UI of your application. Meteor-robotframework automatically installs PhantomJS and ChromeDriver, but you may want to install others yourself. See the section on webdrivers below for more information.

As an example, assuming you already have a Python 2.5, 2.6 or 2.7 installation (but do please check the official documents):

$ sudo pip install robotframework-selenium2library

Install this package using Meteor's package management system:

meteor add rsbatech:robotframework

Basic Usage

Add Robot Framework test and resource files under the `tests/robotframework/suites' folder in your Meteor application.

pybot command path

Meteor-robotframework uses Robot Framework's pybot runner script to launch the tests. If you get an error when running Meteor-Robotframework tests that looks like this:

? (STDERR)
? (STDERR) events.js:72
? (STDERR) throw er; // Unhandled 'error' event
? (STDERR) ^
? (STDERR) Error: spawn ENOTDIR
? (STDERR) at errnoException (child_process.js:1011:11)
? (STDERR) at Process.ChildProcess._handle.onexit (child_process.js:802:34)

this probably indicates the $PATH environment variable is not correctly set up to point to the Robot Framework installation. See the Verifying Installation section of the Robot Framework User Guide for more details.

arguments.txt

Robot Framework supports a long list of command line options. For example, the --include option ensures that only tests with given tag strings will be executed.The rsbatech:robotframework package will use any command line options specified in the tests\robotframework\arguments.txt file, if it exists.

Test file formats

If you use any of the following Robot Framework test file formats and file extensions, Velocity will automatically trigger a test re-run if any of these files are re-saved:

.txt, .robot, .html, .xhtml, .htm, .tsv

Using Selenium2Library

Import Robot Framework's Selenium2Library into your test files to provide a comprehensive set of keywords to drive the UI of your Meteor application.

Example test.txt file:

*** Settings ***
Documentation  Example Robot Framework plain text format test file using Selenium2Library
Resource       resources.txt

*** Variables ***
${TITLE}  test

*** Test Cases ***
Test Home Page Can Open (Plain Text)
  Open Browser To Home Page
  Home Page Should Be Open  ${TITLE}
  [Teardown]  Close Browser

Example resources.txt file:

*** Settings ***
Documentation  Common Web - testing resource user keywords
Library        Selenium2Library

*** Variables ***
# ${MIRROR_URL} is set by rsbatech:robotframework and should not
# be overidden in your tests!

${SPEED}         0.1 seconds
${TIMEOUT}       5 seconds
${HOME URL}      ${MIRROR_URL}

*** Keywords ***
Open Browser To Home Page
    Create Webdriver  PhantomJS  executable_path=${PHANTOMJS_BINPATH}
    Go To  ${LOGIN_URL}
    Maximize Browser Window
    Set Selenium Speed    ${DELAY}
    Set Selenium Timeout  ${TIMEOUT}

Home Page Should Be Open
    [Arguments]  ${_title}
    Title Should Be    ${_title}

${MIRROR_URL}

You must use the provided ${MIRROR_URL} variable as the root URL for instances of the Open Browser or Go To keywords in your tests. This ensures your test executes against a separate mirror instance of your application and database.

Selenium Webdrivers

For convenience and portability, meteor-robotframework automatically installs several webdrivers locally in your project. If you wish to use those in your tests, you must use the Create Webdriver + Go To keywords instead of Open Browser, so that you can specify the correct path to these local webdriver executables:

Webdriver	Keywords to use to launch new browser sessions
PhantomJS	`Create Webdriver PhantomJS executable_path=${PHANTOMJS_BINPATH}` `Go To ${MIRROR_URL}`
Chrome	`Create Webdriver Chrome executable_path=${CHROMEDRIVER_BINPATH}` `Go To ${MIRROR_URL}`

Meteor-robotframework sets the ${PHANTOMJS_BINPATH} and ${CHROMEDRIVER_BINPATH} variables before launching Robot Framework.

Alternatively, you can use any webdrivers that are separately installed on your machine. For example, ensure the path to the relevant executable is on your environment PATH, you can use the Open Browser keyword instead:

Open Browser  PhantomJS  ${MIRROR_URL}

Report.html

Everytime the Robot Framework tests run against your application, you will see this in the Meteor console log:

[rsbatech:robotframework] Robot Framework is running
[rsbatech:robotframework] 3 failed critical tests
[rsbatech:robotframework] Test report available at: http://localhost:3000/robotframework/report.html
[rsbatech:robotframework] Completed

You can open a new browser window and paste in or manually refresh the Test report url to view the Robot Framework - generated report.html file, like this:

http://localhost:3000/robotframework/report.html

Note that you have to manually refresh to see the most recently - generated report results after each Robot Framework test run.

RF_DEBUG

Run your meteor application with the RF_DEBUG environment variable to view Robot Framework console log output during test execution:

RF_DEBUG=1 meteor

Fixtures

Place any Meteor framework code you may need as part of your tests (e.g. fixture Meteor methods) in the tests/robotframework/fixtures folder, and these will get built and added to your application codebase when running in debug or test mode (via meteor --test).

Advanced

You could try adding this Robot Framework library for working with MongoDB databases. This would let you write tests that interacted with the UI of your Meteor application, but could also access the application's MongoDB database at the same time. Use cases for this could include setting up and tearing down test data, and asserting that expected inserts and updates have been made to specific collections.

It should also be possible to wrap one of the Python-based DDP clients in a Robot Framework library, which would also let you write Robot Framework tests against the DDP API of your application (e.g. Meteor methods).

You can execute the Robot Framework test suites contained in your application repository externally, instead of using this package to trigger them via Velocity, by using one of Robot Framework's start scripts. Or use them as part of a Continuous Integration workflow.

Contributions and Issues

Please report all issues and questions related to this package only (ie. Velocity integration) on Github. If you have an issue with Robot Framework itself or any of its test libraries, you will almost certainly get a better response by contacting the authors of those applications directly.

Pull requests gratefully received!

Change Log

See CHANGELOG.md

Acknowledgements

This package is based on the implementation of the meteor-cucumber package.

This package uses the NPM module xml-stream to efficiently parse the output.xml file generated by Robot Framework.

License

This package is (c) RSBA Technology Ltd 2015, and is released under the MIT License (see LICENSE).