Preceptor is a test-runner and test-aggregator that runs multiple tests and testing frameworks in parallel, sequential, or a combination there of, aggregating all of the test-results and coverage-reports.
Main-Features:
Features through plugins:
Table of Contents
Install this module with the following command:
npm install preceptor
To add the module to your package.json
dependencies:
npm install --save preceptor
To add the module to your package.json
dev-dependencies:
npm install --save-dev preceptor
Today, there are a lot of testing frameworks out there, many of which are optimized for testing specific areas. A couple of these frameworks are:
All of these testing-frameworks produce their own test results - often in different output formats. Additional testing tools, like Selenium, create even more results, generating the results for tests that were run in multiple browsers. CI systems help here a bit since they can usually handle multiple results, but all the results are usually plugged into one root-node, making it hard to organize and analyse the test-results. Some engineers spend a good amount of time aggregating all of these results in a reasonable manner, creating often one-off hacks to merge all the results. Others just manually parse through the results, determining if any kind of problems were encountered during the most recent test-run, loosing the ability to have a one-look green/red summary.
Code-coverage metrics pose a similar problem: code-coverage reports are created independently and in isolation of tests that run for the same system in different testing frameworks or in separate tests (unit, component, integration, functional, acceptance, ...). It is challenging to have a complete overview of all coverage metrics as there is never only one coverage-report for all the tests. What is usually the case is that only some tests have coverage reports, possibly having only a subset of the metrics that could have been collected - most testing systems, for example, are not collecting client-side coverage-reports when running integration tests with Selenium; these are valuable coverage results.
In addition, all of these tools need to be coordinated, running sequentially, in parallel, or even a combination thereof. However, many testing tools do not have this capability build-in, resulting in third-party developers create their own solutions which are quite often not much more than a hack, having also quite possibly inconsistent interfaces as they are usually not created by the same people. All of this coordination can easily turn the code-base into an unmaintainable mess, quite possibly triggering false-positives due to flaky testing infrastructure and code.
Preceptor tries to solve all of these problems by using a consistent way of configuring the tools, managing the workflow within a single configuration file, and aggregating the test-results and code-coverage reports in one place. In addition, it adds a flexible and extensible plugin infrastructure to the test-runner, introducing even more features like a Selenium / WebDriver integration, injecting client/server code in setup/teardown methods of different testing frameworks, keeping the test-code free of glue-code.
Preceptor comes with the following features out of the box:
As mentioned, plugins add even more features, including:
Preceptor comes with a command-line tool that takes a configuration file that defines what sequence the task should run, what data needs to be collected, and how some tasks affect others.
Let's have a look at some examples that uses Mocha as sole testing tool. Preceptor can be used to mix multiple testing frameworks and independent tests at the same time, however, I will only use Mocha for the next examples.
To get started, create the following file with the name config.js
:
module.exports = {
"configuration": {
"reportManager": {
"reporter": [
{ "type": "Spec" },
{ "type": "List", "progress": false }
]
}
},
"tasks": [
{ // Task
"type": "mocha",
"title": "Everything", // Just giving the task a title
"configuration": {
"paths": [__dirname + "/mocha/test1.js"]
}
}
]
};
This configuration adds a Mocha task and two reports:
Spec
- A detailed console report that is very similar to Mocha's spec
reporter.List
- Comprehensive list of all problems encountered which will be printed add the end of all test-runs.As SUT, add the following lib.js
file to the same folder:
console.log('Mostly Harmless');
module.exports = {
answerToLifeTheUniverseAndEverything: function () {
return 42;
},
whatToDo: function (phrase) {
return (phrase == "don't panic");
},
allThereIs: function () {
return 6 * 9;
},
worstFirstMillionYears: function () {
return 10 + 10 + 10;
},
message: function () {
console.log("So Long, and Thanks for All the Fish.");
},
startInfiniteImprobabilityDriver: function () {
throw new Error("infinitely improbable");
}
};
The tests live in a sub-folder with the name mocha
. For now, let's add a couple of simple tests that give you a glimpse on how test-results will look.
Copy the following content into mocha/test1.js
:
var assert = require('assert');
var lib = require('../lib');
it('should know the answer to life, the universe, and everything', function () {
assert.equal(lib.answerToLifeTheUniverseAndEverything(), 42);
});
describe('The End', function () {
it('should print something', function () {
lib.message();
});
});
Preceptor adds a command-line tool that can be executed by calling preceptor
in the shell (when installed globally). Run Preceptor with the above created configuration file:
preceptor config.js
The result should look something like this (colors are not shown):
✓ should know the answer to life, the universe, and everything
The End
✓ should print something
You can see that the output that was created in lib.js
isn't visible anywhere. By default, Preceptor will hide any output created by tests, the SUT, and the testing-framework, displaying only the output of loaded reporters. This behavior can be changed by modifying values in the task-options for each individual tasks. For more information on how to do this, please refer to the API documentation or see below.
Now, to show some of the features, let's add two more test files:
mocha/test2.js
var assert = require('assert');
var lib = require('../lib');
it('should find all there is', function () {
assert.equal(lib.allThereIs(), 42);
});
it('should calculate the worst first million years for marvin', function () {
assert.equal(lib.worstFirstMillionYears(), 30);
});
mocha/test3.js
var assert = require('assert');
var lib = require('../lib');
it('should not panic', function () {
assert.ok(lib.whatToDo("don't panic"));
});
it('should panic', function () {
assert.ok(!lib.whatToDo("do not panic"));
});
Let's run them all sequentially. Simply add them to the task-list:
module.exports = {
"configuration": {
"reportManager": {
"reporter": [
{ "type": "Spec" },
{ "type": "List", "progress": false }
]
}
},
"tasks": [
{
"type": "mocha",
"title": "Everything",
"configuration": {
"paths": [__dirname + "/mocha/test1.js"]
}
},
{
"type": "mocha",
"title": "Calculations",
"configuration": {
"paths": [__dirname + "/mocha/test2.js"]
}
},
{
"type": "mocha",
"title": "Phrasing",
"configuration": {
"paths": [__dirname + "/mocha/test3.js"]
}
}
]
};
Run Preceptor:
✓ should know the answer to life, the universe, and everything
The End
✓ should print something
1) should find all there is
✓ should calculate the worst first million years for marvin
✓ should not panic
✓ should panic
1) Root should find all there is
54 == 42
AssertionError: 54 == 42
at Context.<anonymous> (.../mocha/test2.js:5:9)
... // Abbreviated
This example shows a failed test, displaying some details and the stack-trace that is abbreviated here in the output for our examples. Let's keep this failing test around for now.
Great! However, now, the tests appear to run all in the same top-level test-suite - we cannot easily distinguish which tests were run in which task. Let's change this. Add a suite:true
to each task, turning each task into a virtual test-suite that creates a new level of test-suites in the test-results using the title
given.
module.exports = {
"configuration": {
"reportManager": {
"reporter": [
{ "type": "Spec" },
{ "type": "List", "progress": false }
]
}
},
"tasks": [
{
"type": "mocha",
"title": "Everything",
"suite": true,
"configuration": {
"paths": [__dirname + "/mocha/test1.js"]
}
},
{
"type": "mocha",
"title": "Calculations",
"suite": true,
"configuration": {
"paths": [__dirname + "/mocha/test2.js"]
}
},
{
"type": "mocha",
"title": "Phrasing",
"suite": true,
"configuration": {
"paths": [__dirname + "/mocha/test3.js"]
}
}
]
};
The output for this configuration is:
Everything
✓ should know the answer to life, the universe, and everything
The End
✓ should print something
Calculations
1) should find all there is
✓ should calculate the worst first million years for marvin
Phrasing
✓ should not panic
✓ should panic
1) Calculations should find all there is
54 == 42
AssertionError: 54 == 42
at Context.<anonymous> (.../mocha/test2.js:5:9)
... // Abbreviated
Great! That looks already a lot better.
But now, I want to run the first tests in parallel, and the last test should run when the first two tests are both completed. Simply wrap the first two tasks in an array literal. This will group the tasks together.
module.exports = {
"configuration": {
"reportManager": {
"reporter": [
{ "type": "Spec" },
{ "type": "List", "progress": false }
]
}
},
"tasks": [
[{ // <-----
"type": "mocha",
"title": "Everything",
"suite": true,
"configuration": {
"paths": [__dirname + "/mocha/test1.js"]
}
},
{
"type": "mocha",
"title": "Calculations",
"suite": true,
"configuration": {
"paths": [__dirname + "/mocha/test2.js"]
}
}], // <-----
{
"type": "mocha",
"title": "Phrasing",
"suite": true,
"configuration": {
"paths": [__dirname + "/mocha/test3.js"]
}
}
]
};
Let's run it again:
Everything
Calculations
✓ should know the answer to life, the universe, and everything
The End
✓ should print something
1) should find all there is
✓ should calculate the worst first million years for marvin
Phrasing
✓ should not panic
✓ should panic
1) Calculations should find all there is
54 == 42
AssertionError: 54 == 42
at Context.<anonymous> (.../mocha/test2.js:5:9)
... // Abbreviated
You can see that the tests are running in parallel now. However, the Spec
reporter prints the test-results as soon as it receives them from the test-clients, turning the output into a mess. This is the default behavior of most of the console reporters. We can change this behavior by setting the progress
property to false
. When this flag is set to false
, the reporter will organize all test-results in a tree, printing them in an ordered manner just when Preceptor completes all tests.
To make it a little bit more interesting, let's add also the Duration
reporter; it will print the total duration. The value of progress
for the Duration
reporter is by default false
, so no need to set this explicitly.
module.exports = {
"configuration": {
"reportManager": {
"reporter": [
{ "type": "Spec", "progress": false },
{ "type": "List", "progress": false },
{ "type": "Duration" }
]
}
},
// ...
};
The output is now:
Everything
✓ should know the answer to life, the universe, and everything
The End
✓ should print something
Calculations
1) should find all there is
✓ should calculate the worst first million years for marvin
Phrasing
✓ should not panic
✓ should panic
1) Calculations should find all there is
54 == 42
AssertionError: 54 == 42
at Context.<anonymous> (.../mocha/test2.js:5:9)
... // Abbreviated
Time: 531 milliseconds
The test-results are printed correctly now. Also, the duration is printed just as we wanted it.
With Preceptor, you can add as many reporters as you want, all of which are run at the same time - no matter if they are console or file reporters.
Let's add some file reporters: JUnit and TAP
module.exports = {
"configuration": {
"reportManager": {
"reporter": [
{ "type": "Spec", "progress": false },
{ "type": "List", "progress": false },
{ "type": "Duration" },
{ "type": "Junit", path: "junit.xml" },
{ "type": "Tap", path: "tap.txt" }
]
}
},
// ...
};
When running this configuration, two files will be created in the current working directory called junit.xml
and tap.txt
, holding JUnit and TAP results respectively.
Now, I want to collect code-coverage reports from all the tests that were run. For that, add a coverage
section to the global Preceptor configuration, activating it with active:true
, and may be supply additional configuration options.
module.exports = {
"configuration": {
"reportManager": {
"reporter": [
{ "type": "Spec", "progress": false },
{ "type": "List", "progress": false },
{ "type": "Duration" },
{ "type": "Junit", path: "junit.xml" },
{ "type": "Tap", path: "tap.txt" }
]
},
"coverage": {
"active": true, // Activate coverage
"path": "./coverage" // This is the default path
// Add here any other coverage options.
// See below for more information
}
},
// ...
};
The output is now:
Everything
✓ should know the answer to life, the universe, and everything
The End
✓ should print something
Calculations
1) should find all there is
✓ should calculate the worst first million years for marvin
Phrasing
✓ should not panic
✓ should panic
1) Calculations should find all there is
54 == 42
AssertionError: 54 == 42
at Context.<anonymous> (.../mocha/test2.js:5:9)
... // Abbreviated
Time: 572 milliseconds
=============================== Coverage summary ===============================
Statements : 96.3% ( 26/27 )
Branches : 100% ( 0/0 )
Functions : 92.31% ( 12/13 )
Lines : 96.3% ( 26/27 )
================================================================================
By default, the coverage reporter creates three reports:
text-summary
- The short coverage summary in the consolefile
- JSON file holding all coverage details, created in the path
directorylcov
- HTML report in the path
directoryThis concludes the introduction of Preceptor. Please read the general overview that follows, or refer to the API documentation that is included with this module.
Preceptor is started through the command-line that can be found in the bin
folder. You can run the application with
preceptor [options] [config-file]
The config-file
is by default rule-book.js
or rule-book.json
when left-off, where by the first one has priority over the second.
The command-line tool exposes a couple of flags and parameters:
--config j Inline JSON configuration or configuration overwrites
--profile p Profile of configuration
--subprofile p Sub-profile of configuration
--version Print version
--help This help
For profiles, see the profile section below.
The config
options adds the possibility to create or overwrite configuration directly from the console. The inline configuration-options are applied after the profile selection.
Objects are merged if a configuration file was selected, and arrays will be appended to already available lists.
Preceptor injects an object (PRECEPTOR
) into the gloabl scope.
PRECEPTOR.config
- Global configuration object from the Preceptor configurationThe supplied configuration information is very helpful for test customization. For example, you might want to supply a different base-url for tests run locally or on a CI system, or you would want to customize browser window sizes.
Here is an example for a Precpetor configuration file:
module.exports = {
"configuration": {
"reportManager": {
"reporter": [ { "type": "Spec", "progress": true } ]
},
"plugins": [ "preceptor-webdriver" ],
"settings": { // Settings needed by the test-client
"windowWidth": 1280, // Width of browser window
"windowHeight": 768, // Height of browser window
"webBaseUrl": "http://localhost" // Url of website
// May be some other options - add whatever you want here. Preceptor will ignore unknown options.
}
},
"tasks": [
...
]
};
In the client code then, you can gather this information like this:
var settings = PRECEPTOR.config.settings;
var activeWindow = browser.activeWindow();
activeWindow.resize(settings.windowWidth, settings.windowHeight);
activeWindow.navigator.navigateTo(settings.webBaseUrl + '/order/324';
To understand how Preceptor works, we have to understand first what the testing lifecycle is. Tests are run usually through a common set of states that can be defined as the testing lifecycle.
This include:
Each individual state will be triggered in Preceptor so that plugins can react on these state changes. This includes the reporter, the client, the client decorators, the tasks, and Preceptor itself.
Preceptor breaks them (and some additional states) into 4 groups:
Let's describe them in more detail by describing their function and what information in conveyed.
Administrative states will be triggered for events that change the collection and run behavior of Preceptor itself.
The start
state will be triggered just when Preceptor starts to run.
Parameters:
The stop
state will be triggered just when Preceptor completes all tests.
Parameters:
The complete
state will be triggered when Preceptor completed all the export jobs.
Parameters:
These states handle any changes to test-suites.
The suiteStart
state will be triggered when a new test-suite starts. Please be aware that it is possible to have test-suites within other test-suites.
Parameters:
id
- Identifier of test-suite. Preceptor creates a unique identifier for each state to be able to identifier related state changes.parentId
- Parent identifier of parent test-suite. This makes it possible for Preceptor to create a tree of tests and test-suites. For the first test-suite, this will be the id of the start
event.suiteName
- Name of the suite that should be used for reporting.The suiteEnd
state will be triggered when a test-suite completed.
Parameters:
id
- Identifier of test-suite, the same identifier that was given to suiteStart
.Test management states are states that transmit test and test-result information.
The testStart
state will be triggered when a new test starts.
Parameters:
id
- Identifier of test. Preceptor creates a unique identifier for each state to be able to identifier related state changes.parentId
- Parent identifier of parent test-suite. This makes it possible for Preceptor to create a tree of tests and test-suites. For the first test without a parent test-suite, this will be the id of the start
event.testName
- Name of the test that should be used for reporting.The testPassed
state will be triggered when a test has passed.
Parameters:
id
- Identifier of test that was given to the related testStart
.The testFailed
state will be triggered when a test has failed. A failure is an unexpected test-result, often triggered by assertions.
Parameters:
id
- Identifier of test that was given to the related testStart
.message
- Short description of the failurereason
- Detailed description of the failure (often with stack-trace)The testError
state will be triggered when a test had an error. An error is an unexpected exception that was not triggered by an assertion.
Parameters:
id
- Identifier of test that was given to the related testStart
.message
- Short description of the errorreason
- Detailed description of the error (often with stack-trace)The testUndefined
state will be triggered when a test hasn't been defined. This is triggered when the testing-framework recognizes a test, but could not find the test-implementation. Not all testing frameworks support this state.
Parameters:
id
- Identifier of test that was given to the related testStart
.The testSkipped
state will be triggered when a test has been skipped. Tests are usually skipped when for example sub-systems that are dependencies for test are not available.
Parameters:
id
- Identifier of test that was given to the related testStart
.reason
- Reason for skipping this test.The testIncomplete
state will be triggered when a test is available, but incomplete. This happens when tests are written, but are not completed yet.
Parameters:
id
- Identifier of test that was given to the related testStart
.The configuration file has three top-level properties:
configuration
- Global Preceptor configuration that describes Preceptor behavior.shared
- Shared task options.tasks
- Individual task options which are run by default in sequence.The configuration file supports multiple layers of profiles:
As an example, let's use the following abbreviated configuration:
config.js
module.exports = {
"configuration": {
"reportManager": { "reporter": [ { "type": "Spec" } ] }
},
"tasks": [
{
"type": "mocha",
"configuration": {
"paths": [__dirname + "/mocha/test1.js"]
}
}
]
};
Global profiles can be used to toggle between full-configurations:
module.exports = {
"profile1": {
"configuration": {
"reportManager": { "reporter": [ { "type": "Spec" } ] }
},
"tasks": [
{
"type": "mocha",
"configuration": {
"paths": [__dirname + "/mocha/test1.js"]
}
}
]
},
"profile2": {
"configuration": {
"reportManager": { "reporter": [ { "type": "Dot" } ] }
},
"tasks": [
{
"type": "mocha",
"configuration": {
"paths": [__dirname + "/mocha/test2.js"]
}
}
]
}
};
To select the second profile, call Preceptor as follows:
preceptor --profile profile2 config.js
The first profile can be selected by supplying profile1
instead, or whatever name you give the corresponding profiles.
Be sure to always supply a profile when it is available since the default behavior is to not look for a profile.
Task profiles are very similar to global profiles, but are there for toggling task-lists. This can be very useful, when global configurations are shared between multiple profiles. Task profiles are also called sub-profiles since they toggle configuration below the global profile selection.
An example looks like this:
module.exports = {
"configuration": {
"reportManager": { "reporter": [ { "type": "Spec" } ] }
},
"tasks": {
"sub-profile1": [
{
"type": "mocha",
"configuration": {
"paths": [__dirname + "/mocha/test1.js"]
}
}
],
"sub-profile2": [
{
"type": "mocha",
"configuration": {
"paths": [__dirname + "/mocha/test2.js"]
}
}
]
}
};
To select the first sub-profile, call Preceptor as follows:
preceptor --subprofile sub-profile1 config.js
Be sure to always supply a sub-profile when it is available since the default behavior is to not look for a sub-profile.
It is also possible to combine both profile methods.
module.exports = {
"ci": {
"configuration": {
"reportManager": { "reporter": [ { "type": "Dot" } ] }
},
"tasks": {
"acceptance": [
{
"type": "mocha",
"configuration": {
"paths": [__dirname + "/build/mocha/test1.js"]
}
}
],
"integration": [
{
"type": "mocha",
"configuration": {
"paths": [__dirname + "/build/mocha/test2.js"]
}
}
]
}
},
"dev": {
"configuration": {
"reportManager": { "reporter": [ { "type": "Spec" } ] }
},
"tasks": {
"acceptance": [
{
"type": "mocha",
"configuration": {
"paths": [__dirname + "/../mocha/test1.js"]
}
}
],
"integration": [
{
"type": "mocha",
"configuration": {
"paths": [__dirname + "/../mocha/test2.js"]
}
}
]
}
}
};
With this example, you could run the acceptance tests from your local machine with:
preceptor --profile dev --subprofile acceptance config.js
Preceptors task-independent behavior is configured in this section. It has the following properties:
verbose
- Flag for verbose output (default: false)reportManager
- Reporting configurationreportManager.reporter
- List of loaded reportersreportManager.listener
- List of loaded listenerscoverage
- Coverage optionplugins
- List of plugin modules to loadignoreErrors
- When set, errors that occur during testing will not be triggered on the Preceptor process (default: false)The report manager describes what reporters should be used and what it should listen for to receive testing lifecycle events from unsupported clients.
See the preceptor-reporter
project documentation for more information on the reportManager
property.
Coverage report collection can be configured in this section. It has the following options:
active
- Flag that turns coverage collection on or off (default: false - off)root
- Root of coverage data. This is used to reduce depth of coverage paths.path
- Path to where the file coverage reports should be exported to.reports
- List of reports to export to. (default: ['file', 'lcov', 'text-summary'])includes
- List of patterns for files to include scripts into the coverage report. (default: ['*/.js'])excludes
- List of patterns to exclude files and directories of scripts that were previously whitelisted with the includes
option. (default: ['/node_modules/', '/test/', '/tests/'])This feature uses Istanbul for collecting and merging coverage reports. See the Istanbul project website for more information.
Additional Reports
Preceptor adds the file
report to the list of reports to be able to disable the export of the JSON data. The file
value will not be given to Istanbul since it is not available there.
Any value that is assigned to the 'shared' object in the configuration root will be assigned as default for all task options. Task options then can overwrite these values. This gives a developer the opportunity to set own default values for properties, overwriting the default values that were given by the system.
The Preceptor system supports the loading of custom plugins by installing the modules through NPM, and by adding the module name to this list. Preceptor then tries to load each of these modules by executing the loader
function on the exported module interface, giving it the instance of Preceptor itself. The plugin can then register itself to Preceptor. See the preceptor-webdriver
project for an example.
The following tasks for testing-frameworks are supported by default:
Preceptor implements also some general purpose tasks:
The following sections describe the configuration and usage of these tasks
Tasks have a common set of configuration options that can be set on the root of the task. Any custom configuration option, that will be listed for each task below, should be set on the configuration
object instead of the root (see "Object and List Configuration"). The reason for this separation is flexibility for new features in future version of Preceptor without breaking any task or custom task that might add similar named options.
taskId
- Identifier of task. If none is given, then Preceptor will assign an identifier and may use it in error reports.type
- Identifier for task-plugin. For example, type:'mocha'
will use the Mocha task-plugin.name
- Name of task. This is a more user friendly version of taskId
. If none is given, then it uses the taskId
value.title
- Description of task that might be used in test-results.configuration
- Custom configuration for task-specific options. See below for the task options.decorators
- List of client decorators that should be loaded on the client. See "Client Decorators" section for more information.active
- Flag that defines if task is active or inactive. Inactive tasks will be ignored by Preceptor, skipping any tests and results from the task. (default: true)suite
- Flag that defines if task should be used as a virtual test-suite. A virtual test-suite injects itself into the test-results. This makes it possible to compose the test-results however it is needed. (default: false)debug
- Flag that defines if task is run in debug mode. In debug-mode, the client is run directly in the Preceptor process. Output is not caught by Preceptor and directly printed to std-out, and a breakpoint will stop the Preceptor process. Avoid using this flag long-term. (default: false)report
- Flag that determines if task uses the globally configured reporter. (default: true) If deactivated, testing life-cycle events will be muted. coverage
- Flag that determines if task uses the globally configured coverage collector. (default: false) If deactivated, collected coverage won't be merged verbose
- Prints every step taken by Preceptor. This adds a lot of output and might be overwhelming at first. (default: false)failOnError
- Flag that defines if task should skip all other tests and fail Preceptor. (default: false)echoStdOut
- Flag that defines if task should echo all std-out data.echoStdErr
- Flag that defines if task should echo all std-err data.The type
-value for this task is cucumber
.
Task options:
path
- Path to the tests (required)tags
- List of tags to accept. These can be strings with one tag, or an array of tags. These tags should have an '@' and may be the '~' if required. See Cucumber.js documentation for more information.format
- Output format for test results (default: 'progress')functions
- A list of functions to import (still WIP). The function can have a name
property to be used in verbose logging.coffeScript
- Flag that defines that output should be in coffee-script.The type
-value for this task is mocha
.
Task options:
reporters
- Reporter to use (default: 'spec')ui
- API interface to use (default: 'bdd')colors
- Use colors in output (default: true)inlineDiff
- Inline-diff support (default: false)slow
- Defines how many milliseconds is considered as a slow test (default: 75)timeOuts
- Specifies if time-outs should be enforced (default: true)timeOut
- Describes when a test is considered to be in time-out (in milliseconds) (default: 2000)bail
- Stop execution on the first error (default: false)grep
- Filter to use for tests (default: false - off)invert
- Invert grep
filter (default: false)checkLeaks
- Checks for memory leaks (default: false)asyncOnly
- Enforce all tests to be asynchronous (default: false)globals
- List of defined globalspaths
- A list of paths to the tests (default: ['test'])functions
- A list of functions to import as tests. The function can have a name
property to be used in verbose logging.recursive
- Recursive search in the test paths (default: false)require
- List of files to be require'd before tests run.sort
- Sort tests before running the tests (default: false)The type
-value for this task is kobold
.
Task options:
verbose
- Verbose output (default: true)failForOrphans
- Flag that defines if Kobold should fail tests when Orphans are encountered. Orphans are screens that were previously approved, but are missing in the most recent test-run. This can be the case when tests are removed, or when the testing-framework was interrupted. (default: true)failOnAdditions
- Flag that defines if Kobold should fail tests when Additions are encountered. Additions are screens that have never been seen (approved) before, but that are available during the most recent test-run. These usually happen when tests are added; these screens should be reviewed before approving. (default: true)build
- Identifier for the current build. This can be an identifier from a CI or a random generated one. This value is used to distinguish multiple test-runs from each-other, and is given to the remote storage. (default: process.env.BUILD_NUMBER or process.env.USER + blinkDiff
- Configuration for the blink-diff tool.mocha
- Mocha options for Kobold tests since it is build on-top of Mocha. (See above)storage
- Storage configuration for Kobold. See the project website for more information.source
- Source of screens. Uses storage
as default. See the project website for more information.destination
- Destination for screens. Uses storage
as default. See the project website for more information.The storage configuration has also the following values by default:
type
- 'File'options.approvedFolderName
- 'approved'options.buildFolderName
- 'build'options.highlightFolderName
- 'highlight'This task imports already available test-report files, including JUnit and TAP files. The type
-value for this task is loader
.
Task options:
format
- Format of file-import (i.e. junit
, tap
or istanbul
). See the Preceptor-Reporter project for all available options.path
- Glob to select files that should be importedconfiguration
- Custom configuration for each file-format typeExample:
{
"type": "loader",
"title": "JUnit import",
"suite": true, // Wrap it in a suite using the title from above
"configuration": { // Loader-task specific configuration
"format": "junit", // Use "junit" as the format (is default)
"path": "junit-*.xml", // Glob to select import files
"configuration": {
// Custom JUnit loader configuration
"topLevel": false
}
}
}
The type
-value for this task is node
.
Task options:
path
- Path to JavaScript file that should be executed. The JavaScript file will be running in its own process.The type
-value for this task is shell
.
Task options:
cwd
- Defines the current working directory for the script.env
- List of key-value pairs (in an object) of environment variables set for the shell script. (default: {})cmd
- Command to be executed in shell. The command will be run in its own process.The type
-value for this task is group
.
Task options:
parallel
- Flag that defines if tasks should be run in parallel (default: false)tasks
- List of task-options that should be run within this group.The group
Task Decorator uses this task to add the array-literal feature to the configuration.
A task decorator is a plugin that can modify task-options before they are applied. This makes the configuration very flexible as the whole configuration scheme can be changed with these plugins.
The following task decorators are build-in:
group
- Adds the array-literal feature for running tasks in parallel. It replaces the array-literal with a group that has the parallel
flag set, injecting all the tasks into the task
options of the group-task.identifier
- Makes sure that identifiers are given for taskId
, name
, and title
. taskId
will receive an unique id if not given, and all others might inherit the value if they are also missing - name
from taskId
and title
from name
.Client decorators are "tasks" that will be run in the client process by the client-runner. The decorators attach themselves to test-hooks that are triggered during the testing lifecycle (see above).
There are two types of hooks:
There are four activity hooks:
processBefore
- Triggered once before a suite of tests.processBeforeEach
- Triggered before each test-case.processAfterEach
- Triggered after each test-case.processAfter
- Triggered once after a suite of tests.Out of the box, Preceptor supports the following client decorators:
plain
- Prints each of the triggers to the console. This client-decorator should be used for debugging purposes when creating client-decorators.The preceptor-webdriver
plugin adds the WebDriver client-decorator for injecting Selenium setup/tear-down code into the client.
Client decorators can be added to every task that supports the testing lifecycle events (generally all testing frameworks), and it is added through the decorators
list in the task-options.
Client decorators has the following common options:
type
- Name of the plugin for client decoratorsconfiguration
- Client decorator specific configurationThe general client decoration configuration namespace is reserved for future changes to all client decorators. Client decorator specific configuration should be described in configuration
.
Clients are generally run in its own process. Each of these processes communicates with Preceptor, making it also possible to instruct the client to run client decorators in its process. For this to work, each testing-framework needs a client-runner. This is abstracted away, but is required when writing a plugin. The following client-runners are available:
cucumber
- Runs Cucumber.jsmocha
- Runs Mochakobold
- Runs Kobold. Uses internally the mocha
client-runner.node
- Runs Node.JS scriptsPlease see the above files in the source-code for details on how to implement these. Much of the common behavior and the communication is implemented in the client.js
file.
The plugin module naming should follow the Grunt plugin naming convention. Plugins contributed by the community should have "contrib" in the module name (example: preceptor-contrib-hello-world
). Plugins that are supported by the Preceptor team will be named without the "contrib" keyword (example: preceptor-webdriver
).
Generate the documentation with following command:
npm run docs
The documentation will be generated in the docs
folder of the module root.
Run the tests with the following command:
npm run test
The code-coverage will be written to the coverage
folder in the module root.
The following third-party libraries are used by this module:
The MIT License
Copyright 2014 Yahoo Inc.