perjury

perjury is a vows.js work-alike library. Making false vows is perjury.

The motivation is to make the internals of the test framework clearer, so when your tests are mysteriously failing, you have some idea why.

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Example


// You should be able to do this

var vows = require('perjury');

// perjury does not pollute the assert module namespace by default, but
// this should give you the same behaviour

var assert = vows.assert;

// This should look a lot like your regular tests if you're already a vows
// user. If you're not, welcome!

vows
  .describe("My first vows test")
  .addBatch({
    'When we open a file': {
      topic: function() {
        fs.open("/tmp/fakefile", "w", this.callback);
      },
      'it works': function(err, fd) {
        assert.ifError(err);
        assert.isNumber(fd);        
      },
      teardown: function(fd) {
        fs.close(fd, this.callback);
      }
      'and we write to the file': {
        topic: function(fd) {
          fs.write(fd, "My dog has fleas\n", this.callback);
        },
        'it works': function(err, written, buffer) {
          assert.ifError(err);
          assert.greater(written, 0);
          assert.isString(buffer);        
        }
      }
    }
  })
  .run();

Introduction

perjury is an attempt to make a version of vows where I know why my tests fail. This has been surprisingly hard with vows.

Requiring

You require the module like any other module. If you have a lot of code that uses the vows module, you should probably be able to just require the perjury module and be done with it.

Assert macros

However, unlike vows, perjury will not pollute the namespace of the built-in assert module by default. Instead, it exports an assert property with the macros you want as properties.

So, if you use the assert macros from vows, you should change your code that looks like this:

var vows = require('vows');
var assert = require('assert');

to this:

var vows = require('perjury');
var assert = vows.assert;

Data structures

The basic way to use vows-like tests is to build really large hierarchical objects with a particular well-defined form.

Batch

For perjury, the core concept is the test batch. A batch is an object that consists of the following:

A topic function that generates values to be tested
One or more test functions, which accept the results of the topic and use assert macros to validate the results
Zero or more sub-batches
An optional teardown function that cleans up any values generated by the

A batch can be either synchronous or asynchronous. For a synchronous batch, the topic function just returns a value, and the test functions measure that value:

let batch = {
  "We get the answer":  {
    topic() {
      return 6 * 7;
    },
    "it equals 42": (err, answer) => {
      assert.ifError(err);
      assert.equal(answer, 42);
    }
  }
};

For an asynchronous batch, the topic returns its results through the callback property of this. perjury knows that the callback will be used because the result returned by the topic function is undefined.

let batch = {
  "When we get the answer asynchronously":  {
    topic() {
      setImmediate(() => {
        this.callback(null, 6 * 7);
      });
      return undefined;
    },
    "it equals 42": (err, answer) => {
      assert.ifError(err);
      assert.equal(answer, 42);
    }
  }
};

Alternately, a topic can return a Promise. Perjury will resolve the returned Promise and call tests with the same (err, results) format as with other types of call.

let batch = {
  "When we get the answer":  {
    topic() {
      return new Promise((resolve, reject) => {
        fs.open("/tmp/testfile", "w", (err, fd) => {
          if (err) {
            reject(err);
          } else {
            resolve(fd);
          }
        })
      });
    },
    "it equals 42": (err, fd) => {
      assert.ifError(err);
      assert.isNumber(fd);
    }
  }
};

Note that all test functions receive at least an err argument, and then one or more arguments. Synchronous batches can only have one test argument; asynchronous batches can have a lot.

For backwards compatibility, it's possible to call this.callback synchronously in your topic. Perjury will simply call setImmediate to call the callback later. But that is a tricky and confusing way to write your tests, and you should probably avoid it.

A batch can also have sub-batches. These are just properties of the batch that are also batch objects, with their own topic, tests, sub-batches, teardown, etc. The argument to the topic will be the results of the parent batch, in reverse order up the hierarchy.

let batch = {
  "When we get the answer":  {
    topic() {
      return 6 * 7;
    },
    "it equals 42": (err, answer) => {
      assert.ifError(err);
      assert.isNumber(answer);
      assert.equal(answer, 42);
    },
    "and we ask a couple of questions": {
      topic(answer) {
        return ["What is six times seven?", "How many roads must a person walk down?"];
      },
      "they look plausible": (err, questions) => {
        assert.ifError(err);
        assert.isString(question[0]);
        assert.equal(question[0][question[0].length - 1], '?');
        assert.isString(question[1]);
        assert.equal(question[1][question[1].length - 1], '?');
      },
      "and we compare the answer and the question": {
        topic(questions, answer) {
          setImmediate(() => {
            this.callback(null, questions[0], questions[1], answer);
          });
          return undefined;
        },
        "they match up well": (err, question0, question1, answer) => {
          assert.ifError(err);
          // NB: you need to implement isAnswerTo yourself
          assert(isAnswerTo(answer, question0));
          assert(isAnswerTo(answer, question1));
        }
      }
    }
  }
};

Note that if a batch's topic returns more than one value to its callback, they will be provided in order for any sub-batches' topic, but hierarchically in reverse order. This may be a little confusing.

Note also that if an error occurs, in either the topic or the tests, the sub-batches will not be run.

The teardown method is called after all the tests and sub-batches have been run. So, the order is something like this:

topic
tests
sub-batches (if there are no errors)
teardown

The teardown gets the non-error results of the topic as arguments. It's useful for cleaning up things that the topic made a mess of.


batch = {
  'When we open a file': {
    topic: function() {
      fs.open("/tmp/fakefile", "w", this.callback);
    },
    'it works': function(err, fd) {
      assert.ifError(err);
      assert.isNumber(fd);
    },
    teardown: function(fd) {
      fs.close(fd, this.callback);
    }
  }
};

teardown functions can also be synchronous or asynchronous, or they can return a Promise. However, the results are ignored.

let batch = {
  "When we get the answer":  {
    topic() {
      return new Promise((resolve, reject) => {
        fs.open("/tmp/testfile", "w", (err, fd) => {
          if (err) {
            reject(err);
          } else {
            resolve(fd);
          }
        })
      });
    },
    "it equals 42": (err, fd) => {
      assert.ifError(err);
      assert.isNumber(fd);
    },
    teardown(fd) {
      return new Promise((resolve, reject) => {
        if (typeof(fd) != 'number') {
          reject(new Error("File descriptor is not a number"));
        } else {
          fs.close(fd, (err) => {
            if (err) {
              reject(err);
            } else {
              resolve();
            }
          })
        }
      });
    }
  }
};

Note that the teardown will be called regardless of whether errors happened or not, so it's a good idea to check the arguments to make sure they're valid.

Teardowns are called as soon as the batch finishes; this is different from how vows.js works, but it is better.

If you're using a version of node that can handle async/await syntax, (>= 7.10.1), you can use async functions in your topics and teardowns, which can make your aysnchronous test code about as lovely and compact as can be.


const fs = require('fs');
const util = require('util');

// util.promisify is available in node > 8.0.0

const open = util.promisify(fs.open);
const close = util.promisify(fs.close);

let batch = {
  "When we get the answer":  {
    topic: async function () {
      return await open("/tmp/testfile", "w");
    },
    "it equals 42": (err, fd) => {
      assert.ifError(err);
      assert.isNumber(fd);
    },
    teardown: async function (fd) {
      return await close(fd);
    }
  }
};

Suite

Batches are organized into suites. You create a suite with the describe method of vows.

const vows = require('perjury');

let suite = vows.describe('A new suite');

You can then add one or more batches to the suite using the addBatch method.

suite.addBatch(batch1);
suite.addBatch(batch2);
suite.addBatch(batch3);

Finally, you have two options to actually run the test suite. The first is the aptly-named run() method, which runs all the tests and reports the results to stdout. You can then run the script through node and you'll run all your tests.

Alternately, you can use the export() method, passing the current module as an argument. This will change the exports property of the module to be the run() method of the suite. In other words, the module will now export a single function that runs the suite.

The perjury command-line tool can be used to run all your test modules that use export().

./node_modules/.bin/perjury test/*.js

All the suite methods are chainable. The typical way to actually use this library, then, is to require vows, use the describe method to create a suite, use addBatch to add one or more batches, and then use export(module) or more rarely run() to run the suite.

const fs = require('fs');
const vows = require('perjury');
let assert = vows.assert;

vows.describe('Input/output tests')
  .addBatch({
    'When we open a file': {
      topic: function() {
        fs.open("/tmp/fakefile", "w", this.callback);
      },
      'it works': function(err, fd) {
        assert.ifError(err);
        assert.isNumber(fd);
      },
      teardown: function(fd) {
        fs.close(fd, this.callback);
      }
    }
  })
  .export(module);

CoffeeScript

CoffeeScript is a nice pre-processor for JavaScript. If you write your test scripts in CoffeeScript, it's totally OK to run them with the perjury command-line tool, as-is.

./node_modules/.bin/perjury test/*.js test/*.coffee

perjury uses the CoffeeScript package to load the test modules automatically.

Debugging

Test-driven development means roughly that write your tests first, then write the implementations, then keep running the tests till they work.

Unfortunately, vows.js can be really hard to use for TDD, because it doesn't give you a lot of information about where errors are happening. Sometimes it's your test code; at other times it's the code you're trying to test. The structures that vows.js and perjury use can be tricky to get right.

perjury doesn't necessarily do a fantastic job at this, but it's a little better, and it's definitely a goal. perjury uses the debug library to spoot out debug info to stderr at run time. This can be very useful for looking at how the perjury module is running, and figuring out where errors are happening.

To use it, define the DEBUG environment variable when running your tests:

DEBUG=perjury:* ./node_modules/.bin/perjury mytest.js

Watch this space for more help in doing TDD with perjury.

Compatibility

This version is incompatible with vows and previous versions of perjury in a few small ways.

vows will check the arity of test methods and call the method different ways based on that arity. With perjury, tests will always take an error argument and then zero or more result arguments. This should help preserve your sanity and make you write more robust tests.
vows will automatically pollute the namespace of the assert module. perjury makes you use a property instead.
vows handles all teardowns at the same time, without waiting for sub-batch teardowns to finish. perjury handles teardowns when the batch is finished, so you can do things like deleting created files in your sub-batch teardowns, and deleting their directory in your main batch teardown, and things will just work right.
vows and perjury 0.x treat a Promise returned from the topic just like any other results. So test functions will receive the Promise as a results argument. perjury 1.x will resolve the Promise and pass the results to the test instead. So, if your tests expect to receive a Promise passed synchronously, you should change that.

assert

The exposed assert module-ish object has a number of useful methods for doing tests. perjury avoids most of the namespace pollution problems that vows.js has, so you should feel OK using these.

The module exposes all the methods of the built-in assert module. It also has the following utility methods. Each will do a check and if the check fails, will throw a new AssertionError with either the message argument as its message, or a standard message for that macro.

assert.epsilon(eps, actual, expected, message)

Checks that the number actual is within eps from expected.

assert.match(actual, expected, message)

Checks that actual matches the regular expression expected. Note that actual will be coerced to a string if it is not one already.

assert.matches is a synonym.

assert.isTrue(actual, message)

Checks that actual is true (not just truthy; true).

assert.isFalse(actual, message)

Checks that actual is false (not just falsy; false).

assert.isZero(actual, message)

Checks that actual is 0.

assert.isNotZero(actual, message)

Checks that actual is not 0.

assert.greater(actual, expected, message)

Checks that actual is strictly greater than expected.

assert.lesser(actual, expected, message)

Checks that actual is strictly lesser than expected.

assert.inDelta(actual, expected, delta, message)

Checks that actual is less than delta away from expected. It's a lot like assert.epsilon().

assert.include(actual, expected, message)

Checks that actual contains expected. assert.includes is a synonym.

assert.notInclude(actual, expected, message)

Checks that actual does not contain expected. assert.notIncludes is a synonym.

assert.isEmpty(actual, message)

Checks that actual is empty (an empty array or an object with no properties).

assert.isNotEmpty(actual, message)

Checks that actual is not empty.

assert.isArray(actual, message)

Checks that actual is an array.

assert.isObject(actual, message)

Checks that actual is an object.

assert.isNumber(actual, message)

Checks that actual is a number.

assert.isBoolean(actual, message)

Checks that actual is a boolean (true or false).

assert.isNaN(actual, message)

Checks that actual is NaN.

assert.isNull(actual, message)

Checks that actual is null.

assert.isNotNull(actual, message)

Checks that actual is not null.

assert.isUndefined(actual, message)

Checks that actual is undefined.

assert.isDefined(actual, message)

Checks that actual is not undefined.

assert.isString(actual, message)

Checks that actual is a string.

assert.isFunction(actual, message)

Checks that actual is a function.

assert.typeOf(actual, expected, message)

Checks that actual is of type expected.

assert.instanceOf(actual, expected, message)

Checks that actual is an object and an instance of expected.

fuzzy-ai / perjury

readme