Run code before and after a group of test files

[pvdlg]

Description

The before and after hooks are really useful, but they are limited to a given file.

When doing integration test we often have to do some relatively long set up tasks such as starting a database, a server or a docker container. As those tasks take often several dozen of seconds or a few minutes, we want to do them only once per test run.

As before and after are limited to the file in which they are defined, it forces us to write all the integration tests in the same file. This is not ideal as it ends up creating a giant file that is difficult to maintain.

So it would be great to be able to define have a special before and after hook that spans a group of file.

I'm not really sure what would be the best API, but I can think of two solutions.

t.beforeAll(scope, [function]) and t.afterAll(scope, [function])

Those special version of beforeAll and afterAll could be defined in any test file (even by themselves without other tests) and the function is optional.

All the test files that have a beforeAll or an afterAll for a given scope would have to be run only after the function of all the beforeAll of the same scope ran. And it will run the function of all afterAll only when the files having the same scope are completed.

// integration/mocks.js
test.beforeAll('database', async () => {
  await startDatabase();
});
test.beforeAll.always('database', async () => {
  await stopDatabase();
});

// integration/scenario1.js
test.beforeAll('database')

test.serial('First step of scenario 1' async t => {
  // Do something with the database
}

test.serial('Second step of scenario 1' async t => {
  // Do something else with the database
}

test.beforeAll.always('database');

// integration/scenario2.js
test.beforeAll('database')

test.serial('First step of scenario 2' async t => {
  // Do something with the database
}

test.serial('Second step of scenario 2' async t => {
  // Do something else with the database
}

test.beforeAll.always('database');

after.js and before.js in a diretory:

.
+-- test/
|   +-- integration
|       +-- before.js
|       +-- scenario1.js
|       +-- scenario2.js
|       +-- scenario3.js
|       +-- after.js

ava would run before.js, then all the tests in scenario1.js, scenario2.js, scenario3.jsthe same way it does now (in parralel or serially is.serialis used) then runafter.js`.

In other word, ava would guarantee that the files scenario1.js, scenario2.js, scenario3.jsare run only oncebefore.jsresolves and thatafter.jsis called only when all the tests inscenario1.js,scenario2.js, scenario3.js are completed.

[novemberborn]

Have you considered wrapping the ava invocation in a shell script that can do the setup and teardown? Either explicitly or through npm pretest and posttest lifecycle scripts?

[pvdlg]

That would be possible but that would make it impossible to use ava directly during development, for example to select which test to run during with ava <files_to_test>.

I found a workaround that kind of works in the current version:

With the following:

.
+-- test/
|   +-- integration
|       +-- index.test.js
|       +-- scenario-1.test.js
|       +-- scenario-2.test.js

// test/integration/index.text.js
import test from 'ava';
import delay from 'delay';
import requireGlob from 'require-glob';

test.before(async () => {
  console.log('Setup start');
  await delay(1000);
  console.log('Setup complete');
});

requireGlob(['./_*.test.js']);

test.after(async () => {
  console.log('Tear down start');
  await delay(1000);
  console.log('Tear down complete');
});

// test/integration/scenario-1.text.js
import test from 'ava';
import delay from 'delay';

test('Scenario 1 - Test 1', async t => {
  await delay(400);
  t.is(true, true);
});

test('Scenario 1 - Test 2', async t => {
  await delay(100);
  t.is(true, true);
});

// test/integration/scenario-2.text.js
import test from 'ava';
import delay from 'delay';

test('Scenario 2 - Test 1', async t => {
  await delay(200);
  t.is(true, true);
});

test('Scenario 2 - Test 2', async t => {
  await delay(300);
  t.is(true, true);
});

On ava -v I obtain:

Setup start
Setup complete
  ✔ integration › index › Scenario 1 - Test 2 (104ms)
  ✔ integration › index › Scenario 2 - Test 1 (200ms)
  ✔ integration › index › Scenario 2 - Test 2 (300ms)
  ✔ integration › index › Scenario 1 - Test 1 (404ms)
Tear down start
Tear down complete

  4 tests passed

if I use t.test.serial it also work as expected.

The problem is that if define t.before, t.after, t.beforeEach or t.afterEach hooks they are executed within the context of index.test.js. That mean a t.beforeEach defined in scenario-1.test.js will be executed before each test in both scenario-1.test.js and scenario-2.test.js. But as long as all the scenario share the same t.before, t.after, t.beforeEach or t.afterEach hooks its ok.

So, yes there is workarounds that are not too bad. But having such feature in the core would still be nice, even in a simpler than what I proposed.

But feel free to close if you think the improvement of having that in the core vs a workaround doesn't worth the effort/extra code/extra maintenance.

[novemberborn]

That would be possible but that would make it impossible to use ava directly during development, for example to select which test to run during with ava .

You'd have to forward the glob patterns through your (bash) script, much like you're doing in your JavaScript workaround.

So, yes there is workarounds that are not too bad. But having such feature in the core would still be nice, even in a simpler than what I proposed.

But feel free to close if you think the improvement of having that in the core vs a workaround doesn't worth the effort/extra code/extra maintenance.

This would require more coordination between workers than I'd be comfortable with. #1366 is a more generic approach that might work for your use case, but it's very low priority and needs more consideration before we'd commit to it.

Thanks for sharing your use case!

[Fmajor]

I tried the requireGlob workaround, but got Error: All tests and hooks must be declared synchronously in your test file, and cannot be nested within other tests or hooks. Is there any other working workaround for the global before and after feature?

[Fmajor]

I just use the chokidar package to watch file changes and concat all test files into a single one. That's really ugly but work for me. Hope to have the global before and after feature...

[LeBovin]

It's definitely something Ava should have

[sramam]

I needed this capability (to configure/start/recreate a server against which to test). This is the solution I hacked together:

import { run as avaCli } from "ava/lib/cli";

// tslint:disable-next-line:no-empty-interface
interface IGlobalContext{
}

async function run() {
  const { globalContext} = await beforeAll();
  let err;
  try {
    await avaCli();
  } catch (err_) {
    err = err_;
  } finally {
    await afterAll(globalContext, err);
  }
}

// tslint:disable: no-console
async function beforeAll() {
  // this runs before any ava is initialized
  const globalContext = {
  };
  return { globalContext };
}

async function afterAll(globalContext: IGlobalContext, error) {
  // runs after all AVA tests are completed successfully.
  if (error) {
    throw error;
  }
}

if (!module.parent) {
  run().catch(console.error);
}

Suppose this file is stored in src/test/helpers/ava.ts and compiles to dist/test/helpers/ava.js. Replace occurances of ava in package.json:scripts with node dist/test/helpers/ava.js - which works cross platform. All command line options are passed to and consumed by avaCli().

In my case, I needed programmatic access to the module-under-test, and this worked better than a (bash/shell) script.

I would also prefer this to be provided by ava directly with a well-published location/mechanism to look for beforeAll/afterAll functions, but understand that might not be a priority or even desire of core-team. This is a tad hacky, but got the job done.

YMMV.

[jakobrosenberg]

I'm currently working on a new public template for Routify projects and this is part of the current package.json.

"scripts": {
    ...
    "test": "npm run test:dev && npm run test:build",
    "pretest:dev": "node test/scripts setupDev",
    "test:dev": "ava test/{common,dev}/**/*.test.js",
    "posttest:dev": "node test/scripts teardown",
    "pretest:build": "node test/scripts setupBuild",
    "test:build": "ava test/{common,build}/**/*.test.js",
    "posttest:build": "node test/scripts teardown",
}

Unfortunately post hooks don't run after exiting when using -- --watch. Furthermore, speaking from experience, the amount of test scripts will not be well received and I'm looking for an alternative approach that would allow me to shorten it to:

"scripts": {
    ...
    "test": "npm run test:dev && npm run test:build",
    "test:dev": "ava test/{common,dev}/**/*.test.js",
    "test:build": "ava test/{common,build}/**/*.test.js",
}

The preferred solution would be global before and after hooks.

Working within the current constraints of AVA, I suspect I'm forced to either pollute the package.json scripts with partially broken hooks or abstract the test scripts into a separate file ("test:dev": "test/scripts/run-ava.js"), but then I lose the transparency and direct access to the CLI.

[novemberborn]

@jakobrosenberg this is something I had in mind with shared workers, could you share your use case in https://github.com/avajs/ava/issues/2605?