require: [] but long-running in the main process and receives notifications

[jamiebuilds]

Description

I'm trying to build some tooling that runs alongside Ava, I can run it as a separate process, but I need to setup my own file watcher and do lots of work to make it work well alongside Ava.

It would be great if Ava exposed a hook to:

• Load a module
• Let it run for the lifetime of Ava
• Allow it to see what mode Ava is running in (watch or nah)
• Send it notifications when different things happen:
  • Event: Started running tests
  • Event: Completed running tests - with results
  • Maybe more as you see fit

Rough draft:

import onExit from 'signal-exit';

export default function fn(ava) {
  if (!ava.isWatchMode) return;

  ava.on('start', data => {
    // tests have started running
  });

  ava.on('complete', data => {
    console.log('Failed tests', data.failed.length);
    // ...
  });

  onExit(() => {
    // if cleanup is needed
  });

  return () => {
    // or if Ava wants to build in cleanup
  };
}

[novemberborn]

Interesting. Do you see this as a side-kick that runs in the main AVA process, without needing a complete programmatic API? The changes in #1722 would come in handy for this.

[jamiebuilds]

I could either use it in the main process or as another process that Ava starts up. I figure it's a little easier to build in the main process though.

I don't think it needs much of a programmatic API, maybe that can be expanded upon in the future.

It might be interesting to expose some way to start or stop a test run or control what tests are being run based on some other logic. You could build an entire UI around Ava

[novemberborn]

Cool, I'm 👍 on this.

We'd need to decide what configuration syntax is used to load this module. I think it's fine if it's restricted to configuration files only. It shouldn't be something you do on a one-off basis.

• We should load the module using esm
• The ava object should include the version number for compatibility purposes
• I'd prefer to return an optional cleanup function

What test results are you interested in? I'm loath to expose AVA's internals, at least at this stage, since I don't think that has necessarily stabilized yet.

[jamiebuilds]

Modified from: https://github.com/avajs/ava/issues/1768#issuecomment-381214630

type Data = {
  ...,
  snapshots: Array<{
    name: string,
    expected: any,
    actual: any,
    passed: boolean
  }>
};

ava.on('complete', (data: Data) => {
  browserWs.send(JSON.stringify(data.failedSnapshots));
});

This is basically all I care about. As long as I can get a list of the snapshots that failed in a way that I could later tell Ava to go update them, that would be enough for me

[novemberborn]

That's not really the data we communicate back to the main process. What we can expose now:

• test file
• test title
• result
• typeof first failing assertion
• richly formatted failure explanation

The latter is a string. We could also provide the concordance-serialized value, but I worry that might be unnecessarily expensive so may have to be an opt-in.

If we also expose a function for resolving the snapshot location based on the test file, you'd have enough to build on top of.

[jamiebuilds]

Could I subscribe to specifically snapshots?

[novemberborn]

We have various issues about running all assertions, and communicating all assertion results up to the main process: #261, #1356, #1330, #324. I'd be more than happy to expose those to this side-kick. You could then filter out the snapshots.

[novemberborn]

The work I'm doing with https://github.com/avajs/ava/pull/2478 will allow us to run a worker thread. That said, I don't think our internal events are stable enough to expose to user code.

It's been two years since this was opened, I don't think we'll get to this for a while still, so I'm going to close this issue, sorry.