Parallel execution - Allow programmatic registration of per-scenario cleanup hooks

[jan-molak]

🤔 What's the problem you're trying to solve?

To enable the cleanup of resources used during a test scenario (closing web browsers, disconnecting database connections, etc.), Serenity/JS programmatically registers an artificial After hook. This hook returns a Promise which gets resolved when the cleanup is done.

This approach relies on the fact that the Serenity/JS test formatter is initialised in the same Node process as the test scenarios to be executed, which means that the artificial After hook gets merged with user-defined step definitions.

However, when Cucumber.js is executed in parallel mode, Serenity/JS formatter gets initialised in a separate process, which means that Cucumber worker processes are not aware of the cleanup hook and can't execute it.

This makes it impossible for Serenity/JS users to use the framework with Cucumber.js parallel execution mode at the moment (serenity-js/serenity-js#1308).

Note that this problem doesn't affect Serenity/JS + Cucumber.js combo executed by an external parallel runner like WebdriverIO or Protractor, since those runners attach the reporter to the process executing the tests.

✨ What's your proposed solution?

I do like the idea of keeping the reporter process separate from the runner processes and using Cucumber messages to communicate between the worker and the coordinator processes. It makes the overall design cleaner and makes the Cucumber.js execution model similar to how Playwright Test works (which Serenity/JS already supports). I wouldn't change anything about that.

What I'd like to be able to do, though, is instruct Cucumber to perform the additional programmatically-defined clean-up step whenever the worker finishes the test scenario.

Ideally, I'd like to be able to do it from the existing Serenity/JS formatter, without having to write a whole new wrapper around Cucumber (just yet).

⛏ Have you considered any alternatives or workarounds?

My target solution will most likely be to write a Serenity/JS CLI wrapper around Cucumber, as per https://github.com/cucumber/cucumber-js/discussions/2041, assuming that the new programmatic Cucumber API will allow me to register custom After hooks.

Unfortunately, I don't have the capacity to explore that properly yet and would like to enable Serenity/JS community to benefit from Cucumber parallel execution mode before that's complete, if possible.

Alternatively, we could also introduce a concept of per-worker reporter in Cucumber, that's separate from the main reporter.

[davidjgoss]

Sorry for the late reply @jan-molak! Just wanted to clarify a couple of things:

• Does your After hook need context/state from the specific worker? If yes is that serializable?
• Does your After hook need context/state from the parent process that calls Cucumber? If yes is that serializable?

[jan-molak]

Hey @davidjgoss - no problem at all, thanks for getting back to me!

The After hook needs a Serenity/JS "scene ID", which is a serialisable value initialised by Serenity/JS Cucumber reporter on the serenity instance when the reporter receives a Cucumber testCaseStarted message.

So to access this value, the hook could:

• have access to the same serenity instance the reporter uses (which requires them to be in the same process..)
• received the scene ID by some other means

Thinking about it, scene ID is a serialisable value that's initialised in the process where the reporter is running. This is not ideal, but is currently the only place where I can do it.

What would be ideal, is if plugin developers could have a way to define per-worker Before and After hooks for Cucumber to attach to every worker, and then use the Cucumber message protocol to send custom data from such hooks to reporters.

This way I could:

• attach a per-worker Before hook that initialises a scene ID when the Cucumber scenario starts in the worker
• use Cucumber message protocol to send the scene ID from the Before hook in the worker to the reporter (maybe attach custom data to the testCaseStarted message? is that possible?)
• use the After hook in the reporter to do the clean-up (this is an async process, so the After hook should be able to wait for a promise returned from Serenity/JS cleanup).

Other than the scene ID, the per-worker Before and After hooks won't need any other data from the worker or the parent process. Metadata like the worker process id or some worker identifier could be a nice thing to attach to the report, but they're not necessary.

Of course, such programmatic Before hook would need to be executed before any user-defined Before hooks, and the programmatic After hook would need to be executed after any user-defined After hooks.