Add the ability to dynamically retry a test (or avoid retrying it)

[plocket]

🤔 What's the problem you're trying to solve?

We want to only rerun tests that fail because of a timeout, and possibly retry them 2 or 3 times. I'm frustrated when I have to retry all tests 2 or 3 times just to be able to rerun the tests that timed out.

✨ What's your proposed solution?

One possibility is to be able to override retries in the After() hook.

⛏ Have you considered any alternatives or workarounds?

Could try manipulating the object that's handed into the After() hook, but that seems like a bad idea from the recent feedback I've had.

📚 Any additional context?

---

This text was originally generated from a template, then edited by hand. You can modify the template here.

[davidjgoss]

It's been mentioned elsewhere in the Cucumber universe that users will always want their own kind of control over retries and thus we should make it a programmatic extension point rather than adding a broad set of first-party options.

Some considerations:

• Existing retry options (on or off, number of retries, tag filter) would need to be respected but perhaps allow overriding in this extension point.
• We may ultimately break those options out to be a separate plugin using this extension point
• The retry concept has made it into the messages protocol as part of the TestCaseFinished message - we should make sure any custom handling results in the correct value being emitted here

The first phase of the plugins system we're starting to work on will focus on coordinator stuff - filtering out scenarios etc. The next thing to look at will be runtime stuff and this may be a good first thing to tackle on that front.