Align Cluecumber reporting to the official Cucumber BDD ruleset

[bischoffdev]

• Find out if scenarios should fail on AfterStep failure of the last step?
• Research an official document describing the various step and hook result combinations to consider passed, failed and skipped

[bischoffdev]

Contacted Cucumber OSS Community in order to clarify this. Waiting for replies.

[bischoffdev]

If an after hook fails, then Cucumber-Ruby, Cucumber-JVM will mark that scenario as failed in the summary, that is a summary like:

1 Scenario (1 Failed)
3 Steps (3 Passed)

[bischoffdev]

Consider step hook failures and skipped steps in calculating scenario failures:

• A scenario in which a beforeStep or afterStep hook fails -- FAILED
• A scenario in which the last step passes but has a failing afterStep hook -- FAILED
• A scenario which consists of only passing and skipped steps -- PASSED

Don't modify step results when step hooks fail!

[bischoffdev]

Implemented and ready for release.

[mpkorstanje]

@bischoffdev this doesn't look correct

scenario which consists of only passing and skipped steps

-- PASSED

The result of a scenario is the worst result of it's steps (hook steps and pickle steps are both steps). So this should be skipped.

[bischoffdev]

Thanks for the comment, @mpkorstanje ! Is skipped considered worse than passed in any case? If yes, then I need to revise this. It was probably due to us never having skipped steps without prior failures. Or all skipped in case of a deliberate scenario skip. I will check and change this depending on your input.

[mpkorstanje]

Yes. The order is exactly this:

https://github.com/cucumber/cucumber-jvm/blob/main/cucumber-java/src/main/java/io/cucumber/java/Status.java

So you can use Collections.max or Stream.max on the collection of all states to find it. If there are no steps, the scenario passes by definition.

All steps van be skipped by using Junits Assume or Assumption class in the first step.