When running with --retry, exit code should be 0 if a failed scenario passes on retry

[rishi-freshbooks]

Summary

When running cucumber with the --retry flag, the process will exit with a non-zero code if the scenario fails on first try but passes on a retry.

Expected Behavior

Running cucumber with --retry flag should exit with code 0 if scenarios pass on first try or within the specified number of retries .

Context & Motivation

The non-zero exit code causes our build job in JenkinsCI to fail when it should report as success.

[rishi-freshbooks]

@mattwynne does this capture the issue appropriately? Is this something that can be addressed in v3.0.0.pre.1?

[mattwynne]

That's great thanks @rishi-freshbooks. We'll do our best to fix this in a future pre-release of v3, yes.

[streetlogics]

@mattwynne any progress on this? Just encountering this issue myself - I love that I finally got the --retry flag working in circleci (note: I had to put the command at the end of the line otherwise a --format option that had an --out param seemed to swallow the retry attempt), however, once the retry started "working", circle still saw the run as a failure. 😞

[streetlogics]

@mattwynne - I took a really ugly pass at this (https://github.com/cucumber/cucumber-ruby/compare/master...streetlogics:issue/retry-success-exit-code-0-1044) just to get something working. Definitely open to any thoughts you might have on improving this for the actual implementation

[mattwynne]

Awesome, thanks @streetlogics one of us will take a look ASAP

[garie]

Any update? My team is also interested in this fix as a non-zero exit code causes our Bamboo build to fail.

[garside]

Just ran into the --retry flag and the non-zero exit code for circleci. Just hoping to get a pulse check on the status of this retry-success feature that @streetlogics proofed.

@mattwynne, Is there an ETA for it and/or is there anything we could do to help push this across the finish line?

[brasmusson]

Yes, in non-strict mode the exit code should be 0 is all scenarios passes on retry, but in strict mode (option --strict) the exit code should not be 0 is any scenarios were retried.

I think a that first the summary should be updated with a new flaky category (as in 6 Scenarios (1 Passed, 1 Failed, 1 Undefined, 1 Pending, 1 Skipped, 1 Flaky)), then it is straight forward to implement the exit code logic. In non-strict mode the exit code should be 0 if and only if there no failed scenarios. In strict mode the exit code should be 0 if and only if there are no failed, flaky, pending or undefined scenarios.

@streetlogics patch does not handle the strict mode correctly, and as I just stated, I think that the exit code should be based on the summary (if the summary reports a failed scenario - then the exit code should be non-zero)

[reach2jeyan]

Flaky tests still fails the scenario, How can i force cucumber to have a pass if the scenario has passed during the re-run. I see this is the simplest thing I can implement comparatively with cucumber_failures.log or re-run

[xtrasimplicity]

@reach2jeyan have you tried using the flags defined in this (merged) PR?

[reach2jeyan]

@xtrasimplicity Yes! and still I get cucumber has failed with 1 flaky scenario and since the flaky scenario has previously failed

[xtrasimplicity]

@reach2jeyan Hmm, that's interesting. Could you please post a MCVE?

[prashanth-sams]

@xtrasimplicity but re-run failed tests are not going to be updated in the parent report; it will remain as a separate report with left-over failed scenarios

@mattwynne is there a way to get the consolidated report after re-run tests? or any work-around ignoring the failed tests if the tests are passed on --retry

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.