--retry N flag

[jonfinerty]

Hello,

We are using cucumber to run end to end tests which include flaky 3rd party services. By using v3.0.0.pre.1 of cucumber I can use the --retry X flag to rerun tests X times. Functionally this works fine, failed tests are retried. However when it comes to reporting this to TeamCity I encounter a problem. Each test attempt is reported, so if one attempt fails and the retry succeeds the the build itself fails.

This is confused a bit further because tests run on different devices also show under the same test. So here is what running 1 test, on 2 devices, failing the test on the first attempt and passing on the retry attempt (on both devices).

So for us, this scenario has passed at least once on each device, so we would consider it as green, but we are getting red.

Any thoughts? I realise that this is an edge case, and yes you shouldn't have flaky tests but that's where we are at the moment.

Thanks,

Jon

[kevinrood]

@jonfinerty as it turns out, I'm unfamiliar with how the --retry X option affects formatting. I would have to have a consistently failing scenario (for example: fail a scenario twice, then allow it to pass) to be able to inspect the cucumber object model at run-time (which is how I have had to do things in the past).

[jonfinerty]

If I can find the time, I'll come up with a test case for you. I was mainly testing the waters to see if you think this feature (and the logic of any single run passing == green) fits in the reporter. Thanks