search

bingnz / cucumber-teamcity-formatter

Formatter for cucumber-js to write test output to the console in a format parsable by TeamCity.
MIT License
4 stars 3 forks source link

TeamCity treats retried tests that fail and then pass as Failed #60

Closed jonbeller closed 4 years ago

jonbeller commented 4 years ago

As of cucumber-js v6, you can retry failing tests. I'd like to use this feature when running my tests on TeamCity (TC). If a test fails, then is retried, and then passes, I want TC to mark that test as Passed.

Right now, TC will group up runs of the same test, but if there are any failures, it will mark the test as Failed.

Screen Shot 2020-01-10 at 11 12 33 AM

Does anyone have advice for how I could get TC to treat such a test as passing? TeamCity intentionally does not support this behavior, but to me, without this, what's the point of retrying tests?

I imagine if the formatter only flushed the results of a passing or final attempt of each test, that could work.

bingnz commented 4 years ago

Hi @jonbeller. This is probably one of those areas where it's a matter of personal opinion (and it seems the opinion of JetBrains is that one failure counts as an overall failure). As this reporter is just a reporter, its job is to report test results; and TeamCity interprets the reporting of a test failure as it chooses. Also this plugin simply streams the events through; in order for the reporter to reinterpret the results it would have to buffer in some way and encapsulate its own logic about what a single failure means - probably in a configurable way - and that would make it a lot more complex. I'm not sure how you would implement this, because if you don't report the failure you wouldn't necessarily know that there was a retry at all, and that's useful information.

My personal opinion is that I see test retries as being potentially useful in highlighting to developers that a test is flaky (i.e., the test is most likely to be failing due to a timing issue or similar), but that a failure is a failure. If one of my projects had tests that were failing intermittently, I would want to know about those tests and fix them so that they passed on the first try - if nothing else, retries take longer - so the value is that I can then group failing tests into flaky ones that I must improve, and ones that are completely broken that are a higher priority.

The link you posted suggested muting these flaky tests, which is not quite what you want because that won't alert you if all of the tests fail. One other option might be to turn off the TeamCity build failure condition of "at least one test failed" and add another condition that catches a what you consider to be a "real" test failure (specific text, non-zero exit code from cucumber, etc). Would that be possible?

jonbeller commented 4 years ago

Hi @bingnz, thank you for the thoughtful reply. I agree now that you say it that it shouldn't be a formatter's job to withhold output. I've been dealing with flakiness that I haven't been able to totally fix, so I was looking for ways to live with that in case that's what it comes to. But not being aware of the flakiness is probably worse. You make a good point that retries can still help me differentiate flakes from real failures, which is valuable to me.