search

sj26 / rspec_junit_formatter

RSpec results that your CI can read
http://rubygems.org/gems/rspec_junit_formatter
MIT License
302 stars 122 forks source link

How to handle retries? #98

Closed sobrinho closed 2 years ago

sobrinho commented 2 years ago

Hi there!

Our app have some flaky specs and we are retrying using --only-failures to retry just the flaky specs, which in general just works.

We are generating the junit.xml in the first try and then if a spec fails, we run the failed specs again a second time without it but this creates an issue where the CI interfaces says that the build didn't pass (because there are failed specs in the XML) but actually it did pass.

When that happens, we need to look at the build exit status to confirm if it failed or not.

What you think about updating/integrating with the --only-failures to update the previous XML somehow to make the interface more reliable?

PS: We can't use rspec-retry because we to do some cleanup between the retries and also looks more reliable to use the native --only-failures than a 3rd gem.

sj26 commented 2 years ago

This is intentionally a pretty simple formatter and doesn't attempt to read or compose with existing results.

I'd suggest creating separate output files and merging them together afterwards.