Our app have some flaky specs and we are retrying using --only-failures to retry just the flaky specs, which in general just works.
We are generating the junit.xml in the first try and then if a spec fails, we run the failed specs again a second time without it but this creates an issue where the CI interfaces says that the build didn't pass (because there are failed specs in the XML) but actually it did pass.
When that happens, we need to look at the build exit status to confirm if it failed or not.
What you think about updating/integrating with the --only-failures to update the previous XML somehow to make the interface more reliable?
PS: We can't use rspec-retry because we to do some cleanup between the retries and also looks more reliable to use the native --only-failures than a 3rd gem.
Hi there!
Our app have some flaky specs and we are retrying using
--only-failures
to retry just the flaky specs, which in general just works.We are generating the
junit.xml
in the first try and then if a spec fails, we run the failed specs again a second time without it but this creates an issue where the CI interfaces says that the build didn't pass (because there are failed specs in the XML) but actually it did pass.When that happens, we need to look at the build exit status to confirm if it failed or not.
What you think about updating/integrating with the
--only-failures
to update the previous XML somehow to make the interface more reliable?PS: We can't use
rspec-retry
because we to do some cleanup between the retries and also looks more reliable to use the native--only-failures
than a 3rd gem.