Open HoussemNasri opened 2 weeks ago
chibenwa
commented
2 weeks ago It’s designed to identify and rerun flaky tests automatically, which could help in our situation.
How about fixing the flacky test?
When we encoutered an unrelated failing test we just open a ticket about it investigate it and fix it.
rerun flaky tests automatically
I'm against it. That would be hiding issues that would only grow worse. Let's rather fix them.
HoussemNasri
commented
2 weeks ago How about fixing the flacky test?
Question is: is it always possible to fix a flasky test.
When we encoutered an unrelated failing test we just open a ticket about it investigate it and fix it
+1
Could be useful:
I'm against it. That would be hiding issues that would only grow worse. Let's rather fix them.
The plugin would also categorize a test as flasky if it fails and succeeds on rerun so it can be used to detect flasky tests too.
chibenwa
commented
2 weeks ago Question is: is it always possible to fix a flasky test.
Yes
chibenwa
commented
2 weeks ago The plugin would also categorize a test as flasky if it fails and succeeds on rerun so it can be used to detect flasky tests too.
A broken build is IMO a better incetive to fix stuff
Description
Here's a typical scenario: I open a pull request, the build kicks off, and about an hour and a half later, the pipeline fails. When I check the failed tests, it’s something completely unrelated to my changes—a classic flaky test problem.
Here are some suggestions:
Run flaky tests in a separate pipeline If flaky tests have their own pipeline, we can simply rerun that part when it fails instead of rerunning everything.
Disable flaky tests (not ideal)
Try the Jenkins Flaky Test Handler plugin Here’s the plugin. It’s designed to identify and rerun flaky tests automatically, which could help in our situation.
Why This Matters: