Closed imcdo closed 2 years ago
Just a thought, if the error summary is same for all the runs, it would be good if we can just display that 1 error and number of times it failed with the same reason. If a test failed all 5 times with same reason, displaying the same error summary 5 times may be cumbersome to read, especially if we are running a bunch of tests. Otherwise looks good
This is a good idea but Im not sure a good way to display this, would something like this work:
summary:
run 1, 3, 5 summary:
....
run 2, 4 summary:
...
run 6: PASSED
? Thoughts @nagpavan-chilakam @stan-confluent
Looks good overall, thanks Ian!
example fail case:
example flaky case:
Motivation: when getting a flaky test, the summaries are clear due to the last run passing, loosing all data of failed runs. This PR intends to remedy this by changing the summary behavior of deflake runs. In a deflake run, all summaries are included, and demarked by the run number and a
~:Testing: ran unit tests and ran all system tests