search

MobileNativeFoundation / bluepill

Bluepill is a reliable iOS testing tool that runs UI tests using multiple simulators on a single machine
BSD 2-Clause "Simplified" License
3.18k stars 231 forks source link

Distinguishing flaky and failed test results are difficult #466

Closed saim80 closed 3 years ago

saim80 commented 3 years ago

First of all, thanks for this awesome software. This relieved a lot of pain points we had.

This is a feature request.

Currently, we set up to retry 2 additional times to mitigate UI Test flakiness. When the test report is presented via JUnit XML file at our Jenkins TestResult page. We see all failed tests are listed upfront. We cannot distinguish which are the flaky tests, which eventually recovered by retry, and which are the failed tests where all 3 attempts are failed. Currently, we have to manually count the individual test failures to determine this. However, as our test base grows, this is becoming the major pain point for us.

Can Bluepill generate a better JUNIT XML file in this aspect?

saim80 commented 3 years ago

I found that Bluepill is already outputing such report via .js and .html files. And, it seems Jenkins JUnit report cannot show flakiness.