search

MobileNativeFoundation / bluepill

Bluepill is a reliable iOS testing tool that runs UI tests using multiple simulators on a single machine
BSD 2-Clause "Simplified" License
3.19k stars 232 forks source link

[Reporting bug] Tests that passed on retry are not being reported accurately #517

Open ravimandala opened 2 years ago

ravimandala commented 2 years ago

I noticed an issue in Bluepill version 5.10.0 that supports Xcode 13. In a successful execution, 11 tests are reported to have failed. But the execution was successful because those tests passed on retry. Usually when tests pass on retry it should be reported accordingly in 0-test-report.html, which doesn’t seem to be the case in this execution. This might be an edge case specific to the latest version. I will try to get more details or a test case to reproduce this.

jschear commented 2 years ago

I saw similar behavior, and the root cause was a small bug in the collation of test results: https://github.com/MobileNativeFoundation/bluepill/pull/515

Are you seeing invalid XML files created in any of your executions? If an execution produces a Junit XML file that isn't a valid XML document (which we were seeing when a simulator failed to boot; the file contained just the string "NO LOG: JUnitReporter"), other (perfectly valid) test reports would be treated as invalid, and not merged into the final combined report.

ravimandala commented 2 years ago

True. I see the executions where this did happen and I see how it could skip the subsequent reports that might be perfectly valid. Thanks for the fix.

ravimandala commented 2 years ago

@jschear Your PR(https://github.com/MobileNativeFoundation/bluepill/pull/515) might fix this issue. I will resolve this issue after verifying.

cccCody commented 2 years ago

This doesn't seem to be fixed. I'm using 5.11, and I see results for each individual attempt in the xml, so if a test fails, retries, and then passes, I still have a failure in the output. There's an old open PR to add a flag for this here: https://github.com/MobileNativeFoundation/bluepill/pull/467

...and one comment even mentions making it the default behavior, but that doesn't seem to have happened.