search

MobileNativeFoundation / bluepill

Bluepill is a reliable iOS testing tool that runs UI tests using multiple simulators on a single machine
BSD 2-Clause "Simplified" License
3.19k stars 232 forks source link

Proposal: Do not retry the test on app crash #423

Closed ravimandala closed 4 years ago

ravimandala commented 4 years ago

When the app crashes during a test, Bluepill fails the execution by returning a failure exit code BPExitStatusAppCrashed = 7, even if the test passes on retry. But if a test times out and passes on retry (and no other app crash related failures), Bluepill deems it to be okay by returning an exit status of BPExitStatusTestsAllPassed = 0.

In summary, App times out and passes on retry ==> Bluepill passes the execution. App crashes and passes on retry ==> Bluepill fails the execution. This is confusing.

Proposal: How about we keep the end result as-is but not retry test that crashed the app? \cc @ob

valeraz commented 4 years ago

This would work well for us 👍

ob commented 4 years ago

@ravimandala this is already fixed in master right? Would it be worth adding something in the docs that mention that not retrying on app crashes is by design and not an omission?

ravimandala commented 4 years ago

@ob This is not fixed in master yet. I will implement it and update the documentation.

ravimandala commented 4 years ago

Implemented and included in v5.4.1 (and above).