search

MobileNativeFoundation / bluepill

Bluepill is a reliable iOS testing tool that runs UI tests using multiple simulators on a single machine
BSD 2-Clause "Simplified" License
3.19k stars 232 forks source link

Consider app crash tests as non-fatal if they pass on retry #456

Closed ravimandala closed 3 years ago

ravimandala commented 3 years ago

Introducing a flag retry-app-crash-tests to retry tests that. might have caused app crashes. When this flag is turned on, the execution will not fail if all app crash tests pass on retry.

The retries will honor the error-retries count, restarts the sim before retrying the crashed test and makes the crash logs available, like before.

Ref: https://github.com/linkedin/bluepill/issues/455

\cc @ob @chenxiao0228

ravimandala commented 3 years ago

Code looks good but I wonder if we really need another flag. Our flag set is already too complicated so if we can find a way to roll this under an existing flag (or simplify the flag set somehow) that'd be much better.

Oscar, I agree a few flags might have gotten complicated over time and may even have deviated from the original purpose. So, I am going to take the latter route to simplify the flag set and deprecate some of them. Created issue https://github.com/linkedin/bluepill/issues/457 for further discussion and follow-up on this.

ravimandala commented 3 years ago

let's clean up the flags after this...

Sure. Merging this now.