search

MobileNativeFoundation / bluepill

Bluepill is a reliable iOS testing tool that runs UI tests using multiple simulators on a single machine
BSD 2-Clause "Simplified" License
3.19k stars 232 forks source link

Make app crash tests being fatal, as opt-out behind a flag #455

Closed ravimandala closed 3 years ago

ravimandala commented 3 years ago

Bluepill, as of today, treats app crash tests as fatal. So, when a test crashes the app then it is treated as fatal by returning a final non-zero exit code even if the test passes on retry.

Proposals:

  1. Make the non-zero exit code on app crashes as an opt-out feature behind a flag. (Eg. make-app-crashes-fatal defaults to YES)
  2. Put retry of an app crash test optional behind an opt-in flag. (Eg. retry-app-crash-tests defaults to NO)

LinkedIn's large regression test suite would benefit from these features. Please let us know if you would be interested to adopt the same. As always, comments and suggestions are welcome. \cc @chenxiao0228 @ob @oliverhu @RainNapper @jerrymarino

ravimandala commented 3 years ago

Alternatively, to avoid having too many flags, avoid having flags with different defaults (YES/NO) and keep things simple... we can use retry-app-crash-tests or retryAppCrashTests to consider app crash tests as non-fatal if they pass on retry.

AhmedEid commented 3 years ago

This would definitely help us. We have an issue where a test crashes, but we cannot re-run it and get a 🟢 build if it passes on a rerun.

ravimandala commented 3 years ago

Since Bluepill v5.4.2, the app crash tests are not retried by default and the execution fails if one or more app crash tests are present. But if retry-app-crash-tests flag is set to true, then the app crash tests are retried and the execution passes only if all those tests pass on retry. \cc @AhmedEid