search

MobileNativeFoundation / bluepill

Bluepill is a reliable iOS testing tool that runs UI tests using multiple simulators on a single machine
BSD 2-Clause "Simplified" License
3.18k stars 231 forks source link

Limit the number of failures before a simulator is restarted for a retry #474

Closed ravimandala closed 3 years ago

ravimandala commented 3 years ago

Often due to flakiness in tests there might be a bunch of failures during a simulator run. If the tests are configured to retry, then they may all pass upon restarting the simulator and retrying them. However, since there is no limit on how many tests a simulator can let fail before going for a retry it could waste a lot of time failing on one test after another until it goes through all remaining tests in the bundle. LinkedIn's flagship app is facing this issue most likely due to iOS 14 simulator flakiness.

The proposal is to put a limit on number of test failures each simulator can take before restarting, in terms of a Bluepill configuration. \cc @ob @chenxiao0228 @RainNapper

chenxiao0228 commented 3 years ago

@ravimandala Can you start a conversation internally first?