search

MobileNativeFoundation / bluepill

Bluepill is a reliable iOS testing tool that runs UI tests using multiple simulators on a single machine
BSD 2-Clause "Simplified" License
3.19k stars 232 forks source link

Consolidate and simplify Bluepill config flags #457

Open ravimandala opened 3 years ago

ravimandala commented 3 years ago

Bluepill flags have grown in number and complexity over time. It is time to simplify the logic, deprecate some, merge behaviors. Here are a few proposals.

Some proposals:

  1. Flags that have grown complicated over time especially those that got deviated from its original meaning (Eg. only-retry-failed used for other purposes too)
  2. Flags whose defaults are good enough and are rarely overridden.
  3. Merge flags with similar or confusing purposes. (Eg. error-retries and failure-tolerance can be merged.)
  4. Rarely used flags, if any. (printf-config, test-bundle-path, additional-unit-xctests, additional-ui-xctests)
  5. Non-functional flags, if any.
  6. Optionally consolidate internal-only flags.

Comments and suggestions are welcome.

\cc @ob @RainNapper @jerrymarino @chenxiao0228 @oliverhu

ob commented 3 years ago

I think what makes it hard to reason about the flags is that they are non-orthogonal (that is you can't reason about them independently of the other flags). How about focusing on the tests themselves and the possible results?

We have individual tests that might end up in one of four states: success, failure, crash, and timeout. Are there others?

And on each run, we want to be able to specify how much tolerance we have for each of those states right? We could do something like:

--timeout-tolerance=4 --failure-tolerance=2 --crash-tolerance=0 and mean allow 4 timeouts 2 failures and 0 app crashes. Would that make it simpler?