search

MobileNativeFoundation / bluepill

Bluepill is a reliable iOS testing tool that runs UI tests using multiple simulators on a single machine
BSD 2-Clause "Simplified" License
3.19k stars 232 forks source link

Ability to run same test multiple times #424

Open ravimandala opened 4 years ago

ravimandala commented 4 years ago

Bluepill should support a use case to run a particular test like 100 times. Even better, if Bluepill can support running a given set of tests any given number of times, then it would help in debugging flaky tests.

Can this be done by using include and repeat-count parameters in conjunction? 🤔

ravimandala commented 4 years ago

It's worth testing if that flow still works. Wondering if anyone of the Bluepill users is using this regularly!!!???

RainNapper commented 4 years ago

I'm currently working on parallel retries to deflake tests.

The plan is:

  1. Run all tests once, with no retries
  2. Run all failures 2x Flaky tests are all tests with at least 1 pass.

I originally had hoped there was some minimal configuration on a single run like:

bp --failure-tolerance=2 --run-retries-in-parallel

I did not see an existing flag that looked like --run-retries-in-parallel. So instead, I worked on having multiple runs:

bp --failure-tolerance=0
[parse failures and rerun]
bp --repeat-count=2 --include failedTest1 --include failedTest2

Based on the output below, it looks like the repeats are run in series. As a result, this doesn't work for us either.

{73078} 20200324.144027 [  INFO  ] (BLUEPILL) BP-1 (PID 73098) has finished with exit code 1.
{73078} 20200324.144027 [  INFO  ] (BLUEPILL) PID 73098 exited 1.
{73078} 20200324.144027 [  INFO  ] (BLUEPILL) Started BP-2 (PID 73279).

In your case of running a particular case 100 times, it would be prohibitively slow if these repeats were NOT run in parallel.

Based on above, I think the ability to either run repeats or retries in parallel would be a very useful feature to add. I'd also imagine this is not too difficult considering the parallelism is already there for running multiple different tests.

ravimandala commented 4 years ago

The parallel execution of repeats is certainly possible, just that it might not be the way it is done now. Especially if a test or a set of tests has to be run n number of times, then running in series is not a good option. Thanks for pointing that out.

RainNapper commented 4 years ago

I was able to add parallel execution of repeats with a fairly small change. I will make a PR.

That said, parallel retries seems a bit more complicated. A task is not retried by adding a new bundle to the queue, but rather a task restarting itself. This means that it will be more complicated to have a task restart itself multiple times. To support this, I think it would be a good idea to first have tasks add themselves back into the queue and let the scheduler handle the rest.

ravimandala commented 3 years ago

@RainNapper It would be great to have this implemented in some form, whether it is integrating https://github.com/RainNapper/bluepill/tree/mmark/pinpill/pinpill/pinpill in some capacity or in full capacity into the Bluepill. Do you have some time to revisit this?