Is there a way to re-use simulators between invocations of bluepill? And would that be faster than creating them?

[samskiter]

I noticed Bluepill's reuse-simulators setting, the simulators are still deleted after bluepill is done. Is possible to leave Bluepill simulators around between invocations?

Rather than deleting them, could you just reset them? (reset content and settings)

Would this be any faster than deleting and creating them anew as bluepill currently does?

[ob]

There is an option --reuse-simulator that will keep the simulators between runs (assuming that nothing wrong happened, like a simulator locking up). Back when we implemented it (@njuliyu wrote it), we timed it and it wasn't a big speed gain. Looking back on it I think the problem was that we were installing/uninstalling the app between runs and that was where most of the time was being spent.

There is an idea floating around about having a master simulator with the app already installed and then having each parallel run just clone that simulator. However, we haven't started working on that. We're currently focusing on getting the reports and statistics to be accurate in the face of failures, errors, timeouts, and retries.

I would encourage you to play with the --reuse-simulator option and report back how it works and how we could make it better.

Thanks!

[samskiter]

Interesting, my reading suggested otherwise:

When all test bundles have been finished, and the device id pool to reuse is not empty, the bluepill driver will launch new bp instance with “delete-simulator:[device id string]” for each device to shutdown and delete all the running simulators. But it is fire & forget so the driver won’t wait for the bp to return.

We've started to compare XCode 9's headless mode against bluepill's, and early results seem to show that invoking xcodebuild directly was a lot faster (I've not seen this directly myself but a colleague said we managed it in 2 mins spread over 3 devices rather than usually around 20mins with bluepill!). Does this sound likely/reasonable to you? (is there some legacy way that BP is achieving headless that doesn't apply now that XCode 9 can support it natively?)

[oliverhu]

It doesn't sound likely to us. We use Mac Mini 2014 Gen in our data centers and from our observations, it takes ~2 min to spread over 5 simulators.

20180118.145342 Bluepill runtime version and compile time version are matched: 9.1 (9B55)
... // bootstrap
20180118.145504 [  INFO  ] (BP-2) All Tests started.
20180118.145504 [ PASSED ] (BP-2)   0.095646s Tests/testA

[ob]

It depends on how many tests you have and how the .xctest bundles are configured. If you have mostly unit tests then it will probably be faster to just run them all in xcodebuild than parallelize them. The reason is that the overhead of creating a simulator, booting it, installing the app, etc. overtakes the overhead of actually running the tests.

Where bluepill becomes faster is when you have thousands of tests so that running them linearly would be slower than parallelizing. As I said, we're looking into minimizing some of the overhead, but splitting the tests might be slower if there are not too many tests and/or if they are fast.

[oliverhu]

Ugh, I thought @samskiter was complaining over the simulator initial booting time. It would be better if you could give more context on your testing environment (how many unit/UI tests) and how you configured to run Bluepill. Logs would be appreciated.

It doesn't sound right even if you only have unit tests, give xcodebuild takes 2 minutes, I estimate you have roughly ~1000 unit tests. With Bluepill, it should never take 20 minute..

[samskiter]

I guess I'm talking about 2 different things now. Boot times and xcodebuild vs bluepill for headless tests.

We're mainly concerned with UI tests times. but boot time do slow unit tests down too.

My colleague said he split our tests into different shemes, in each one selecting different tests. then fired off all the schemes with xcodebuild test-without-building. He's had up to 10 spun up on a MacBook Pro (pre-touchbar). Although that didn't involve resetting the simulator, running the tests themselves seemed a lot faster.

[samskiter]

does --reuse-simulator clean the sims at the start?

[oliverhu]

close old issues, feel free to reopen