Even with boot-simulators, devices show UI

kdubb commented 7 years ago

Been trying this out with Xcode 8.2. Using boot-simulators seems to correctly put the simulators into the booted state (verified via xcrun simctl --path <simset> list). From that point, using run-tests causing the previously "headless" simulators to suddenly grow a head. I've checked & double checked the options I'm passing (even going as far verify them by debugging pxctest).

What else could this issue be?

FYI, we've fully automated our iOS builds with CI & CD using build-for-testing and 'test-without-building`. Would love to help out with this project to, hopefully drastically, reduce our integration times.

plu commented 7 years ago

I need some more information to understand what might be causing this:

output of pxctest version
what options are you passing to boot-simulators?
what options are you passing to run-tests?
if possible, content of the xctestrun file, alternatively how many test targets it contains and what type of targets they are

kdubb commented 7 years ago

While validating my results I inadvertently discovered the issue. Our xctestrun file has two testable bundles, Unit & UI. It seems pxctest attempts to handle this by running them in parallel as well.

Unfortunately there doesn't seem to be a way to boot multiple simulators of the same device type and OS version to handle this case. If it needs two sims of the same type I cannot boot them using boot-simulators. I can boot multiple identical simulators using xcrun simctl and it operates in the background as expected running the test bundles in parallel.

Maybe a better method would be for pxctest to determine the required simulators and boot them, run the tests on them, shut them all down? A configuration for max parallel simulators would allow it to cycle through the required sims without running out of resources. Smart sorting of the test bundles + sim required to run it would ensure the fastest execution of all bundles on all destinations.

plu commented 7 years ago

~$ mkdir /tmp/sims
~$ pxctest boot-simulators --deviceset /tmp/sims --destination 'name=iPhone 5' --destination 'name=iPhone 5'
Jan  7 06:09:33  FBControlCore[69153] <Info>: Erasing 404C8C9A-8B24-48C3-AEEC-66DCED7F7A2E | iPhone 5 | Shutdown | iPhone 5 | iOS 10.2 | 0 | 0
Jan  7 06:09:33  FBControlCore[69153] <Info>: Erased 404C8C9A-8B24-48C3-AEEC-66DCED7F7A2E | iPhone 5 | Shutdown | iPhone 5 | iOS 10.2 | 0 | 0
Jan  7 06:09:33  FBControlCore[69153] <Info>: Erasing 8BA12FFD-1D6E-498C-9AE0-312B0B71F4D2 | iPhone 5 | Shutdown | iPhone 5 | iOS 10.2 | 0 | 0
Jan  7 06:09:33  FBControlCore[69153] <Info>: Erased 8BA12FFD-1D6E-498C-9AE0-312B0B71F4D2 | iPhone 5 | Shutdown | iPhone 5 | iOS 10.2 | 0 | 0
Jan  7 06:09:33  FBControlCore[69153] <Info>: 404C8C9A-8B24-48C3-AEEC-66DCED7F7A2E | iPhone 5 | Shutdown | iPhone 5 | iOS 10.2 | 0 | 0
Jan  7 06:09:34  FBControlCore[69153] <Info>: 404C8C9A-8B24-48C3-AEEC-66DCED7F7A2E: Simulator Did launch => Process launchd_sim | PID 69155
Jan  7 06:09:34  FBControlCore[69153] <Info>: 404C8C9A-8B24-48C3-AEEC-66DCED7F7A2E: Did Change State => Creating
Jan  7 06:09:34  FBControlCore[69153] <Info>: 404C8C9A-8B24-48C3-AEEC-66DCED7F7A2E: Simulator Did Terminate => Process launchd_sim | PID 69155 Expected 0
Jan  7 06:09:34  FBControlCore[69153] <Info>: 404C8C9A-8B24-48C3-AEEC-66DCED7F7A2E: Did Change State => Shutdown
Jan  7 06:09:34  FBControlCore[69153] <Info>: 8BA12FFD-1D6E-498C-9AE0-312B0B71F4D2: Did Change State => Creating
Jan  7 06:09:34  FBControlCore[69153] <Info>: 8BA12FFD-1D6E-498C-9AE0-312B0B71F4D2: Did Change State => Shutdown
Jan  7 06:09:34  FBControlCore[69153] <Info>: 404C8C9A-8B24-48C3-AEEC-66DCED7F7A2E: Did Change State => Booting
Jan  7 06:09:34  FBControlCore[69153] <Info>: 404C8C9A-8B24-48C3-AEEC-66DCED7F7A2E: Simulator Did launch => Process launchd_sim | PID 69155
Jan  7 06:09:34  FBControlCore[69153] <Info>: 404C8C9A-8B24-48C3-AEEC-66DCED7F7A2E: Did Change State => Booted
Jan  7 06:10:09  FBControlCore[69153] <Info>: 404C8C9A-8B24-48C3-AEEC-66DCED7F7A2E: Connection Did Connect => Bridge: Framebuffer ((null)) | HID Indigo HID Port: 17455 | Simulator Bridge: Connected
Jan  7 06:10:09  FBControlCore[69153] <Info>: 8BA12FFD-1D6E-498C-9AE0-312B0B71F4D2 | iPhone 5 | Shutdown | iPhone 5 | iOS 10.2 | 0 | 0
Jan  7 06:10:10  FBControlCore[69153] <Info>: 8BA12FFD-1D6E-498C-9AE0-312B0B71F4D2: Simulator Did launch => Process launchd_sim | PID 69407
Jan  7 06:10:10  FBControlCore[69153] <Info>: 8BA12FFD-1D6E-498C-9AE0-312B0B71F4D2: Did Change State => Booting
Jan  7 06:10:10  FBControlCore[69153] <Info>: 8BA12FFD-1D6E-498C-9AE0-312B0B71F4D2: Did Change State => Booted
Jan  7 06:10:27  FBControlCore[69153] <Info>: 8BA12FFD-1D6E-498C-9AE0-312B0B71F4D2: Connection Did Connect => Bridge: Framebuffer ((null)) | HID Indigo HID Port: 6099 | Simulator Bridge: Connected
~$ xcrun simctl --set /tmp/sims list devices
== Devices ==
-- iOS 8.1 --
-- iOS 8.2 --
-- iOS 8.3 --
-- iOS 8.4 --
-- iOS 9.0 --
-- iOS 9.1 --
-- iOS 9.2 --
-- iOS 9.3 --
-- iOS 10.0 --
-- iOS 10.1 --
-- iOS 10.2 --
    iPhone 5 (404C8C9A-8B24-48C3-AEEC-66DCED7F7A2E) (Booted)
    iPhone 5 (8BA12FFD-1D6E-498C-9AE0-312B0B71F4D2) (Booted)
-- tvOS 9.0 --
-- tvOS 9.1 --
-- tvOS 10.1 --
-- watchOS 2.0 --
-- watchOS 2.1 --
-- watchOS 3.1 --
~$ pxctest version
pxctest 0.3.0 (cd208cd)

Another option you have is passing --only UnitTestTargetName and --only UITestTargetName to the run-tests command, to only execute one target at a time - probably even split your CI job into two.

apkatsikas commented 7 years ago

hi @kdubb - if you are looking to run the same simulator type in parallel, and divvy up your tests to each simulator, check out my python script where i accomplished this - https://github.com/apkatsikas/xctest_pxctest_distributed

plu / pxctest

Even with boot-simulators, devices show UI #12