search

MobileNativeFoundation / bluepill

Bluepill is a reliable iOS testing tool that runs UI tests using multiple simulators on a single machine
BSD 2-Clause "Simplified" License
3.18k stars 231 forks source link

Xcode 13.2.1 CRASH with "App crashed before tests started" #533

Closed renatoaugustofranca closed 1 year ago

renatoaugustofranca commented 2 years ago

We are updating our project to use the Xcode 13.2.1 and we are facing an issue when running the tests using LATEST bluebill as follows:

{29160} 20220419.201324 [ CRASH ] (BP-1) App crashed before tests started.

For more context the complete log:

Debug Enabled == 0 {29160} 20220419.200923 [ INFO ] (BP-1) Running Tests. Attempt Number 1. {29160} 20220419.200923 [ INFO ] (BP-1) [Attempt 1] Create Simulator {29160} 20220419.200923 [ INFO ] (BP-1) Booting a simulator without launching Simulator app {28868} 20220419.200923 [ INFO ] (BLUEPILL) 1 BP(s) still running. [1] {28868} 20220419.200923 [ INFO ] (BLUEPILL) Using 360 of 2048 processes. {29160} 20220419.200923 [ INFO ] (BP-1) Simulator A4C1859B-7484-4E02-8F42-D1F2B0C07DBD achieved the BOOTED state Booted {29160} 20220419.200923 [ INFO ] (BP-1) Completed: [Attempt 1] Create Simulator A4C1859B-7484-4E02-8F42-D1F2B0C07DBD {29160} 20220419.200923 [ INFO ] (BP-1) [Attempt 1] Install Application {28868} 20220419.200953 [ INFO ] (BLUEPILL) 1 BP(s) still running. [1] {28868} 20220419.200953 [ INFO ] (BLUEPILL) Using 462 of 2048 processes. {29160} 20220419.201009 [ INFO ] (BP-1) Completed: [Attempt 1] Install Application {29160} 20220419.201009 [ INFO ] (BP-1) [Attempt 1] Launch Application {29160} 20220419.201021 [ INFO ] (BP-1) Launch succeeded {29160} 20220419.201021 [ INFO ] (BP-1) No error {29160} 20220419.201021 [ INFO ] (BP-1) Completion block for launch {29160} 20220419.201021 [ INFO ] (BP-1) Calling completion block with: (null) - 29354 {29160} 20220419.201021 [ INFO ] (BP-1) Completed: [Attempt 1] Launch Application {29160} 20220419.201023 [ INFO ] (BP-1) Test manager daemon control session started (0) {28868} 20220419.201023 [ INFO ] (BLUEPILL) 1 BP(s) still running. [1] {28868} 20220419.201023 [ INFO ] (BLUEPILL) Using 510 of 2048 processes. {29160} 20220419.201023 [ INFO ] (BP-1) Test plan started! {29160} 20220419.201024 [ INFO ] (BP-1) Got proxy channel request from test bundle {29160} 20220419.201024 [ INFO ] (BP-1) Test bundle is connected. Capabilities = <XCTCapabilities: 0x7f991ac15a40>: { "ubiquitous test identifiers" = 1; } {28868} 20220419.201053 [ INFO ] (BLUEPILL) 1 BP(s) still running. [1] {28868} 20220419.201053 [ INFO ] (BLUEPILL) Using 519 of 2048 processes. {28868} 20220419.201123 [ INFO ] (BLUEPILL) 1 BP(s) still running. [1] {28868} 20220419.201123 [ INFO ] (BLUEPILL) Using 520 of 2048 processes. {28868} 20220419.201153 [ INFO ] (BLUEPILL) 1 BP(s) still running. [1] {28868} 20220419.201153 [ INFO ] (BLUEPILL) Using 519 of 2048 processes. {28868} 20220419.201223 [ INFO ] (BLUEPILL) 1 BP(s) still running. [1] {28868} 20220419.201223 [ INFO ] (BLUEPILL) Using 519 of 2048 processes. {28868} 20220419.201254 [ INFO ] (BLUEPILL) 1 BP(s) still running. [1] {28868} 20220419.201254 [ INFO ] (BLUEPILL) Using 519 of 2048 processes. {29160} 20220419.201324 [ INFO ] (BP-1) DTXSocketTransport disconnected {29160} 20220419.201324 [ INFO ] (BP-1) DTXConnection disconnected. {28868} 20220419.201324 [ INFO ] (BLUEPILL) 1 BP(s) still running. [1] {29160} 20220419.201324 [ CRASH ] (BP-1) App crashed before tests started. {28868} 20220419.201324 [ INFO ] (BLUEPILL) Using 517 of 2048 processes. {29160} 20220419.201324 [ INFO ] (BP-1) Finished {29160} 20220419.201324 [ INFO ] (BP-1) Writing JUnit report to: /Users/nubank/jenkins_home/workspace/nu-mobile-ios-validate/native/ios/output/BP-1/TEST-CoreTestsSwift3-1-results.xml {29160} 20220419.201324 [ INFO ] (BP-1) [Attempt 1] Delete Simulator {29160} 20220419.201324 [ INFO ] (BP-1) Shutting down Simulator {29160} 20220419.201329 [ INFO ] (BP-1) DTXSocketTransport disconnected {29160} 20220419.201329 [ INFO ] (BP-1) Daemon connection Disconnected. {29160} 20220419.201330 [ INFO ] (BP-1) Completed: [Attempt 1] Delete Simulator A4C1859B-7484-4E02-8F42-D1F2B0C07DBD {29160} 20220419.201330 [ INFO ] (BP-1) Attempt's Exit Status: BPExitStatusAppCrashed, Bundle exit status: BPExitStatusAppCrashed

Running the tests using the command line or running directly from XCode IDE tests run correctly.

Eg.

xcodebuild test-without-building -xctestrun ./MyTarget-Debug_iphonesimulator14.4-x86_64.xctestrun -destination 'platform=iOS Simulator,name=iPhone 12 Pro Max'

Also, we have searched through the logs (with diagnostics option enabled) and we found this:

[97318:3182032] Error while preparing for testing: Error Domain=com.apple.dt.XCTest.XCTFuture Code=1000 "Timed out while requesting ready for testing." UserInfo={NSLocalizedDescription=Timed out while requesting ready for testing.}

Has anyone faced a similar issue and know a fix or a workaround for this?

nanwng commented 1 year ago

We encountered the same issue. I think the issue is that we started the test plan too early that the test bundle has not been connected yet, so the XCTFuture object kept waiting till timeout.

I have a fix on this branch https://github.com/MobileNativeFoundation/bluepill/tree/fix-test-trigger will test it internally for a few days to confirm.