Unable to have successful build running unit test using Xcode 15

lmedranoz commented 1 year ago

Description

Hello,

We had been running the pipeline using Xcode 14.3.1 but now we are trying to update the pipeline to compile with Xcode 15.0 but since them pipeline takes forever to run the unit test.

The error below is what it shows in the console of the pipeline:

Executed 285 tests, with 4 failures (0 unexpected) in 2.154 (2.222) seconds
2023-11-02 05:21:59.207 xcodebuild[16838:97963]  IDETestOperationsObserverDebug: Failure collecting diagnostics from simulator: Timed out after 600.0 seconds while waiting for a response from the invoked process
2023-11-02 05:22:00.295 xcodebuild[16838:76000] [MT] IDETestOperationsObserverDebug: 1567.389 elapsed -- Testing started completed.
2023-11-02 05:22:00.296 xcodebuild[16838:76000] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
2023-11-02 05:22:00.296 xcodebuild[16838:76000] [MT] IDETestOperationsObserverDebug: 1567.390 sec, +1567.390 sec -- end
Failing tests:
  URLSession_ExtensionTest.testLoadFlightList()
  PaperworkUpdaterTests.test_ShouldFailToUpdatePaperworkAndIncrementAttempts_ForServiceError()
  PaperworkUpdaterTests.test_ShouldUpdatePaperwork()
  AppInfoSendFeedbackFormViewModelTests.testShouldCallApptentiveAndToggleShowConfirmation()

** TEST FAILED **

This is how we executing the unit test


- task: Xcode@5
  displayName: 'Xcode test'
  condition: ne(variables['TestConfiguration'], '')
  inputs:
    actions: 'test'
    configuration: 'Debug'
    sdk: '$(TestSDK)'
    xcWorkspacePath: 'MyApp.xcworkspace'
    scheme: '$(BuildScheme)'
    xcodeVersion: 'specifyPath'
    xcodeDeveloperDir: '$(Xcode_Version_Path)'
    packageApp: false
    signingOption: 'nosign'
    destinationPlatformOption: 'iOS'
    destinationSimulators: 'iPhone 14,OS=16.4'
    publishJUnitResults: true
    args: '-verbose'

Please let us know if you need any more information

Platforms affected

[X] Azure DevOps
[ ] GitHub Actions - Standard Runners
[ ] GitHub Actions - Larger Runners

Runner images affected

[ ] Ubuntu 20.04
[ ] Ubuntu 22.04
[ ] macOS 11
[ ] macOS 12
[X] macOS 13
[ ] Windows Server 2019
[ ] Windows Server 2022

Image version and build link

variables:

name: Xcode_Version_Path value: /Applications/Xcode_15.0.app
name: ProjectPathForExecution value: $(Build.SourcesDirectory)/Block2Block/
name: BuildName value: MyApp.ipa

Is it regression?

No, production build

Expected behavior

Get a successful build

Actual behavior

The build is failing running unit test

Repro steps

The build is failing using Azure develops pipeline

erik-bershel commented 1 year ago

Hey @lmedranoz! Sad to say it, but it is a infamous bug in XCode. 😞 Same here: https://github.com/actions/runner-images/issues/8693 But you may find a workaround here: https://github.com/actions/runner-images/discussions/8651 I'm going to close this item as a duplicate. But feel free to ask issue related questions or open new issues in case of other concerns.

lmedranoz commented 1 year ago

@erik-bershel Question,

In the link you mention above mentions this:

- name: Install yeetd
   run: |
       wget https://github.com/biscuitehh/yeetd/releases/download/1.0/yeetd-normal.pkg
       sudo installer -pkg yeetd-normal.pkg -target /
       yeetd &

Do you know where in the YAML file should add this lines?

erik-bershel commented 1 year ago

@lmedranoz should be placed just before the task for XCode. This is a daemon that monitors certain processes and partially resolves the problem.

lmedranoz commented 1 year ago

@erik-bershel I add it the code above:

But I'm getting the following error:

Do you know how can fix this error?

erik-bershel commented 1 year ago

@lmedranoz How about to test something like that:

steps:
- script: |
    wget https://github.com/biscuitehh/yeetd/releases/download/1.0/yeetd-normal.pkg
    sudo installer -pkg yeetd-normal.pkg -target /
    yeetd &
  displayName: 'Download and Install yeetd'
  continueOnError: true

- task: Xcode@5
  displayName: 'Xcode test'
  condition: ne(variables['TestConfiguration'], '')
  inputs:
    actions: 'test'
    configuration: 'Debug'
    sdk: '$(TestSDK)'
    xcWorkspacePath: 'MyApp.xcworkspace'
    scheme: '$(BuildScheme)'
    xcodeVersion: 'specifyPath'
    xcodeDeveloperDir: '$(Xcode_Version_Path)'
    packageApp: false
    signingOption: 'nosign'
    destinationPlatformOption: 'iOS'
    destinationSimulators: 'iPhone 14,OS=16.4'
    publishJUnitResults: true
    args: '-verbose'

lmedranoz commented 1 year ago

@erik-bershel I add the changes above but now I'm getting this error:

/usr/bin/xcodebuild -version
Xcode 15.0.1
Build version 15A507
/usr/bin/xcodebuild -sdk iphonesimulator -configuration Debug -workspace /Users/runner/work/1/s/Block2Block.xcworkspace -scheme Block2Block -destination platform=iOS Simulator,name=iPhone 14,OS=16.4 test -verbose CODE_SIGNING_ALLOWED=NO | /usr/local/lib/ruby/gems/3.0.0/bin/xcpretty -r junit --no-color
--- xcodebuild: WARNING: Using the first of multiple matching destinations:
{ platform:iOS, id:dvtdevice-DVTiPhonePlaceholder-iphoneos:placeholder, name:Any iOS Device }
{ platform:iOS Simulator, id:dvtdevice-DVTiOSDeviceSimulatorPlaceholder-iphonesimulator:placeholder, name:Any iOS Simulator Device }
{ platform:iOS Simulator, id:C9DCBA09-A1F2-4025-9AB1-786CC66DA50E, OS:17.0.1, name:iPad (10th generation) }
{ platform:iOS Simulator, id:739533D9-600B-4B8C-B3D9-FDB0D3960C94, OS:17.0.1, name:iPad Air (5th generation) }
{ platform:iOS Simulator, id:1B2C3FF8-EAB0-4B25-B692-53B9318F494E, OS:17.0.1, name:iPad Pro (11-inch) (4th generation) }
{ platform:iOS Simulator, id:B0DB33E3-3158-4FC4-AF89-65B035539035, OS:17.0.1, name:iPad Pro (12.9-inch) (6th generation) }
{ platform:iOS Simulator, id:C5F73642-F279-414B-AA07-30ED12A39E7B, OS:17.0.1, name:iPad mini (6th generation) }
{ platform:iOS Simulator, id:25300F54-86DE-4AC7-B194-A9DD4CBF3B76, OS:17.0.1, name:iPhone 14 }
{ platform:iOS Simulator, id:7A31EA29-B544-4AB3-A859-E3349605CE46, OS:17.0.1, name:iPhone 14 Plus }
{ platform:iOS Simulator, id:28E70F93-B182-4A2E-9B9B-D44C0AC4CB48, OS:17.0.1, name:iPhone 14 Pro }
{ platform:iOS Simulator, id:3830B3FC-110F-454A-931D-3BE468269719, OS:17.0.1, name:iPhone 14 Pro Max }
{ platform:iOS Simulator, id:6C6B0B47-223C-465E-B449-54A161ED15CC, OS:17.0.1, name:iPhone 15 }
{ platform:iOS Simulator, id:C376764B-8D43-4014-978F-A096C1A2BD68, OS:17.0.1, name:iPhone 15 Plus }
{ platform:iOS Simulator, id:6DFAD023-11E3-46B2-8E0F-979E6D8C5D42, OS:17.0.1, name:iPhone 15 Pro }
{ platform:iOS Simulator, id:6CAD835A-F451-458B-B8AC-87C958863F85, OS:17.0.1, name:iPhone 15 Pro Max }
{ platform:iOS Simulator, id:1D27CE51-1EA1-40DC-85A2-4FB76F0A57A0, OS:17.0.1, name:iPhone SE (3rd generation) }
2023-11-02 20:48:27.860 xcodebuild[12650:55327] Writing error result bundle to /var/folders/q4/3s60g5510ydbd3qc3p4xflm80000gn/T/ResultBundle_2023-02-11_20-48-0027.xcresult
xcodebuild: error: Failed to build workspace Block2Block with scheme Block2Block.: Cannot test target Block2BlockTests on Any iOS Device: Tests must be run on a concrete device
##[error]Error: /usr/bin/xcodebuild failed with return code: 70
Finishing: Xcode test

from the YAML file:

Do you know why I may be getting this error?

erik-bershel commented 1 year ago

@lmedranoz take a look here: https://github.com/actions/runner-images/issues/8621 The end of discussion looks related to your current issue.

actions / runner-images