search

actions / runner-images

GitHub Actions runner images
MIT License
10.2k stars 3.06k forks source link

MacOS 10.5 20201011.1 build resolution issue #1804

Closed crisvergara closed 4 years ago

crisvergara commented 4 years ago

Description
I have a React Native application in App Center. At some point yesterday, 20201011.1 was rolled out, and now all of my builds are failing. Specifically, there are errors resolving files within a Pod dependency (Braintree). There doesn't seem to be any release branch or documentation in this repository, so I'm not sure what exactly changed between 20201011.1 and the last working version 20201003.1.

Area for Triage:
Apple

Question, Bug, or Feature?:
Bug

Virtual environments affected

Expected behavior
In 20201003.1, the build log shows the following output for building the Braintree Pod:

CreateBuildDirectory /Users/runner/Library/Developer/Xcode/DerivedData/bookofthemonth-ajnelrlzioikrefiqgehrdieliqy/Build/Intermediates.noindex/ArchiveIntermediates/bookofthemonth/IntermediateBuildFilesPath (in target 'BraintreeDropIn-Braintree-UIKit-Localization' from project 'Pods')
CreateBuildDirectory /Users/runner/Library/Developer/Xcode/DerivedData/bookofthemonth-ajnelrlzioikrefiqgehrdieliqy/Build/Intermediates.noindex/ArchiveIntermediates/bookofthemonth/InstallationBuildProductsLocation (in target 'BraintreeDropIn-Braintree-UIKit-Localization' from project 'Pods')
CreateBuildDirectory /Users/runner/Library/Developer/Xcode/DerivedData/bookofthemonth-ajnelrlzioikrefiqgehrdieliqy/Build/Intermediates.noindex/ArchiveIntermediates/bookofthemonth/BuildProductsPath (in target 'BraintreeDropIn-Braintree-UIKit-Localization' from project 'Pods')
WriteAuxiliaryFile /Users/runner/Library/Developer/Xcode/DerivedData/bookofthemonth-ajnelrlzioikrefiqgehrdieliqy/Build/Intermediates.noindex/ArchiveIntermediates/bookofthemonth/IntermediateBuildFilesPath/Pods.build/Release-iphoneos/Braintree.build/Braintree-own-target-headers.hmap (in target 'Braintree' from project 'Pods')
write-file /Users/runner/Library/Developer/Xcode/DerivedData/bookofthemonth-ajnelrlzioikrefiqgehrdieliqy/Build/Intermediates.noindex/ArchiveIntermediates/bookofthemonth/IntermediateBuildFilesPath/Pods.build/Release-iphoneos/Braintree.build/Braintree-own-target-headers.hmap
WriteAuxiliaryFile /Users/runner/Library/Developer/Xcode/DerivedData/bookofthemonth-ajnelrlzioikrefiqgehrdieliqy/Build/Intermediates.noindex/ArchiveIntermediates/bookofthemonth/IntermediateBuildFilesPath/Pods.build/Release-iphoneos/Braintree.build/Braintree.hmap (in target 'Braintree' from project 'Pods')
write-file /Users/runner/Library/Developer/Xcode/DerivedData/bookofthemonth-ajnelrlzioikrefiqgehrdieliqy/Build/Intermediates.noindex/ArchiveIntermediates/bookofthemonth/IntermediateBuildFilesPath/Pods.build/Release-iphoneos/Braintree.build/Braintree.hmap

I verified that neither the Braintree Pod nor any of its dependencies were updated yesterday.

Actual behavior
In 20201011.1, the build logs shows the following output instead:

CreateBuildDirectory /Users/runner/Library/Developer/Xcode/DerivedData/bookofthemonth-ajnelrlzioikrefiqgehrdieliqy/Build/Intermediates.noindex/ArchiveIntermediates/bookofthemonth/IntermediateBuildFilesPath (in target 'Braintree' from project 'Pods')
CreateBuildDirectory /Users/runner/Library/Developer/Xcode/DerivedData/bookofthemonth-ajnelrlzioikrefiqgehrdieliqy/Build/Intermediates.noindex/ArchiveIntermediates/bookofthemonth/InstallationBuildProductsLocation (in target 'Braintree' from project 'Pods')
CreateBuildDirectory /Users/runner/Library/Developer/Xcode/DerivedData/bookofthemonth-ajnelrlzioikrefiqgehrdieliqy/Build/Intermediates.noindex/ArchiveIntermediates/bookofthemonth/BuildProductsPath (in target 'Braintree' from project 'Pods')
WriteAuxiliaryFile /Users/runner/Library/Developer/Xcode/DerivedData/bookofthemonth-ajnelrlzioikrefiqgehrdieliqy/Build/Intermediates.noindex/ArchiveIntermediates/bookofthemonth/IntermediateBuildFilesPath/Pods.build/Release-iphoneos/Braintree.build/Braintree.hmap (in target 'Braintree' from project 'Pods')
write-file /Users/runner/Library/Developer/Xcode/DerivedData/bookofthemonth-ajnelrlzioikrefiqgehrdieliqy/Build/Intermediates.noindex/ArchiveIntermediates/bookofthemonth/IntermediateBuildFilesPath/Pods.build/Release-iphoneos/Braintree.build/Braintree.hmap
WriteAuxiliaryFile /Users/runner/Library/Developer/Xcode/DerivedData/bookofthemonth-ajnelrlzioikrefiqgehrdieliqy/Build/Intermediates.noindex/ArchiveIntermediates/bookofthemonth/IntermediateBuildFilesPath/Pods.build/Release-iphoneos/Braintree.build/Braintree-own-target-headers.hmap (in target 'Braintree' from project 'Pods')
write-file /Users/runner/Library/Developer/Xcode/DerivedData/bookofthemonth-ajnelrlzioikrefiqgehrdieliqy/Build/Intermediates.noindex/ArchiveIntermediates/bookofthemonth/IntermediateBuildFilesPath/Pods.build/Release-iphoneos/Braintree.build/Braintree-own-target-headers.hmap

Repro steps
I will try to reproduce with a clean React Native project that matches our dependency setup and post here.

maxim-lobanov commented 4 years ago

Hello @crisvergara , could you please provide repro steps for your issue and full build logs? If your repository is private, I suggest contact App Center Support team and they will be able to retrieve information for investigation.

crisvergara commented 4 years ago

I got a response from appcenter support. 

Thanks for your reply. After checked the build log, I found all the errors are double-quoted include "XXX.h" in framework header, expected angle-bracketed instead [-Werror,-Wquoted-include-in-framework-header].

This issue can be solved in two ways.

Upgrading Cocoapods to the pre-release version 1.10.0.You can upgrade the cocoapods by build script: $ [sudo] gem install cocoapods --pre. Then click save and build from build config
Another is to try to set have CLANG_WARN_QUOTED_INCLUDE_IN_FRAMEWORK_HEADER to NO in Xcode Settings.
Please let us know if these works or not.

The flag CLANG_WARN_QUOTED_INCLUDE_IN_FRAMEWORK_HEADER is unset in my project. The default is NO, but that's changing in Xcode 12: https://github.com/CocoaPods/CocoaPods/issues/9902

I'm going to try running gem install cocoapods --pre in my build script and see if that helps, but AFAIK the new image still uses Xcode 11.3, and shouldn't be having this problem.

crisvergara commented 4 years ago

Build logs here: https://gist.github.com/crisvergara/078218e45379321c5924e0546b118ab2

maxim-lobanov commented 4 years ago

Thank you for additional details. Changelog for this image will be available soon. The most important changes:

Considering that you use Node.JS 10 and Xcode 11.3, It shouldn't impact you. Also no changes in cocoapods version or something similar. We don't have much experience with Cocoapods so probably App Center supports have better knowledge here.

We can just share a few advices:

Target was changed Braintree to BraintreeDropIn-Braintree-UIKit-Localization between builds. I guess that it is root cause but not sure why it happened. We don't do something on image side that could cause it.

crisvergara commented 4 years ago

I tried pushing a previously successful commit, and unfortunately I had the same issue. For that reason, I don't think my code is the problem. Only change as far as I could tell was the image.

I'm still trying to get my post-clone script to trigger, but app center can be finicky sometimes. I'll keep you updated.

crisvergara commented 4 years ago

Updating cocoapods in a post-clone script seems to fix the issue for me.

LeonidLapshin commented 4 years ago

I'll close the issue for now, but feel free to reopen if the problem will occur again or if you have any concerns. Thank you!

evnik commented 4 years ago

@LeonidLapshin We have the similar issue for our native iOS project. We use Xcode 12, have pods versions fixed and manually install 1.8.4 version of CocoaPods in the Post Clone Script. We didn't have any code changes, but our App Center builds started to fail since yesterday on the virtual environment with version: 20201011.1 with this error:

CompileC /Users/runner/Library/Developer/Xcode/DerivedData/Project-fanhuvervdijnmceetejwbevyblj/Build/Intermediates.noindex/ArchiveIntermediates/Target/IntermediateBuildFilesPath/Pods.build/Release-iphoneos/Braintree-7ed364bf.build/Objects-normal/arm64/PPOTSwitchRequest.o /Users/runner/work/1/s/Pods/Braintree/BraintreePayPal/PayPalOneTouch/Models/PPOTSwitchRequest.m normal arm64 objective-c com.apple.compilers.llvm.clang.1_0.compiler (in target 'Braintree-7ed364bf' from project 'Pods')
    cd /Users/runner/work/1/s/Pods
    export LANG\=en_US.US-ASCII
    /Applications/Xcode_12.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -x objective-c -target arm64-apple-ios8.0 -fmessage-length\=0 -fdiagnostics-show-note-include-stack -fmacro-backtrace-limit\=0 -std\=gnu11 -fobjc-arc -fobjc-weak -fmodules -gmodules -fmodules-cache-path\=/Users/runner/Library/Developer/Xcode/DerivedData/ModuleCache.noindex -fmodules-prune-interval\=86400 -fmodules-prune-after\=345600 -fbuild-session-file\=/Users/runner/Library/Developer/Xcode/DerivedData/ModuleCache.noindex/Session.modulevalidation -fmodules-validate-once-per-build-session -Wnon-modular-include-in-framework-module -Werror\=non-modular-include-in-framework-module -fmodule-name\=Braintree -Wno-trigraphs -fpascal-strings -Os -fno-common -Wno-missing-field-initializers -Wno-missing-prototypes -Werror\=return-type -Wdocumentation -Wunreachable-code -Wquoted-include-in-framework-header -Wno-implicit-atomic-properties -Werror\=deprecated-objc-isa-usage -Wno-objc-interface-ivars -Werror\=objc-root-class -Wno-arc-repeated-use-of-weak -Wimplicit-retain-self -Wduplicate-method-match -Wno-missing-braces -Wparentheses -Wswitch -Wunused-function -Wno-unused-label -Wno-unused-parameter -Wunused-variable -Wunused-value -Wempty-body -Wuninitialized -Wconditional-uninitialized -Wno-unknown-pragmas -Wno-shadow -Wno-four-char-constants -Wno-conversion -Wconstant-conversion -Wint-conversion -Wbool-conversion -Wenum-conversion -Wno-float-conversion -Wnon-literal-null-conversion -Wobjc-literal-conversion -Wshorten-64-to-32 -Wpointer-sign -Wno-newline-eof -Wno-selector -Wno-strict-selector-match -Wundeclared-selector -Wdeprecated-implementations -DPOD_CONFIGURATION_RELEASE\=1 -DCOCOAPODS\=1 -DNS_BLOCK_ASSERTIONS\=1 -DOBJC_OLD_DISPATCH_PROTOTYPES\=0 -isysroot /Applications/Xcode_12.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.0.sdk -fstrict-aliasing -Wprotocol -Wdeprecated-declarations -g -Wno-sign-conversion -Winfinite-recursion -Wcomma -Wblock-capture-autoreleasing -Wstrict-prototypes -Wno-semicolon-before-method-body -Wunguarded-availability -fembed-bitcode -iquote /Users/runner/Library/Developer/Xcode/DerivedData/Project-fanhuvervdijnmceetejwbevyblj/Build/Intermediates.noindex/ArchiveIntermediates/Target/IntermediateBuildFilesPath/Pods.build/Release-iphoneos/Braintree-7ed364bf.build/Braintree-generated-files.hmap -I/Users/runner/Library/Developer/Xcode/DerivedData/Project-fanhuvervdijnmceetejwbevyblj/Build/Intermediates.noindex/ArchiveIntermediates/Target/IntermediateBuildFilesPath/Pods.build/Release-iphoneos/Braintree-7ed364bf.build/Braintree-own-target-headers.hmap -I/Users/runner/Library/Developer/Xcode/DerivedData/Project-fanhuvervdijnmceetejwbevyblj/Build/Intermediates.noindex/ArchiveIntermediates/Target/IntermediateBuildFilesPath/Pods.build/Release-iphoneos/Braintree-7ed364bf.build/Braintree-all-non-framework-target-headers.hmap -ivfsoverlay /Users/runner/Library/Developer/Xcode/DerivedData/Project-fanhuvervdijnmceetejwbevyblj/Build/Intermediates.noindex/ArchiveIntermediates/Target/IntermediateBuildFilesPath/Pods.build/Release-iphoneos/Braintree-7ed364bf.build/all-product-headers.yaml -iquote /Users/runner/Library/Developer/Xcode/DerivedData/Project-fanhuvervdijnmceetejwbevyblj/Build/Intermediates.noindex/ArchiveIntermediates/Target/IntermediateBuildFilesPath/Pods.build/Release-iphoneos/Braintree-7ed364bf.build/Braintree-project-headers.hmap -I/Users/runner/Library/Developer/Xcode/DerivedData/Project-fanhuvervdijnmceetejwbevyblj/Build/Intermediates.noindex/ArchiveIntermediates/Target/BuildProductsPath/Release-iphoneos/Braintree-7ed364bf/include -I/Users/runner/Library/Developer/Xcode/DerivedData/Project-fanhuvervdijnmceetejwbevyblj/Build/Intermediates.noindex/ArchiveIntermediates/Target/IntermediateBuildFilesPath/Pods.build/Release-iphoneos/Braintree-7ed364bf.build/DerivedSources-normal/arm64 -I/Users/runner/Library/Developer/Xcode/DerivedData/Project-fanhuvervdijnmceetejwbevyblj/Build/Intermediates.noindex/ArchiveIntermediates/Target/IntermediateBuildFilesPath/Pods.build/Release-iphoneos/Braintree-7ed364bf.build/DerivedSources/arm64 -I/Users/runner/Library/Developer/Xcode/DerivedData/Project-fanhuvervdijnmceetejwbevyblj/Build/Intermediates.noindex/ArchiveIntermediates/Target/IntermediateBuildFilesPath/Pods.build/Release-iphoneos/Braintree-7ed364bf.build/DerivedSources -F/Users/runner/Library/Developer/Xcode/DerivedData/Project-fanhuvervdijnmceetejwbevyblj/Build/Intermediates.noindex/ArchiveIntermediates/Target/BuildProductsPath/Release-iphoneos/Braintree-7ed364bf -Wall -Werror -Wextra -include /Users/runner/work/1/s/Pods/Target\ Support\ Files/Braintree-7ed364bf/Braintree-7ed364bf-prefix.pch -MMD -MT dependencies -MF /Users/runner/Library/Developer/Xcode/DerivedData/Project-fanhuvervdijnmceetejwbevyblj/Build/Intermediates.noindex/ArchiveIntermediates/Target/IntermediateBuildFilesPath/Pods.build/Release-iphoneos/Braintree-7ed364bf.build/Objects-normal/arm64/PPOTSwitchRequest.d --serialize-diagnostics /Users/runner/Library/Developer/Xcode/DerivedData/Project-fanhuvervdijnmceetejwbevyblj/Build/Intermediates.noindex/ArchiveIntermediates/Target/IntermediateBuildFilesPath/Pods.build/Release-iphoneos/Braintree-7ed364bf.build/Objects-normal/arm64/PPOTSwitchRequest.dia -c /Users/runner/work/1/s/Pods/Braintree/BraintreePayPal/PayPalOneTouch/Models/PPOTSwitchRequest.m -o /Users/runner/Library/Developer/Xcode/DerivedData/Project-fanhuvervdijnmceetejwbevyblj/Build/Intermediates.noindex/ArchiveIntermediates/Target/IntermediateBuildFilesPath/Pods.build/Release-iphoneos/Braintree-7ed364bf.build/Objects-normal/arm64/PPOTSwitchRequest.o
In file included from /Users/runner/work/1/s/Pods/Braintree/BraintreePayPal/PayPalOneTouch/Models/PPOTSwitchRequest.m:15:
In file included from /Users/runner/work/1/s/Pods/Braintree/BraintreePayPal/PayPalDataCollector/PPDataCollector_Internal.h:8:
/Users/runner/work/1/s/Pods/Braintree/BraintreePayPal/PayPalDataCollector/Public/PPDataCollector.h:9:9: error: double-quoted include "PPRMOCMagnesResult.h" in framework header, expected angle-bracketed instead [-Werror,-Wquoted-include-in-framework-header]
#import "PPRMOCMagnesResult.h"
        ^~~~~~~~~~~~~~~~~~~~~~
        <PPRMOCMagnesResult.h>
1 error generated.
evnik commented 4 years ago

@LeonidLapshin @maxim-lobanov @MaksimZhukov Could you please reopen this issue?

maxim-lobanov commented 4 years ago

@evnik , could you please provide minimal project with repro steps for this issue? Unfortunately, we can't reproduce it from our side so we can't proceed with further investigation until we have repro steps.

We have a bunch of Cocoapods test apps that we use for image validation and they work as expected with new image. I guess that it can be something specific related to Braintree module.

maxim-lobanov commented 4 years ago

I have found a few related issues where customers report the same problem (inside CI and even on local machine):

Could you please try provided solutions to see if it helps?

evnik commented 4 years ago

@maxim-lobanov thank you for reopening. The issues you mentioned exist for a while, but we've never seen them on our project. And we use specific CocoaPods version. The image version is the only thing that was changed for our builds. We have raised an issue with App Center support and the issue ID should be #KG8AE0CL, will you be able to follow up using that? Meanwhile I can try to build an example to reproduce the issue.

evnik commented 4 years ago

@maxim-lobanov actually, it's quite easy to reproduce:

  1. I've forked Braintree iOS SDK
  2. Setup App Center build for master branch in that repo with the next configuration:
    • Project/Workspace: Braintree.xcworkspace
    • Shared Scheme: Demo
    • Xcode version: 12.0
  3. Save & Build

Build is failed with similar errors. Here is the build log BraintreeDemoBuild1.zip

MaksimZhukov commented 4 years ago

Hello everyone! We have investigated this problem deeper, analyzed change log of new image but unfortunately, we still don’t know the root cause of this problem. Neither CocoaPods nor any other tools have been updated.

Actually, this issue is known issue with CocoaPods on Xcode 12, please follow these links for more information: 1) https://github.com/CocoaPods/CocoaPods/issues/9902 2) https://stackoverflow.com/questions/63646607/xcode-12-using-firebase-pods-lexical-or-preprocessor-issue-pb-h-file-not-fou/63648164#63648164 3) https://developer.apple.com/forums/thread/651941

The root cause is that Xcode setting CLANG_WARN_QUOTED_INCLUDE_IN_FRAMEWORK_HEADER default value has been changed from No to Yes in Xcode 12.  CocoaPods team has acknowledged the issue and fixed it in 1.10.0.rc.1.

There are the following ways to mitigate an issue: 1) Use Cocoapods 1.10.0.rc.1:

sudo gem uninstall cocoapods --all
sudo gem install cocoapods -v 1.10.0.rc.1

2) Set CLANG_WARN_QUOTED_INCLUDE_IN_FRAMEWORK_HEADER to No in your project settings

The single question is why it started to fail only on the new image. Based on information in issues, this problem is related to Xcode 12.0 itself and should have reproduced earlier with Xcode 12. The single suspicious change on this image is removing Xcode 12 beta 6, previously it was installed side by side with Xcode 12.0 GM but we have removed it because it is not required anymore (please find the details here). Previously, we have already faced with issues when multiple Xcode versions installed side by side can conflict and override settings of each other. Probably, it is the reason why we didn’t see this issue earlier.

We are working on further investigation.

evnik commented 4 years ago

Set CLANG_WARN_QUOTED_INCLUDE_IN_FRAMEWORK_HEADER to No in your project settings

The project settings change won't fix it, but change in the Pods project does. To do it, you might add post_install hook to the Podfile:

post_install do |installer|
  installer.pods_project.build_configurations.each do |config|
    config.build_settings['CLANG_WARN_QUOTED_INCLUDE_IN_FRAMEWORK_HEADER'] = 'NO'
  end
end

I confirm this fixes the issue on my fork with CocoaPods 1.9.3

evnik commented 4 years ago

issues when multiple Xcode versions installed side by side can conflict and override settings of each other

@MaksimZhukov That makes sense, because I could reproduce the issue even if I select Xcode 11.x in the build configuration (I mean before the workaround for CocoaPods was applied)

maxim-lobanov commented 4 years ago

Hello @evnik @crisvergara

We have found root cause of this issue. It is caused by one of Cocoapods dependencies. Despite the fact that Cocoapods itself was not updated, one of its dependencies xcodeproj was updated. As a result, pod install forces CLANG_WARN_QUOTED_INCLUDE_IN_FRAMEWORK_HEADER to YES for all projects. It explains why it impacts all Xcode versions.

For now, we are working to rollback previous image and deploy the new one with freezing previous version of xcodeproj dependency. I have reported an issue to cocoapods. Let's wait for their answer: https://github.com/CocoaPods/CocoaPods/issues/10153

J5Dev commented 4 years ago

Hello @evnik @crisvergara

We have found root cause of this issue. It is caused by one of Cocoapods dependencies. Despite the fact that Cocoapods itself was not updated, one of its dependencies xcodeproj was updated. As a result, pod install forces CLANG_WARN_QUOTED_INCLUDE_IN_FRAMEWORK_HEADER to YES for all projects. It explains why it impacts all Xcode versions.

For now, we are working to rollback previous image and deploy the new one with freezing previous version of xcodeproj dependency. I have reported an issue to cocoapods. Let's wait for their answer: CocoaPods/CocoaPods#10153

Thank you for this, and your response on this Issue: https://github.com/CocoaPods/CocoaPods/issues/10153

We also use App Centre, and can confirm that rolling back to Cocoapods version 1.10.0.rc1 has resolved the issue and our builds are working again.

dmitry-shibanov commented 4 years ago

Hello everyone, we rolled back our image and added tests to prevent such kind of issues in future. I close the issue. Feel free to reopen the issue if you have any concerns.