Carthage 0.37.0 goes into an infinite loop of building subprojects forever

1oo7 commented 3 years ago

carthage install method: [ ] .pkg, [ ] homebrew, [X] source
which carthage: (using a custom nested location inside our repo)
carthage version: 0.37.0
xcodebuild -version: 12.3 (12C33)
Are you using --no-build? no
Are you using --no-use-binaries? yes
Are you using --use-submodules? no
Are you using --cache-builds? no
Are you using --new-resolver? yes

Cartfile

github "kean/Nuke" ~> 9.0

Carthage Output

carthage build --no-use-binaries --use-xcframeworks --cache-builds --platform iOS --configuration Release  
*** xcodebuild output can be found in /var/folders/w7/d9dqqx_938qfd43_vrf060900000gp/T/carthage-xcodebuild.CIhZ9f.log
*** No cache found for Nuke, building with all downstream dependencies
*** Building scheme "Nuke" in Nuke.xcodeproj
xcodebuild timed out while trying to read NukeDemo.xcodeproj 😭

Actual outcome On Carthage 0.37.0, building Nuke always fails with timeout.

This is because Carthage builds all projects in a repo, and Nuke's repo includes NukeDemo, which in turn has the following four remote Swift Package dependencies:

Alamofire
SwiftSVG
Gifu
Nuke

Cloning each of those repos requires over 2 minutes on my laptop with gigabit fiber ethernet. Here is the result of those repos getting cloned and then checked out:

Subsequent carthage bootstrap operations result in Carthage building each of the Xcode projects found nested in the Alamofire/, SwiftSVG/, Gifu/, and Nuke/ subdirectories of Carthage/Checkouts/Nuke/Demo/DerivedData/NukeDemo/SourcePackages/checkouts/. Since that Nuke directory also contains a copy of the NukeDemo project, these subsequent carthage bootstrap operations result in an ongoing infinite loop of building NukeDemo and all its dependency projects, punctuated only by timeout errors when Carthage eventually gives up.

Here is a detailed analysis over time of the events that occur during the first carthage bootstrap operation, with timestamps:

[16:43:42] ran carthage bootstrap --use-ssh --no-use-binaries --platform iOS --configuration Release --cache-builds --new-resolver
[16:44:51] Carthage begins cloning Nuke into Carthage/Checkouts/Nuke
[16:46:15] Carthage begins building Carthage/Checkouts/Nuke/Nuke.xcproj
[16:46:23] Carthage begins building Carthage/Checkouts/Nuke/Demo/NukeDemo.xcproj
[16:46:24] SPM begins cloning Nuke into Carthage/Checkouts/Nuke/Demo/DerivedData/NukeDemo/SourcePackages/repositories/Nuke-8db81083
[16:46:32] SPM begins cloning Gifu into Carthage/Checkouts/Nuke/Demo/DerivedData/NukeDemo/SourcePackages/repositories/Gifu-05fd1ae0
[16:47:53] SPM begins cloning SwiftSVG into Carthage/Checkouts/Nuke/Demo/DerivedData/NukeDemo/SourcePackages/repositories/SwiftSVG-ae27d62c1
[16:48:11] SPM begins cloning Alamofire into Carthage/Checkouts/Nuke/Demo/DerivedData/NukeDemo/SourcePackages/repositories/Alamofire-11b78eba
[16:49:13-14] SPM creates subdirectories and checks out these repos into Nuke/, Gifu/, SwiftSVG/, and Alamofire/ in Carthage/Checkouts/Nuke/Demo/DerivedData/NukeDemo/SourcePackages/checkouts/
[16:49:15] carthage boostrap operation fails with error: xcodebuild timed out while trying to read NukeDemo.xcodeproj 😭

Then I ran a subsequent carthage bootstrap operation:

[16:52:01] ran carthage bootstrap --use-ssh --no-use-binaries --platform iOS --configuration Release --cache-builds --new-resolver
[16:53:15] Carthage begins trying to build Nuke again, picking up where it left off in step 9. above.
[16:53:48] Carthage begins building Alamofire in Carthage/Checkouts/Nuke/Demo/DerivedData/NukeDemo/SourcePackages/checkouts/Alamofire/Alamofire.xcproj
[16:54:00] Carthage begins building SwiftSVG in Carthage/Checkouts/Nuke/Demo/DerivedData/NukeDemo/SourcePackages/checkouts/SwiftSVG/SwiftSVG.xcproj
[16:54:05] Carthage begins building Gifu in Carthage/Checkouts/Nuke/Demo/DerivedData/NukeDemo/SourcePackages/checkouts/Gifu/Gifu.xcproj
[16:54:09] Carthage begins building Nuke in Carthage/Checkouts/Nuke/Demo/DerivedData/NukeDemo/SourcePackages/checkouts/Nuke/Nuke.xcproj
[16:54:26] Carthage begins building NukeDemo in Carthage/Checkouts/Nuke/Demo/DerivedData/NukeDemo/SourcePackages/checkouts/Nuke/Demo/NukeDemo.xcproj
[16:54:30] SPM begins cloning Nuke into Carthage/Checkouts/Nuke/Demo/DerivedData/NukeDemo/SourcePackages/checkouts/Nuke/Demo/DerivedData/NukeDemo/SourcePackages/repositories/Nuke-8db81083
[16:54:38] SPM begins cloning Gifu intoCarthage/Checkouts/Nuke/Demo/DerivedData/NukeDemo/SourcePackages/checkouts/Nuke/Demo/DerivedData/NukeDemo/SourcePackages/repositories/Gifu-05fd1ae0
[16:55:51] SPM begins cloning SwiftSVG into Carthage/Checkouts/Nuke/Demo/DerivedData/NukeDemo/SourcePackages/checkouts/Nuke/Demo/DerivedData/NukeDemo/SourcePackages/repositories/SwiftSVG-ae27d62c1
[16:57:21] carthage boostrap operation fails with error: xcodebuild timed out while trying to read NukeDemo.xcodeproj 😭

If I run carthage boostrap again, guess what happens?

The infinite cycle continues.

Each time carthage bootstrap gets run after this, Carthage adds another nested build of NukeDemo, SPM clones all its SPM dependencies (including Nuke!), and Carthage builds those, which results in the further nested NukeDemo getting built, all its SPM dependencies getting cloned, etc. until timeout and 😭.

I.e., we will end up with:

Carthage/Checkouts/ Nuke/Demo/DerivedData/NukeDemo/SourcePackages/checkouts/ Nuke/Demo/DerivedData/NukeDemo/SourcePackages/checkouts/ Nuke/Demo/DerivedData/NukeDemo/SourcePackages/checkouts/ Nuke/Demo/DerivedData/NukeDemo/SourcePackages/checkouts/ ...etc. forever.

But why does Carthage build both Nuke.xcproj and NukeDemo.xcproj, in the first place? Shouldn't Carthage ignore other projects that are nested in subfolders?

Well, I looked into this question, and I learned the following information:

Carthage is designed to build all the .xcproj's in each dependency repo. This is how it is intended to work. (RxSwift issue 34, which was fixed in RxSwift PR 35; Carthage issue 1974, particularly this comment)
Carthage building NukeDemo is therefore not a bug, and is not due to anything peculiar to my local setup.
Carthage building the things inside Nuke/Demo/DerivedData/NukeDemo/

This comment from Carthage issue 1974 mentions that "Carthage always prefers to build through a workspace," making it sound like adding an .xcworkspace that only contains Nuke (but not NukeDemo) could be a possible solution. However, I tried this, and the problem still happens. So that's not a workable solution.

Interestingly, I found an older thread from 2014 where someone had made this comment:

Carthage only finds and builds a single project. If that weren't the case, it would erroneously pick up dependency/example/demo projects, which is rarely what one wants. I think the current behavior is reasonable.

... and this comment:

Carthage finds the root-most project and builds that. So it expects to find one framework project, and ignores all others (some of which might include examples/demos, dependencies, etc.). This results in the best behavior in ≥ 80% of cases. So, if you have multiple framework projects, the correct solution is to combine them into one, and use targets and schemes to separate builds—not projects.

So.. it seems at some point that Carthage's developers decided "the current behavior is reasonable" was wrong, and that it was better to "erroneously pick up dependency/example/demo projects" than to just build the root project of a repo. Perhaps for some projects that's fine, but obviously for projects like Nuke, which have a nested demo project that depends on the parent project as a remote SPM dependency, that can cause an infinite loop causing Carthage to always fail.

Perhaps the old wisdom of only building the root project was much better, however I doubt Carthage's current maintainers will want to reverse course on this, since it would probably break something else.

Instead I might suggest to simply not build any .xcproj with the word "Demo" in the name. Or, don't build any .xcproj that happens to be nested inside a DerivedData folder. But ah, something should probably be done about this.

Note: the current workaround we're using for our app to pass CI, is to run carthage checkout, then rm -rf Carthage/Checkouts/Nuke/Demo, then carthage build.

Expected outcome Carthage should not get stuck in an infinite loop building SPM checkouts.

emawby commented 3 years ago

This same issue is occurring for the OneSignal SDK because it also contains the OneSignalDemo project

1oo7 commented 3 years ago

This same issue is occurring for the OneSignal SDK because it also contains the OneSignalDemo project

What we did with Nuke, was that we forked it, and in the fork, we deleted the folder Demo. Then we updated our Cartfile and Cartfile.resolved to point at that fork instead of the mainline.

Another option is that you can have a CI script that does a carthage checkout, then rm -rf Carthage/Checkouts/Nuke/Demo, then carthage build. That's fine for CI, but if local developers are doing manual carthage bootstraps etc., they might not run the same script as CI.

elliottwilliams commented 3 years ago

@1oo7 If I'm following correctly, Nuke's derived data is ending up within it's checkout directory, right? That's very odd. Typically a dependency's derived data should be in ~/Library/Caches/org.carthage.CarthageKit/DerivedData/, where Xcode can safely do things like clone swift package repos.

1oo7 commented 3 years ago

Elliot, the problem is not that Nuke has a DerivedData folder.

The problem is that Carthage doesn't just build Nuke (which is defined by the root-level project. It also builds Carthage/Checkouts/Nuke/Demo/NukeDemo.xcproj, which in turn, uses Swift Package Manager for four dependencies—one of which, is Nuke! By the time those four dependencies have been cloned (which takes forever because SPM does deep clones), the build will have timed out.

Then on the next Carthage Build, now Carthage recursively finds and tries to build all the nested Xcode projects hidden within Carthage/Checkouts/Nuke/Demo/DerivedData/NukeDemo/SourcePackages/checkouts/—one of which is yet another copy of both Nuke and NukeDemo.

Now, I read somewhere that if you add an .xcworkspace to your dependency repo, then Carthage is supposed to build only the stuff that is actually linked into that .xcworkspace. However, it totally ignores that .xcworkspace!

@elliottwilliams wrote:

Nuke's derived data is ending up within it's checkout directory, right?

All of my Carthage dependencies, for as long as I can remember, have always had a DerivedData folder in Checkouts/DEPENDENCY_NAME/.

That's very odd. Typically a dependency's derived data should be in ~/Library/Caches/org.carthage.CarthageKit/DerivedData/, where Xcode can safely do things like clone swift package repos.

Actually on my computer, a DerivedData appears in both places. For example right now I have:

~/Library/Caches/org.carthage.CarthageKit/DerivedData/12.3_12C33/Nuke/9.1.0/Build/Products/Release-iphonesimulator/ (which contains Nuke.framework and Nuke.framework.dSYM)
~/code/REDACTED/Carthage/Checkouts/nuke/DerivedData/Nuke/Build/Intermediates.noindex/Nuke.build/Release/Nuke.build/ (which only contains a single, mysterious file: dgph)

Meanwhile the DerivedData for NukeDemo only appears in Carthage/Checkouts/Nuke/Demo/DerivedData but not in /Library/Caches/org.carthage.CarthageKit/DerivedData/12.3_12C33/Nuke/9.1.0.

BTW, dgph is evidently a "dependency graph" file that Xcode creates to facilitate incremental builds. Here are its contents:

DGPH1.04Dec  1 202012:28:19/UsersREDACTEDcodeios-12-3Carthage   CheckoutsNuke

Note: The build was conducted 12 Mar. 2021 at 2:43 AM PDT. I have no idea why Dec. 1, 2020 is referenced in this file.

Note:

In Xcode itself, I always use "Relative" for DerivedData folder paths. Could that be messing things up somehow?
Also, we're using the following workaround script, which is necessary due to this bug: https://github.com/Carthage/Carthage/issues/3149 and because we need to override the settings of a particular outdated dependency so that it can still build:

#!/usr/bin/env bash

# carthage-build.sh
# Usage example: ./carthage-build.sh --platform iOS

set -euo pipefail

xcconfig=$(mktemp /tmp/static.xcconfig.XXXXXX)
trap 'rm -f "$xcconfig"' INT TERM HUP EXIT

# override other build settings to resolve certain
# build issues with older dependencies whose projects haven't
# been updated to be compatible with Xcode 12 and later

echo 'GCC_TREAT_WARNINGS_AS_ERRORS=NO' >> $xcconfig
echo 'GCC_C_LANGUAGE_STANDARD=compiler-default' >> $xcconfig
echo 'SWIFT_TREAT_WARNINGS_AS_ERRORS=NO' >> $xcconfig

# Fix for issue where Carthage would strip iPad icons out of REDACTED.
# TODO: Remove this once Carthage has been updated to address this.
# https://rREDACTED

echo 'TARGET_DEVICE_MODEL=""' >> $xcconfig
echo 'ASSETCATALOG_FILTER_FOR_DEVICE_MODEL=""' >> $xcconfig

export XCODE_XCCONFIG_FILE="$xcconfig"
Scripts/CarthageXC/__carthage "$@"

... where Scripts/CarthageXC/__carthage is the Carthage 0.37.0 binary. (We keep the Carthage binary checked into our source repo so that its version will be pinned to a specific commit. It's also nice in that it allows us to test bugfixes to Carthage without having to update whatever carthage version happens to be avaialble on a given CI node.

But I'm sure that using Carthage from this special path has nothing to do with the issue described above, because I've tried building Nuke using a freshly-installed Carthage straight from homebrew.

This really seems like a bug where Carthage recursively tries to build every single xcode project anywhere in a the tree of folders starting with the root folder of each repo.

elliottwilliams commented 3 years ago

All of my Carthage dependencies, for as long as I can remember, have always had a DerivedData folder in Checkouts/DEPENDENCY_NAME/.

@1oo7 oh, this might explain why I can't reproduce this then! Open any project in Xcode, and check you derived data locaiton settings (Xcode > Preferences > Locations):

If your default is "Relative", then it's going to put derived data alongside checkouts, and cause this issue. Set it to "Default", delete your Carthage/Checkouts/Nuke, and I bet this will fix itself.

I'd be interested to know if Carthage can override this default somehow in its build environment. It seems like we would only want to use a relative derived data path if a project is specifically configured for it.

This really seems like a bug where Carthage recursively tries to build every single xcode project anywhere in a the tree of folders starting with the root folder of each repo.

Yeah, I can't speak to 2014-era Carthage, but the status quo is that we build any framework in a repo that we have a shared scheme for. I don't think we can reasonably change that at this point – there are too many libraries that depend on it.

emawby commented 3 years ago

@elliottwilliams I don't believe that this is caused by using "relative" for the derived data location. I always use "default" and my path looks just like yours, but I am able to reproduce this issue for the OneSignal SDK (github "OneSignal/OneSignal-iOS-SDK"). This does not happen using Carthage 0.34.

elliottwilliams commented 3 years ago

This does not happen using Carthage 0.34.

@emawby To make sure I understand: on 0.34.0, you still have a derived data folder in the checkout directory, but it doesn't infinitely loop like this?

When I try to reproduce this, I don't get a Carthage/Checkouts/Nuke/Demo/DerivedData, so I'm inclined to think it's some setting on your system that's causing problems.

emawby commented 3 years ago

@elliottwilliams No I never have a derived data folder in the checkouts directory, but I get a timeout building the OneSignalDemo project when trying to use the OneSignalSDK from Carthage

elliottwilliams commented 3 years ago

These seem like different issues. OneSignal appears to be having problems resolving its swift packages. If I check it out and run

xcodebuild -list -project Carthage/Checkouts/OneSignal-iOS-SDK/Examples/ObjectiveCExample/OneSignalDemo.xcodeproj/

it hangs at Fetching https://github.com/OneSignal/OneSignal-iOS-SDK.git.

Update: since I wrote the above, I tried again and I can carthage build OneSignal now. Still having problems on your end?

emawby commented 3 years ago

@elliottwilliams It isn't working yet but I see that is the problem. Thank you for your help! When I remove the demo projects from the repository and point my cartfile at my locally changed repo I am not able to build using the new --use-xcframeworks option (the non XCFramework option works fine). I get the error

Failed to write to /Users/elliotmawby/Documents/GitHub/CarthageTest/Carthage/Build/OneSignal.xcframework: Error Domain=NSCocoaErrorDomain Code=260 "The file “OneSignal.xcframework” couldn’t be opened because there is no such file." UserInfo={NSFilePath=/var/folders/zs/95c835q97xzdxpkvmsqb5hlm0000gn/T/carthage-xcframework-FPvXFV/OneSignal.xcframework, NSUnderlyingError=0x7fc1a400c6f0 {Error Domain=NSPOSIXErrorDomain Code=2 "No such file or directory"}}

vanthanhtran245 commented 3 years ago

@emawby It seems you're using Xcode 11 right?. Should try with Xcode 12

1oo7 commented 3 years ago

Update: since I wrote the above, I tried again and I can carthage build OneSignal now. Still having problems on your end?

It's important to note, there is an issue in Xcode 12.4 and earlier, in which SPM makes a deep clone of any source repos into the DerivedData folder. Sometimes this can take long enough to cause Carthage to timeout while building. But on a subsequent rebuild, it's usually fine because it doesn't have to do the entire deep cone from scratch (as long as the DerivedData of that Carthage Checkouts has not been deleted.)

We communicated this issue to Apple. As a response, in Xcode 12.5, SPM will make a per-user cache somewhere in ~/Library, that stores these deep clones. That way, once you've done that first deep clone, subsequent carthage bootstrap operations won't be at risk of failing, unless you delete that deep clone in ~/Library.

Hopefully this will reduce that timeout issue.

1oo7 commented 3 years ago

@elliottwilliams

If your default is "Relative", then it's going to put derived data alongside checkouts, and cause this issue. Set it to "Default", delete your Carthage/Checkouts/Nuke, and I bet this will fix itself.

Interesting. I also thought of trying that, but then I forgot to try it, haha.

I think the infinite loop bug would also be resolved if Carthage would simply ignore any path with "SourcePackages". Seems like that would be easy to do.

Another option could be to add support for a .carthageIgnore file to a repo, with a syntax like a .gitignore file. That way repo authors could add a .carthageIgnore file for any Demo projects or similar. And you could also add a .carthageIgnore to your dependencies, to act like a global override so you can take matters into your own hands to exclude some thing that is getting built even though it shouldn't.

Just some ideas. Not sure how feasible they are but thought I would throw it out there. Hope it's helpful

emawby commented 3 years ago

@vanthanhtran245 Thank you I was using Xcode 12 but my command line tools were set to Xcode 11. It is working now thank you very much!

1oo7 commented 3 years ago

So I think this is still an issue. The most reasonable approach to solving this problem would be to ignore any projects that exist inside a folder named, "DerivedData", and/or ignore any projects that have the word "Demo" in the name.

Ideally this would be added to Carthage as a build option like this:

carthage bootstrap --platform iOS --ignore-deriveddata --no-use-binraries --other-args --etc --etc

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

imWildCat commented 2 years ago

Stil seeing this issue using Xcode 13.x. Any news?

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

ilammy commented 2 years ago

Just because nobody wants to fix the issue – doesn't mean it's not actual anymore.

_{(Well, I personally decided to stop caring. Carthage build fails again? Whatever, it can keep crying wolf. So you can say that it is not actual, in a way.)}

iosdevben commented 2 years ago

Is Carthage end of life?

vixentael commented 2 years ago

We still experience this... As a workaround we remove all test projects before compiling Carthage.

https://github.com/Carthage/Carthage/issues/3148#issuecomment-843676007

Carthage / Carthage

Carthage 0.37.0 goes into an infinite loop of building subprojects forever #3148