chore(ios): release build failed with `xcodebuild timed out while trying to read iOS-ObjectiveC.xcodeproj`

[mcdurdin]

https://build.palaso.org/buildConfiguration/Keyman_iOS_Master/437227?buildTab=log&linesState=589.646.667.678&logView=flowAware&focusLine=634

xcodebuild timed out while trying to read iOS-ObjectiveC.xcodeproj

[mcdurdin]

Rebuild resolved this. Assuming network or similar timeout.

[mcdurdin]

Turns out the rebuild was on another agent. Original agent continued to fail.

This fixed it there:

rm -rf ~/Library/Caches/org.carthage.CarthageKit
rm -rf ~/Library/Caches/carthage

Don't recommend making this change to the agents for all builds because Carthage is SLOW after clearing caches.

Ref: https://stackoverflow.com/a/45504898/1836776

[mcdurdin]

The cache clearing resolved the problem, but only for one build. After one successful build on the agent, the same error returned. Not clear if this is a Carthage bug or what but the cache clearing took over a minute and then the carthage part of the build took 25-30 minutes longer than normal so really not keen on running this regularly.

[mcdurdin]

I have, for now, put the cache-clearing incantation into the CI scripts for Test iOS and Release iOS. Will monitor.

[mcdurdin]

No good. Still failing sometimes.

https://build.palaso.org/buildConfiguration/Keyman_iOS_TestSamplesAndTestProjects/438157?buildTab=log&linesState=84&logView=flowAware&focusLine=112

17:38:48   Downloading sil_euro_latin.kmp from downloads.keyman.com
17:38:48   Downloading nrc.en.mtnt.model.kmp from downloads.keyman.com
17:38:50   *** Checking out Reachability.swift at "v5.1.0"
17:38:50   *** Checking out DeviceKit at "5.1.0"
17:38:50   *** Checking out sentry-cocoa at "8.15.2"
17:38:50   *** Checking out dependency-XCGLogger at "57a7b975dbb6fe4fe90cef3d1bc52b8adbd89113"
17:38:50   *** Checking out ZIPFoundation at "0.9.17"
17:38:52   *** xcodebuild output can be found in /var/folders/8p/cxrv8j995qjfb5kgz2p_d5_w0000gn/T/carthage-xcodebuild.uBAJTW.log
17:38:57   *** Building scheme "ObjcExceptionBridging (iOS)" in XCGLogger.xcodeproj
17:39:05   *** Building scheme "XCGLogger (iOS)" in XCGLogger.xcodeproj
17:39:26   *** Building scheme "DeviceKit" in DeviceKit.xcworkspace
17:39:46   *** Building scheme "Reachability" in Reachability.xcodeproj
17:43:13   xcodebuild timed out while trying to read Sentry.xcodeproj

[mcdurdin]

Feels like the appropriate answer is to attempt to migrate away from Carthage because this kind of instability in the package management is probably not solvable at our end. Plenty of discussion online about the design choices in Carthage which are contributing to this, but no real solutions.

[mcdurdin]

        let task = xcodebuildTask("-list", BuildArguments(project: self))

        return task.launch()
            .ignoreTaskData()
            .mapError(CarthageError.taskError)
            // xcodebuild has a bug where xcodebuild -list can sometimes hang
            // indefinitely on projects that don't share any schemes, so
            // automatically bail out if it looks like that's happening.
            .timeout(after: 60, raising: .xcodebuildTimeout(self), on: QueueScheduler())

https://github.com/Carthage/Carthage/blob/edf085cc652b9be8270720c21d2ca1b9fb12ff94/Source/CarthageKit/XCDBLDExtensions.swift#L71-L79

and

    public static func load(with arguments: BuildArguments, for action: BuildArguments.Action? = nil, with environment: [String: String]? = nil) -> SignalProducer<BuildSettings, CarthageError> {
        // xcodebuild (in Xcode 8.0) has a bug where xcodebuild -showBuildSettings
        // can hang indefinitely on projects that contain core data models.
        // rdar://27052195
        // Including the action "clean" works around this issue, which is further
        // discussed here: https://forums.developer.apple.com/thread/50372
        //
        // "archive" also works around the issue above so use it to determine if
        // it is configured for the archive action.
        let task = xcodebuildTask(["archive", "-showBuildSettings", "-skipUnavailableActions"], arguments, environment: environment)

        return task.launch()
            .ignoreTaskData()
            .mapError(CarthageError.taskError)
            // xcodebuild has a bug where xcodebuild -showBuildSettings
            // can sometimes hang indefinitely on projects that don't
            // share any schemes, so automatically bail out if it looks
            // like that's happening.
            .timeout(after: 60, raising: .xcodebuildTimeout(arguments.project), on: QueueScheduler(qos: .default))

https://github.com/Carthage/Carthage/blob/edf085cc652b9be8270720c21d2ca1b9fb12ff94/Source/CarthageKit/BuildSettings.swift#L47-L65

are the only two places where this timeout can happen. What would be the outcome if we extended that timeout to say 5 minutes? It feels like it's actually just busy because sometimes it works, and sometimes it doesn't... (I have no idea why these actions would take 1 minute in the first place)

[mcdurdin]

I've installed a custom build of carthage onto hba-03 which has disabled the timeouts for these commands. Will continue to monitor.

[mcdurdin]

Closing as several builds have passed, will reopen if problem is not resolved.