XCRemoteCache prototype yielding unexpected results and looking for guidance on how to debug

[cezarsignori]

I am trying out XCRemoteCache and got an unexpected outcome. I am unsure how to interpret it and about how best to investigate and solve the problems I found. Thought of asking here for guidance.

What did I do?

I performed the "automatic integration" of XCRemoteCache in my workspace (for 329 xcodeproj files). I set my machine up to be a producer, then built the app and verified all (329) artifacts were correctly stored in localhost. Then, I re-configured my machine to be a consumer, deleted the DerivedData directory, and clean built the app.

What happened?

Unfortunately only 60s were saved (from a total of 30 minutes).

After inspecting the Xcode build log, I noticed the following:

1. All 329 projects have this warning in the pre-build step:

"all-product-headers.yaml doesn't exist. Skipping overlay for the best-effort mode."

2. 157 projects reported "[RC] Cached build for X target"
3. 172 projects reported "[RC] Disabled remote cache for X"

The problem

After inspecting the XCRemoteCache code, I found out "[RC] Cached build for X target" means a cache hit and "[RC] Disabled remote cache for X" the opposite. Now, I expect 157 modules cache hits to save a lot more than 60s total. So I wonder if XCRemoteCache is actually doing what it is supposed to do on my project and if not, why?

Also, I clean built the same branch without any code changes in the same machine. Why did 172 modules failed to hit cache? And how do I investigate the issue more effectively? Right now, I am reading the XCRemoteCache code looking for issues that could cause a miss and trying to identify a pattern across those modules in my workspace, since all build artifacts are they where should be (logs match files and location on local server).

And lastly, it is not clear to me if the pre-build warning is a problem or not and how to get rid of it.

I appreciate any insights. Thank you!

[dwirandytlvk]

I think you can see xcremotecache logs by using

# Logs from the all XCRemoteCache commands (last 10 mins)
log show --predicate 'sender BEGINSWITH "xc"' --style compact --info --debug -last 10m

See more: https://github.com/spotify/XCRemoteCache/blob/master/docs/FAQ.md

[cezarsignori]

Thanks, @dwirandytlvk. Following your tip, I found out those 172 modules the consumer is failing to find in the cache have the following error:

Prebuild step failed with error: missingFile(file:///$home/Library/Developer/Xcode/DerivedData/$appname-bkgyochpbymbjodvowqymxzemazn/Build/Products/Debug-iphonesimulator/$library.swiftmodule/x86_64-apple-ios-simulator.swiftmodule.md5)

There are no errors in the producer log and the md5 files can be found on the server.

I do not know yet why these md5 files are missing in the consumer.

I also found 175 occurrences of Df xcswiftc[61764:3194e5] Fallbacking to compilation using swiftc. in the consumer logs. It is not an error but I am not sure how it influences the error above, if at all. I noticed this warning can show up if new Swift files were added, but in my case, git status shows no new files (I'm on the very same commit). Not sure what's going on.

Did anyone experience these issues or have any hypotheses on what could be going on?

[polac24]

Hello! The warning about all-product-headers.yaml is not fatal, it should be just reported to non-user-facing logs. Will fix that.

I prepared a list of steps to troubleshoot low cache hit rate - PR: https://github.com/spotify/XCRemoteCache/pull/93 I can recommend starting with 7. & 8: find the first target with a miss and review its meta json file for any absolute paths - there are many reasons why absolute paths may sneak into it.

Fallbacking to compilation using swiftc is an indication of the cache miss.

[cezarsignori]

Thanks, @polac24. By following your troubleshooting steps I discovered those Prebuild step failed with error: missingFile errors do in fact happen for modules with absolute paths in their json files.

The xcodeproj files are generated via BUCK and all the .h and .modulemap files in the buck-out directory are listed using absolute paths. The same happens with .h, .modulemap, and .swiftinterface files from dependencies set up as Pods.

Looks like I need to fix this on the BUCK side.

Are any other integration problems with BUCK known?

....

Additionally, I also saw the following errors in the producer side:

1. 14x Postbuild step failed with error: zipFail
2. 23x Postbuild step failed with error: Error Domain=NSCocoaErrorDomain Code=260 "The file “x86_64” couldn’t be opened because there is no such file." UserInfo={NSURL=file:///_$user_/Library/Developer/Xcode/DerivedData/_$project_-bkgyochpbymbjodvowqymxzemazn/Index/Build/Intermediates.noindex/_$library_.build/Debug-iphonesimulator/_$library_.build/Objects-normal/x86_64, NSFilePath=/_$user_/Library/Developer/Xcode/DerivedData/_$project_-bkgyochpbymbjodvowqymxzemazn/Index/Build/Intermediates.noindex/_$library_.build/Debug-iphonesimulator/_$library_.build/Objects-normal/x86_64, NSUnderlyingError=0x7fd02a805ba0 {Error Domain=NSPOSIXErrorDomain Code=2 "No such file or directory"}} and those files are really not there for these targets in successful builds.

And these errors on the consumer side:

1. 7x Prebuild step was disabled for current commit: unavailable
2. 335x Couldn't verify if should disable RC for 49c1401c91ff4de9284f80d6cea12bd667b43393" (then RC downloads and unzips successfully)
3. 187x Prebuild step failed with error: missingFile......swiftmodule.md5

[polac24]

Hi! We haven't verify BUCK's integration, but according to your comment you just consume .swiftmodule/.(xc)framework/.a etc. so it should be easy to map these external buck paths using out_of_band_mappings: ["BUCK": "/absolute/path/of/the/buck/output] in your .rcinfo. Similarly for Pods dependencies.

Producer:

1. We have never observed any zipFail error, would interesting to investigate that. The error sounds really serious as failing to create an artifact will lead to a cache miss and as I wrote earlier, that might have a ripple effect.
2. I recall that issue reported in #33 but #42 should fix it. Probably you are on M1. Do your targets have XCRC_PLATFORM_PREFERRED_ARCH a workaround for Xcode's bug? If so, does XCRC_PLATFORM_PREFERRED_ARCH resolve to an expected architecture (like arm64 for AppleSilicon simulator)? Are you trying to build an artifact for a current architecture or a fat one according to docs?

Consumer:

1. Probably caused by Producer's 1.
2. Oh, that is a red-herring. You can ignore that.
3. Probably caused by absolute paths in meta and/or Producer's 2.

[cezarsignori]

Thank you @polac24 for the insights and sorry for the delay. I went on holidays and now changed machines.

I am (now) using a M1 MAX with Xcode 13.2.1 and RC 0.3.7. My xcodeproj files define ARCHS = arm64 armv7 and ONLY_ACTIVE_ARCH = YES. PLATFORM_PREFERRED_ARCH is not set, only XCRC_PLATFORM_PREFERRED_ARCH = arm64.

With the new machine, the zipFail errors are gone.

As suggested, I used the out_of_band_mappings config with the absolute paths. Then the producer logs showed occurrences of Found url to remapp and error wise, only Prebuild step disabled, selected mode: producer and 400+ ocurrences similar to:

2022-03-03 10:08:26.811 E  xcodebuild[62776:1cec9b] objc[62776]: Class AMSupportURLSession is implemented in both /usr/lib/libamsupport.dylib (0x1f798b6c8) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x106760318). One of the two will be used. Which one is undefined.

As far as I can tell, libamsupport.dylib is not a dependency of my project.

And the consumer build does not leverage the cached artifacts, also shows the Found url to remapp logs and still contains 174+ issues similar to:

2022-03-03 10:26:35.874 E  xcprebuild[74293:1da2b8] (_{$module}_) Prebuild step failed with error: missingFile(file:///Users/csignori/Library/Developer/Xcode/DerivedData/_{$app_project}_-bkgyochpbymbjodvowqymxzemazn/Build/Products/Debug-iphonesimulator/_{$library}_.swiftmodule/arm64-apple-ios-simulator.swiftmodule.md5)

where _{$library}_ is always one of the same 12.

The consumer logs also contain the same amount (174) of Fallbacking to compilation using swiftc messages.

So, it looks like:

1. There is an issue with 12 specific modules that is failing RC for 174 modules;
2. That libamsupport.dylib error I don't know where it is coming from;
3. The out_of_band_mappings does remove the absolute paths but requires actual absolute paths to be set in the .rcinfo file (e.g., /Users/csignori/Projects/iphone). Given those paths change for each developer working on the project, it cannot really be used as a solution for that purpose right now. Relative path (e.g. to the root repo) support would change that. I see someone requested that feature (https://github.com/spotify/XCRemoteCache/issues/77). Any plans on that?

Regarding #1, I did not find yet any issues or particularities about those 12 modules.

[cezarsignori]

I tried to not use RC for those 12 modules to see if cache hits would improve, but what I saw instead was the return of those "The file “x86_64” couldn’t be opened because there is no such file." errors in the producer log.

I simply deleted the .rcinfo, user.rcinfo and re-generated those projects. I checked the ARCHS, ONLY_ACTIVE_ARCH, XCRC_PLATFORM_PREFERRED_ARCH settings and they are unchanged. I also deleted DerivedData and the cache directory in the sample REST server.

As I was surprised to see that again, I deleted DerivedData, cache and ~/Library/Caches/XCRemoteCache directories and clean built again.

And I am now consistently stuck with that error again.

[polac24]

Hi! Did you check these ARCHS, ONLY_ACTIVE_ARCH, XCRC_PLATFORM_PREFERRED_ARCH from the build logs or only in the Xcode IDE? I found that Xcode doesn't correctly report ARCH for instance. I am asking because ARCHS = arm64 armv7 suggests you are building for a device, not simulator.

1. On a consumer side, I suggest taking one of 12 _{$library}_, clean its DerivedData/Intermediate/{Library}and build it from Xcode (most likely with a dedicated scheme that builds only that particular target). If logs don't give any hints, you may need to debug it locally. I would also look if that build adds something to the~/Library/Caches/XCRemoteCache. A new file in theartifactsdir suggests that the local state hash matches with the producer. 1b. Worth checking if there is anything in/Users/csignori/Library/Developer/Xcode/DerivedData/_{$appproject}-bkgyochpbymbjodvowqymxzemazn/Build/Products/Debug-iphonesimulator/{$library}.swiftmodule/`. Do you have {arch}.{swiftmodule|swiftdoc} at least? Do you maybe use library-evolution (.swiftinterface) for that library?
2. That AMSupportURLSession seems to be a log noise, saw a comment on AppleForum
3. You can workaround that in two ways: 1) generated out_of_band_mappings on-fly locally and add it to the user.rcinfo (or any other extra_configuration_file you define), 2) leverage a brand new custom_rewrite_envs configuration. Probably you already have an Xcode build setting that points to the buck's output folder so you could add the name of that build settings to custom_rewrite_envs
4. Probably not but aren't you running Xcode in the Rosetta? That could set x86_64.

[cezarsignori]

Hi!

Let's reduce the problem scope.

I changed workspaces to another that builds within 1 minute. It has 70 projects, each with one or more targets. That is a sub-project of the previous one and therefore contains some of those 12 libraries.

I am including my own projects, as well as submodules and Pods (which are committed to the repo).

I am consistently getting the x86_64 error on the producer side (just like before). I think we should ensure the producer is working fine before attempting to understand what's going on the consumer side.

So I checked the Xcode IDE for ARCHS, ONLY_ACTIVE_ARCH, XCRC_PLATFORM_PREFERRED_ARCH. They do show arm64 armv7 and I am building against iPhone 13 Pro simulator using CMD+B or CMD+R.

Now that you brought it up, I also checked the .xcactivitylog file using xclogparser and I only see arm64 as value for the architecture task fields.

And I am not running Xcode in the Rosetta.

Still, when I run log show --predicate 'sender BEGINSWITH "xc"' --style compact --info --debug -last 1m > ~/Desktop/producer.txt, I see many xcpostbuild x86_64 couldn’t be opened because there is no such file errors.

[polac24]

I think that architecture in the .xcactivitylog corresponds to your machine's architecture, not the ARCHS you are building.

The easiest way to inspect ARCH and other ENVs is adding export right after the xcpostbuild command and/or check show ENVs. Then, in the report navigator, you will see all resolved ENVs for that build step:

[cezarsignori]

Yes!

So these are the ENVs containing ARCH I see:

export ARCHS="arm64"
export ARCHS_STANDARD="arm64 x86_64"
export ARCHS_STANDARD_32_64_BIT="arm64 i386 x86_64"
export ARCHS_STANDARD_32_BIT="i386"
export ARCHS_STANDARD_64_BIT="arm64 x86_64"
export ARCHS_STANDARD_INCLUDING_64_BIT="arm64 x86_64"
export ARCHS_UNIVERSAL_IPHONE_OS="arm64 i386 x86_64"
export NATIVE_ARCH="arm64"
export NATIVE_ARCH_32_BIT="arm"
export NATIVE_ARCH_64_BIT="arm64"
export NATIVE_ARCH_ACTUAL="arm64"
export PLATFORM_PREFERRED_ARCH="x86_64"
export VALID_ARCHS="arm64 arm64e i386 x86_64"
export XCRC_PLATFORM_PREFERRED_ARCH="arm64"
export arch="undefined_arch"
export CURRENT_ARCH="undefined_arch"

PLATFORM_PREFERRED_ARCH is indeed x86_64, but also in every other ENV arm64 is 1st in the list (e.g., ARCHS).

I also checked the other ENVs listed in #42 and for me these are their values:

export CURRENT_VARIANT="normal"
export LINK_FILE_LIST_normal_arm64="/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/DMLProto.build/Debug-iphonesimulator/DMLProto_Showdown.build/Objects-normal/arm64/DMLProto_Showdown.LinkFileList"

And LINK_FILE_LIST_normal_x86_64 is not defined, as expected.

From what I read on #33, #39 and #42, I should not be seeing this issue. Do you have any clues?

[cezarsignori]

BTW I forced PLATFORM_PREFERRED_ARCH = arm64 to see if the problem would go away and even though the export run script build step shows the arm64 value, the RC log still shows the x86_64 errors.

[cezarsignori]

@polac24 I have good and bad news 📦

I got both producer and consumer to work. The producer logs did not contain Postbuild step failed with error and the consumer logs did not contain Prebuild step failed with error: missingFile errors. Plus a clean consumer build (I deleted DerivedData and ~/Library/Caches/XCRemoteCache) took 13s instead of 44s! (Note: It would be nice to have a confirmation everything worked as expected in the RC log).

I basically cut all the corners (to identify what problems have to be addressed). Meaning that I overrode all those ENVs so they don't not contain x86_64 and manually moved each auto-generated [RC] RemoteCache_prebuild step to the first position after Dependencies step.

With that working, my plan was to undo change individually to find out what is causing the issue.

However (here come the bad news), without doing any changes (just deleting caches) and re-building in producer mode a few more times and randomly got Postbuild step failed with error (this time with "The file “arm64” couldn’t be opened because there is no such file.") and zipFail errors again.

That means I will not be able to undo changes individually and reliably test the outcome to identify the issue.

Have you seen this behavior before? Any tips on how to debug or isolate the problem?

[polac24]

I was thinking about that and the case is really strange. I have never seen that. One last guess could be outdated ModeCache. How do you delete cache? Do you nuke full DerivedData including ModuleCache or just CMD+SHIFT+K? I would fully delete DerivedData.

TBH, probably building xcpostbuild from Xcode and debugging the flow locally is inevitable. For instance, it is possible that some custom tooling adds something to the compiler and *.d file contains non-existing references (Docs on how we read .d).

[cezarsignori]

To delete the caches, I run rm -rf ~/Library/Developer/Xcode/DerivedData and rm -rf ~/Library/Caches/XCRemoteCache, then, in the docker, I run rm -rf /tmp/cache/cache. Once that's done, I CMD+B on Xcode for Producer mode.

Then, to switch to consumer mode, I do the same (but don't delete the docker folder) and run the automatic integration commands, open Xcode and CMD+B.

About building xcpostbuild from Xcode and debugging the flow locally, is it possible to do that when building my project? How would I got about doing that? (currently the compiled swift binaries are called from the sh build steps)

Thanks again for all the support!

[polac24]

Your steps look legit.

Debugging XCRemoteCache locally is really simple. I prepared a short instruction. Until #99 is in review, you can take a look on https://github.com/polac24/XCRemoteCache/blob/20220304-document-debugging/docs/Development.md#debugging-the-app-in-a-real-project

[cezarsignori]

Thank @polac24

Before we go into solving the Producer mode issue (random missing file errors), I did some more testing with the successful case I mentioned (14s consumer build with no errors) and found:

1. The debugger is not working. Xcode does not stop on breakpoints and shows "Xcode won't pause at this breakpoint because it has not been resolved" when hovering over the breakpoint; Since there are no absolute paths in the meta files and no errors in the log, I was surprised to find out the debugger is not working.

2. If I hit "Run" a second time, the Build fails with a long error message about a "Cycle in dependencies" between 2 targets and citing their RC pre_build steps. It seems like RC is causing the cyclic dependency error as well. I still need to dig deeper on that one, but BUCK does prevent cyclic dependencies.

These are key issues that take priority over the Producer issues at this point.

Can you help with these issues?

[polac24]

Hello!

1. If you see a hollow breakpoint, please try if you have any remaps in ~/.lldbinit that are too broad. The easiest way is to stop LLDB and call settings show target.source-map. XCRemoteCache by default adds to ~/.lldbinit and maybe you have some leftovers. I assume you have already tried the tip with sleep(1).
2. You mentioned earlier that you reordered build steps. xcprebuild should be right before the compilation step, not earlier. If you still have an error, maybe you can share a cycle message here.

[cezarsignori]

Hi!

This is the content of ~/.lldbinit:

/Users/csignori/Projects/iphone
❯ cat ~/.lldbinit
#RemoteCacheCustomSourceMap
settings set target.source-map /xxxxxxxxxx /Users/csignori/Projects/iphone/vendor/cache/ruby/2.6.0/gems/ffi-1.15.4/ext/ffi_c/libffi

I see the same when running settings show target.source-map:

(lldb) settings show target.source-map
target.source-map (path-map) =
[0] "/xxxxxxxxxx" -> "/Users/csignori/Projects/iphone/vendor/cache/ruby/2.6.0/gems/ffi-1.15.4/ext/ffi_c/libffi"

Deleting the ~/.lldbinit and re-running xcprepare integrate recreates the file with the same content.

This is the content of all my .rcinfo files:

primary_repo: {xcodeproj_repo}
cache_addresses:
- http://localhost:8080/cache
primary_branch: csignori/xcremotecache-proto
out_of_band_mappings: {
 'TOOLS_BUCK-OUT':'/Users/csignori/Projects/iphone/tools/config/buck-out',
 'CORE_MODULES_BUCK-OUT':'/Users/csignori/Projects/iphone/{root_project}/Modules/CoreModules/buck-out',
 'BDUX_BUCK-OUT':'/Users/csignori/Projects/iphone/{root_project}/Modules/Platform/Bdux/buck-out', 
 'MISC_BUCK-OUT':'/Users/csignori/Projects/iphone/{root_project}/Modules/Miscellaneous/buck-out', 
 'PODS_BUCK-OUT':'/Users/csignori/Projects/iphone/Pods/buck-out',
 'SUBMODULES_BUCK-OUT':'/Users/csignori/Projects/iphone/Submodules/buck-out',
 'ROOT_BUCK-OUT':'/Users/csignori/Projects/iphone/buck-out',
 'PODS':'/Users/csignori/Projects/iphone/Pods',
}

Regarding the sleep(1) tip, if you are referring to this one:

Tip: If your breakpoints don't hit, try adding a dummy sleep(1) in the process entry point (e.g. at the top of XCPostbuild.main function)

No, I have not yet built RC from code, so far I just tried making things work on the RC user side, using the binaries provided in the release page. What's behind that sleep(1)?

Regarding the order of build steps, the step I manually moved after the RC prebuild step is a custom code gen step. I did that because in a 100% clean build scenario, the code gen step will gen code first and then RC prebuild would trigger. That could cause RC prebuild step to fail. And even if it did not, if RC prebuild hits the cache, I don't want Xcode to invest time generating code that is not going to be compiled. In any case, I tried building the consumer without reordering the steps and the 2nd build failed as well. Here is the cycle message:

Showing All Errors Only
Cycle in dependencies between targets 'DMLProto_Showdown' and 'Networking'; building could produce unreliable results. This usually can be resolved by moving the target's Headers build phase before Compile Sources.
Cycle path: DMLProto_Showdown → Networking → DMLProto_Showdown
Cycle details:
→ Target 'DMLProto_Showdown' has copy command from '/Users/csignori/Projects/iphone/buck-out/xcode/xcframework-slices/DMLProto_Showdown_Binary-baed0d227d0182fee0f99035cde61e6d6bfb312f/OTPublishersHeadlessSDK-Slice/OTPublishersHeadlessSDK.framework' to '/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Products/Debug-iphonesimulator/DMLProto_Showdown.app/Frameworks/OTPublishersHeadlessSDK.framework'
○ Target 'DMLProto_Showdown' has target dependency on Target 'Networking'
→ Target 'Networking' has compile command for Swift source files
○ That command depends on command in Target 'Networking': script phase “[RC] RemoteCache_prebuild”
○ Target 'Networking' has copy command from '/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/Networking.build/Debug-iphonesimulator/Networking.build/Objects-normal/arm64/Networking-Swift.h' to '/Users/csignori/Projects/iphone/{project}/Modules/CoreModules/buck-out/xcode/derived-sources/Networking-720ecdc4254a9a596d1698dc8a91303c9363b022/Networking-Swift.h'
○ Target 'Networking' has compile command for Swift source files

Raw dependency cycle trace:

target:  ->

node: <all> ->

command: <all> ->

node: /Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Products/Debug-iphonesimulator/DMLProto_Showdown.app/Frameworks/OTPublishersHeadlessSDK.framework ->

command: target-DMLProto_Showdown-111e58bab44a349562ba8c0c4233d65b6d925b55934b21c9ad78429ecb6851c1-:Debug:PBXCp /Users/csignori/Projects/iphone/buck-out/xcode/xcframework-slices/DMLProto_Showdown_Binary-baed0d227d0182fee0f99035cde61e6d6bfb312f/OTPublishersHeadlessSDK-Slice/OTPublishersHeadlessSDK.framework /Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Products/Debug-iphonesimulator/DMLProto_Showdown.app/Frameworks/OTPublishersHeadlessSDK.framework ->

node: <target-DMLProto_Showdown-111e58bab44a349562ba8c0c4233d65b6d925b55934b21c9ad78429ecb6851c1--phase4-copy-bundle-resources> ->

command: Gate target-DMLProto_Showdown-111e58bab44a349562ba8c0c4233d65b6d925b55934b21c9ad78429ecb6851c1--phase4-copy-bundle-resources ->

node: <target-DMLProto_Showdown-111e58bab44a349562ba8c0c4233d65b6d925b55934b21c9ad78429ecb6851c1--phase2-compile-sources> ->

command: Gate target-DMLProto_Showdown-111e58bab44a349562ba8c0c4233d65b6d925b55934b21c9ad78429ecb6851c1--phase2-compile-sources ->

node: <target-DMLProto_Showdown-111e58bab44a349562ba8c0c4233d65b6d925b55934b21c9ad78429ecb6851c1--phase1--rc--remotecache-prebuild> ->

command: Gate target-DMLProto_Showdown-111e58bab44a349562ba8c0c4233d65b6d925b55934b21c9ad78429ecb6851c1--phase1--rc--remotecache-prebuild ->

node: <target-DMLProto_Showdown-111e58bab44a349562ba8c0c4233d65b6d925b55934b21c9ad78429ecb6851c1--phase0-slicing-xcframeworks> ->

command: Gate target-DMLProto_Showdown-111e58bab44a349562ba8c0c4233d65b6d925b55934b21c9ad78429ecb6851c1--phase0-slicing-xcframeworks ->

node: <target-DMLProto_Showdown-111e58bab44a349562ba8c0c4233d65b6d925b55934b21c9ad78429ecb6851c1--ModuleMapTaskProducer> ->

command: Gate target-DMLProto_Showdown-111e58bab44a349562ba8c0c4233d65b6d925b55934b21c9ad78429ecb6851c1--ModuleMapTaskProducer ->

node: <target-DMLProto_Showdown-111e58bab44a349562ba8c0c4233d65b6d925b55934b21c9ad78429ecb6851c1--HeadermapTaskProducer> ->

command: Gate target-DMLProto_Showdown-111e58bab44a349562ba8c0c4233d65b6d925b55934b21c9ad78429ecb6851c1--HeadermapTaskProducer ->

node: <target-DMLProto_Showdown-111e58bab44a349562ba8c0c4233d65b6d925b55934b21c9ad78429ecb6851c1--GeneratedFilesTaskProducer> ->

command: Gate target-DMLProto_Showdown-111e58bab44a349562ba8c0c4233d65b6d925b55934b21c9ad78429ecb6851c1--GeneratedFilesTaskProducer ->

node: <target-DMLProto_Showdown-111e58bab44a349562ba8c0c4233d65b6d925b55934b21c9ad78429ecb6851c1--ProductStructureTaskProducer> ->

command: Gate target-DMLProto_Showdown-111e58bab44a349562ba8c0c4233d65b6d925b55934b21c9ad78429ecb6851c1--ProductStructureTaskProducer ->

node: <target-DMLProto_Showdown-111e58bab44a349562ba8c0c4233d65b6d925b55934b21c9ad78429ecb6851c1--begin-compiling> ->

command: Gate target-DMLProto_Showdown-111e58bab44a349562ba8c0c4233d65b6d925b55934b21c9ad78429ecb6851c1--begin-compiling ->

node: <target-Networking-f5133eebb2ffb8190194ce0ce8a1ee02a28b59ec223466b45be229c4860cfbd9--modules-ready> ->

command: Gate target-Networking-f5133eebb2ffb8190194ce0ce8a1ee02a28b59ec223466b45be229c4860cfbd9--modules-ready ->

node: /Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/Networking.build/Debug-iphonesimulator/Networking.build/Objects-normal/arm64/JSONExtractable.o ->

CYCLE POINT ->

command: target-Networking-f5133eebb2ffb8190194ce0ce8a1ee02a28b59ec223466b45be229c4860cfbd9-:Debug:CompileSwiftSources normal arm64 com.apple.xcode.tools.swift.compiler ->

node: /Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/Networking.build/Debug-iphonesimulator/Networking.build/Objects-normal/arm64/Networking.SwiftFileList ->

command: target-Networking-f5133eebb2ffb8190194ce0ce8a1ee02a28b59ec223466b45be229c4860cfbd9-:Debug:WriteAuxiliaryFile /Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/Networking.build/Debug-iphonesimulator/Networking.build/Objects-normal/arm64/Networking.SwiftFileList ->

node: <target-Networking-f5133eebb2ffb8190194ce0ce8a1ee02a28b59ec223466b45be229c4860cfbd9--phase0--rc--remotecache-prebuild> ->

command: Gate target-Networking-f5133eebb2ffb8190194ce0ce8a1ee02a28b59ec223466b45be229c4860cfbd9--phase0--rc--remotecache-prebuild ->

node: /Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/Networking.build/Debug-iphonesimulator/Networking.build/rc.enabled ->

command: target-Networking-f5133eebb2ffb8190194ce0ce8a1ee02a28b59ec223466b45be229c4860cfbd9-:Debug:PhaseScriptExecution [RC] RemoteCache_prebuild /Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/Networking.build/Debug-iphonesimulator/Networking.build/Script-AA5B8D6D56EF38FEF8F7FDC5.sh ->

node: /Users/csignori/Projects/iphone/{project}/Modules/CoreModules/buck-out/xcode/derived-sources/Networking-720ecdc4254a9a596d1698dc8a91303c9363b022/Networking-Swift.h ->

command: target-Networking-f5133eebb2ffb8190194ce0ce8a1ee02a28b59ec223466b45be229c4860cfbd9-:Debug:PBXCp /Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/Networking.build/Debug-iphonesimulator/Networking.build/Objects-normal/arm64/Networking-Swift.h /Users/csignori/Projects/iphone/{project}/Modules/CoreModules/buck-out/xcode/derived-sources/Networking-720ecdc4254a9a596d1698dc8a91303c9363b022/Networking-Swift.h ->

node: /Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/Networking.build/Debug-iphonesimulator/Networking.build/Objects-normal/arm64/Networking-Swift.h/ ->

directoryTreeSignature: � ->

directoryContents: /Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/Networking.build/Debug-iphonesimulator/Networking.build/Objects-normal/arm64/Networking-Swift.h ->

node: /Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/Networking.build/Debug-iphonesimulator/Networking.build/Objects-normal/arm64/Networking-Swift.h ->

command: target-Networking-f5133eebb2ffb8190194ce0ce8a1ee02a28b59ec223466b45be229c4860cfbd9-:Debug:CompileSwiftSources normal arm64 com.apple.xcode.tools.swift.compiler

DMLProto_Showdown is the workspace I am building and running and Networking is one of the dependencies.

[polac24]

Hi!

1. Released version of XCRemoteCache is not debuggable. You have to build it locally
2. Does that cycle exist if you don't have xcprebuild? It would be nearly impossible to track your cyclic error without access to sources. If you publish a small project that reproduces a problem, that would work.

[cezarsignori]

Okay, so I have now tried to debug RC following the instructions on Development.md. But it did not work :O

The RC project builds and runs (waiting for execution) just fine, but when I build my project (with build steps updated to point to the produced binary in DerivedData), instead stopping on the RC breakpoints, Xcode shows "Finished running xcpostbuild : xcpostbuild" in the status bar and my project also does fully build the code (it does not reuse the cache). RC logs do not show any errors (I only notice some Swiftc noop for arm64-apple-ios13.2-simulator info messages).

Xcode stopped to use the cached artifacts once I run xcprepare integrate using the locally built XCRemoteCache. Is the cache invalidated because I am using a "different version of XCRemoteCache" to the producer's?

I also added the sleep(1) as suggested but my project still can't be debugged. Though since the cache was not used and RC's project was automatically finished, maybe the test is invalid.

And regarding the cycle, no, neither BUCK nor Xcode complain of cycles unless I ran xcprepare integrate.

[cezarsignori]

@polac24 I just realized I might not have been clear when I first said that debugging is not working, but I meant my project is no longer debuggable after xcprepare integrate is run. I didn't originally mean I can't debug the released version of RC.

In my last message, I did download the RC code and was also unable to debug it by following the instructions. The first time around, I built the Aggregator and manually changed the pre and post build steps to use the binaries from DerivedData. It did not work. Then I re-run the xcprepare integrate using the binaries from DerivedData (instead of released version), which automatically creates the pre/post build steps pointing to the binaries in DerivedData. At this point, my project's build time went back to before RC integration (44s instead of 14s) and the Xcode "Finished running" in the RC project instead of stopping in breakpoints there.

[polac24]

It becomes challenging to track so many questions in a single issue. Can you create separate issues for independent problems?

• Regarding sleep: xcprebuild/xcpostbuild are quite often quick (exit within miliseconds) thus LLDB cannot attach a debugger session, before it finishes. A sleep slows the process down a bit so Xcode has some extra time to set up a session.

• Regarding producer/consumer XCRemoteCache version incompatibility: it is possible that artifacts generated from an older version are not valid so the general rule of thump is to use the same version for a producer and consumer. There are many code changes that may affect the fingerprint, e.g. here

[cezarsignori]

Sure, thanks @polac24!

I created the following issues to cover all the pending topics on this thread:

• 100

• 101

• 102

• 103

From my point of view, these issues are sorted by priority. At this point I am looking to validate RC as a viable solution (and the debugger must work on my application) and to estimate the effort to adopt it properly in case it is viable. But I do need to sort out all these issues (particularly the first 3 in case I move forward with RC).

Thanks again!

[polac24]

@cezarsignori, can we then close this ticket?

[cezarsignori]

Sorry, it took me a while to see your message. Sure. Thanks for your help!

[hendych]

Hi @cezarsignori did you find out the root cause of missingFile(file:///Users/xxx/Library/Developer/Xcode/DerivedData/App-ahzgtkptlculxqeozvwnarksfyoq/Build/Products/Debug-iphonesimulator/$LIB/$LIBframework/Modules/$LIB.swiftmodule/x86_64-apple-ios-simulator.swiftmodule.md5?

From this issue I understand that it has something to do with absolute path? What is the expected path and what to fix?

Thank you!

[cezarsignori]

Hey @hendych , sorry for the late response. I was off on holidays. The reason and fix is unclear. I shifted focus to other RC adoption topics and eventually it went away. My hypothesis is that the issue was limited to indexing. From Xcode 13, indexing also builds. The fix was to set ARCHS=arm64 when indexing. Please see full discussion at https://github.com/spotify/XCRemoteCache/issues/102.

[hendych]

Hi @cezarsignori thanks for your respond! I found the issue, btw. For future devs bumping into this issue: I was trying to implement xcrc on modular environment, while using monolith project for the cache producer. The root cause of my issue is that while using cocoapods-xcremotecache turns out I was using different subspec in one of my pod between the monolith and modular environment. Thus the project configuration is different between those environment.