Closed ahmednafei closed 2 years ago
I would like to share an interesting example:
Last Friday EoD I pulled from master, regenerated the project and clean built the app successfully as RC consumer with 100% cache hit. Then, first thing in the morning today (Monday) I just hit rebuild and the build failed (no pull, no xcprepare
).
The build error was about a missing header file (ImageNameConstants.h
) in GeneratedResources
module.
I found the missing header was referenced in the GeneratedResources.xcodeproj
and the GeneratedResources.h
umbrella header did import it. A symlink existed under buck-out
but did not resolve, as the file in fact, does not exist. It doesn't exist because 3 commits prior to the one I pull and built on Friday migrated that code to Swift (now, there is a ImageNameConstants.swift
instead).
So I regenerated the GeneratedResources
project manually (just like it is automatically done when I regenerate the app workspace as I did on Friday). This process also triggered xcprepare
calls.
Then re-building the app did not fail but I got a few cache misses due to a 404 error when fetching some file from GeneratedResources
.
(GeneratedResources) Prebuild step failed with error: unsuccessfulResponse(status: 404)
I looked for the URL in the RC log. It shows the meta file URL and so I manually requested it. It resolved just fine. Then I saw the headers in the content were correct (no ImageNameConstants.h
) and manually requested the zip file (fileKey
) and it also resolved fine (only header present was -Swift.h
).
Then I regenerated the entire workspace (calling xcprepare
) and the build succeeded with the same cache misses and same 404 error for the same module.
I checked the server logs for 404 error on meta or zip file using the URL and all 3 requests I found were 200 responses.
I then deleted ~/Library/Developer/Xcode/DerivedData
and re-built. This time, no 404 and 100% cache hit rate.
It is strange that the local state of GeneratedResource.xcodeproj
in my machine did not break build on Friday but did today without any local changes.
And it is also strange that the very module who showed the problem had 404 errors on RC consumer side (without matching 404 errors on server log...maybe it failed a file other than meta or zip?) until I deleted DerivedData
.
Here is a another example/use case:
I pulled from master, re-generated project, ran xcprepare.
I did not clean DerivedData
nor Caches
and tried to build the app.
In this case, as I had zero local changes and the producer build succeeded on CI, I expected a successful build with 100% cache hit.
But the build failed with 11 cache misses (and a lot of cache hits).
There were 4 build errors (Command CompileSwiftSources failed with a nonzero exit code
). Only one with clear error message:
clang: error: no such file or directory: '/Users/csignori/Projects/iphone/Modules/Presentation/UseCases/DealsUIComponents/Sources/ViewControllers/ViewsAndModels/CampaignsInfo/BCDealsCampaignBottomSheetController.m' clang: error: no input files
Looking at RC logs I found:
(DealsUIComponents) Prebuild step failed with error: missingFile(file:///Users/csignori/Projects/iphone/Modules/Presentation/UseCases/DealsUIComponents/Sources/ViewControllers/ViewsAndModels/CampaignsInfo/BCDealsCampaignBottomSheetController.h)
The BCDealsCampaignBottomSheetController.h
is really not there (nor in the respective buck-out
directory). But it shows in the meta (which I downloaded manually using the URL in the logs) file under the same location (and also under respective buck-out
directory).
Issue 1: If on the current commit the source file is not there, why does it show up in the meta file?
The other 3 modules all failed due to a local fingerprint mismatch error. DealsUISwiftComponents
mismatched directly and the others depended on a common module which mismatched.
(DealsUISwiftComponents) Local fingerprint Fingerprint(raw: "08b368e2c0c66260f2b719aa17b28607", contextSpecific: "1bbc2585eb75943df45cc57a7b190d01") does not match with remote one be989aa9ef59ce4453a21fd0f2af098c. 2022-05-02 13:24:21.360 I xcprebuild[87326:6479b2] (DealsUISwiftComponents) Remote cache cannot be used ... 2022-05-02 13:23:38.780 I xcprebuild[85272:645596] (BasicUI) Local fingerprint Fingerprint(raw: "ba37be22cf7dcbbf21bfe17573782fd6", contextSpecific: "15891428ea45643221f8c9094bd966d5") does not match with remote one ea26e6cba611d6408ec2b3296b0ead6f. 2022-05-02 13:23:38.780 I xcprebuild[85272:645596] (BasicUI) Remote cache cannot be used
And all the 11 cache misses also either directly suffered the local fingerprint mismatch issue or a dependency did.
Issue 2: I need to delete the Caches
directory to avoid this error.
I then deleted the Caches
directy and re-built the app. I got a different set of build failures matching @Ahmed's:
/Users/csignori/Projects/iphone/Modules/Presentation/UI/SearchResultsList/Sources/ViewControllers/SearchResultsListViewController.swift:21:12: error: cannot load underlying module for 'ApplicationServices' import ApplicationServices
But ApplicationServices
was a cache hit and built sucessfully! And the cache misses from before remained with Prebuild step failed with error: missingFile
erros for a dependency that also missed cache but no reason listed in the log.
I then deleted both the Caches
and DerivedData
directories and re-built the app. Again the build failed with the same error: cannot load underlying module for 'ApplicationServices'
error.
I then deleted both the Caches
and DerivedData
and buck-out
directories. The Xcode workspace lost the destination options. So I re-generated the workspace without triggering xcprepare
. This resulted in the same set of errors.
Issue 3: I believe this indicates the issue is not only with dirty local copy.
It is worth noting that during my testing other commits where made by other developers and pushed to master, triggering other producer runs.
So I deleted everything again, re-generated the workspace and called xcprepare
.
This time, the build succeeded with 3 cache misses. The misses were because of:
2022-05-02 14:31:53.328 I xcprebuild[25786:67b289] (PrivacyUI) Local fingerprint Fingerprint(raw: "4d11fa244eac74dabfade1c1292ce4ee", contextSpecific: "931c853ebe4f19618e8b8273abb0ed2a") does not match with remote one 1206103db295239bf9a6d5b18f877cae. 2022-05-02 14:31:53.328 I xcprebuild[25786:67b289] (PrivacyUI) Remote cache cannot be used
That is an error I also do not expect to see in this scenario, as everything was fetched from the cache server and there were no local changes.
Regarding the cache server, I don't see any errors there nor xcodebuild
failures in the producer side (which would happen upon any server side issues). Unfortunately I don't have access to RC producer logs.
Hello! Glad to hear that you managed to integrate the project with Buck! Great success!
@cezarsignori , regarding So I regenerated the GeneratedResources project manually (...) This process also triggered xcprepare calls.
. That suggests that you call xcprepare
(without mark) several times, one time per .xcodeproj. That could be a reason of the described problem, if each .xcodeproj has unique remote_commit_file
. Can you make sure all projects point to the same file on a disk (you could verify if more than one arc.rc
is present (recursively) in the top project directory)?
I have a premonition that some projects use artifacts generated for different sha. Even at glance I don't see any obvious problems, it might lead to some undiscovered edge cases.
To analyze the original could you share more details about your setup:
xcprepare mark ...
? It seems you use .xcworkspace and maye the docs is not clear that that commit has to be called only one per combination: sha_commit/platform/architecture/Xcode/configuration. For instance if you support 1 Xcode version, simulator&hardware, have x86_64 and arm64 consumers, only iOS app (no WatchApp) and always Debug
: then there should be max 3 invocations per commit : commit/iphonesimulator/arm64/1300/Debug
, commit/iphonesimulator/x86_64/1300/Debug
, commit/iphone/arm64/1300/Debug
xcparepare mark ...
as a last step of the producer, when all artifacts are already uploaded to the server?cache_commit_history = 10
may be too lowIf you CI generation takes significant amount of time, I can suggest having a secondary branch (e.g. main-with-cache
) that is a bit delayed main
but pointing the most recent commit sha with all artifacts uploaded. Then, you could recommend developers to use master-with-cache
for the best dev experience.
Yes?
On Mon, May 2, 2022, 9:25 AM Bartosz Polaczyk @.***> wrote:
Hello! Glad to hear that you managed to integrate the project with Buck! Great success!
@cezarsignori https://github.com/cezarsignori , regarding So I regenerated the GeneratedResources project manually (...) This process also triggered xcprepare calls.. That suggests that you call xcprepare (without mark) several times, one time per .xcodeproj. That could be a reason of the described problem, if each .xcodeproj has unique remote_commit_file. Can you make sure all projects point to the same file on a disk (you could verify if more than one arc.rc is present (recursively) in the top project directory)? I have a premonition that some projects use artifacts generated for different sha. Even at glance I don't see any obvious problems, it might lead to some undiscovered edge cases.
To analyze the original could you share more details about your setup:
- on the producer side, how many times per commit do you call xcprepare mark ...? It seems you use .xcworkspace and maye the docs is not clear that that commit has to be called only one per combination: sha_commit/platform/architecture/Xcode/configuration. For instance if you support 1 Xcode version, simulator&hardware, have x86_64 and arm64 consumers, only iOS app (no WatchApp) and always Debug: then there should be max 3 invocations per commit : commit/iphonesimulator/arm64/1300/Debug, commit/iphonesimulator/x86_64/1300/Debug, commit/iphone/arm64/1300/Debug
- Do you call xcparepare mark ... as a last step of the producer, when all artifacts are already uploaded to the server?
- In the git, do you use merge commit, rebase or squash strategy? If you just rebase without squashing, then default cache_commit_history = 10 may be too low
- That shouldn't be relevant, but maybe could help with troubleshooting: how does your backend server support duplicated PUT requests? Does it overwrite the file or leave the file intact? Assuming you first send PUT with contentA and then PUT with contentB, which will be returned in GET?
- Do you have different projects integrated as git submodules?
If you CI generation takes significant amount of time, I can suggest having a secondary branch (e.g. main-with-cache) that is a bit delayed main but pointing the most recent commit sha with all artifacts uploaded. Then, you could recommend developers to use master-with-cache for the best dev experience.
— Reply to this email directly, view it on GitHub https://github.com/spotify/XCRemoteCache/issues/126#issuecomment-1114877751, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEDHC22AHTJMUWVZSLLWFLVH7JTHANCNFSM5U3BWKIQ . You are receiving this because you were mentioned.Message ID: @.***>
Glad to hear that you managed to integrate the project with Buck! Great success!
Thanks! We are happy too! But we need these issues to go away in order to roll it out to others and validate impact.
That suggests that you call xcprepare (without mark) several times, one time per .xcodeproj. That could be a reason of the described problem...
I call (automatically after project generation) xcprepare
on the consumer side once per repository.
That means xcprepare
on the consumer side is called as many times as that happens (typically once right after pull).
Do you mean that calling it multiple times causes functional problems? Why?
Can you make sure all projects point to the same file on a disk (you could verify if more than one arc.rc is present (recursively) in the top project directory)?
Yes, there is a single arc.rc
file per repo. When there is more than one xcodeproj
file in the repo, their .rcinfo
file refers to that arc.rc
file (e.g. remote_commit_file: ../../../arc.rc
)
On the producer side, how many times per commit do you call
xcprepare mark ...
?
mark
is called once per repository (main repo and submodules) after a successsful build.
Therefore, it is possible that mark
is called more than once per commit, but only for git submodules
(none of the errors experienced so far are related to git submodules
).
Currently, CI builds only 1 time as Producer for each commit in the main repo (Debug
, arm64
, Xcode 13.2.1
).
Do you call xcparepare mark ... as a last step of the producer, when all artifacts are already uploaded to the server?
Yes. First I call xcodebuild
then I call a custom script that calls mark
once per repo.
In the git, do you use merge commit, rebase or squash strategy? If you just rebase without squashing, then default cache_commit_history = 10 may be too low.
Direct master commits are forbidden and Merge Requests are merged via merge commit with option to squash.
That shouldn't be relevant, but maybe could help with troubleshooting: how does your backend server support duplicated PUT requests?
I am using AmazonS3
Java Library and Bucket Versioning
is disabled in AWS. So existing objects are overwritten.
Per official AWS documentation:
"If versioning is not enabled, this operation will overwrite an existing object with the same key; Amazon S3 will store the last write request. Amazon S3 does not provide object locking. If Amazon S3 receives multiple write requests for the same object nearly simultaneously, all of the objects might be stored. However, a single object will be stored with the final write request."
But does RC create duplicated filenames across different commits?
Do you have different projects integrated as git submodules?
Yes. 12.
If you CI generation takes significant amount of time, I can suggest having a secondary branch...
What do you mean? Are these issues expected and part of day to day when using RC?
I pulled commit ff954ae5
from master after CI producer job finished.
The build failed due to -Swift.h
headers.
After deleting both Caches
and DerivedData
directories, the build succeeded with 100% cache hit rate.
After that, rebuilding the app is consistently successful, even calling xcprepare
multiple times (while no other CI job run).
I started a new build of the same commit right when CI was running the second producer build. The build failed with same ApplicationServices
error and after checking the RC logs I met with several Prebuild step failed with error: unsuccessfulResponse(status: 404)
for different modules. I reproduced that three times in a row (before the 3rd attempt I called xcprepare
).
3 commits were made (that I never pulled locally) in between my successful and failed consumer builds of the same commit. Those 3 commits were merged as a single MR.
I picked a random module that failed with a 404 and noticed the meta URL changed:
A) The successful build's meta URL: /rc/meta/ff954ae571d09151f570663db466e41b37061287-DSAssets-Debug-iphonesimulator-13C100-2d9d6c5e8a0c6beb6c7e3676196cf173.json
B) The failed build's meta URL: /rc/meta/ff954ae571d09151f570663db466e41b37061287-DSAssets-Debug-iphonesimulator-13C100-f3e580d5f4035f670832934055078cfa.json
And URL B does not resolve.
It seems like:
When is the meta URL computed? Why is it changed? Maybe the problem is with how the fingerprint is computed?
Your flow seems legit and calling xcprepare
(both with and without mark
) multiple times is fine if you have git submodules (once per a submodule).
The case with different URL A and URL B is really weird, generating the meta URL is quite simple:
https://github.com/spotify/XCRemoteCache/blob/0c2afc15fc1c6add9f9491a8e885eb26ceed2ac6/Sources/XCRemoteCache/Network/URLBuilderImpl.swift#L78
where the envFingerprint
is just a hash (MD5 at the moment) of several ENVs: ENVs listed here:
https://github.com/spotify/XCRemoteCache/blob/3b614c6172ffa50b39163ee25325533beca93029/Sources/XCRemoteCache/Fingerprint/EnvironmentFingerprint.swift#L23-L34
with extra ENVs defined in custom_fingerprint_envs
. version
is hardcoded now to 5
(it is bumped if XCRemoteCache introduces a breaking change):
https://github.com/spotify/XCRemoteCache/blob/3b614c6172ffa50b39163ee25325533beca93029/Sources/XCRemoteCache/Config/XCRemoteCacheConfig.swift#L30
Can you compare these ENVs between successful and failing builds for the same commit?
I am still thinking about other reasons:
xcodebuild clean build...
on the CI prior to the producer build? That is required.But does RC create duplicated filenames across different commits?
. Yes, current implementation relentlessly makes PUT requests, even that file exists on the server (otherwise a prob HEAD would be required). It would be quite simple to add a mode that optimizes PUT requests and never overrides existing files.What do you mean? Are these issues expected and part of day to day when using RC?
. Not at all. That is just a recommendation to have a higher hit rate if developers don't absolutely need the most up-to-date master. However, XCRemoteCache should still be able to selectively build locally only these targets that are not available yet.ApplicationServices
target generated for the commit you built on Friday and the commit built on Monday morning? Do you see ImageNameConstants.h
listed as a dependency only for the Friday one and not for the Monday one?I invested the whole morning in trying to reproduce the 404s until I did by re-generating the xcode projects and calling xcprepare
without pulling and after other producer job rans.
In both successful and failed builds, Xcode logs show the same values for those environment variables. For example:
GCC_PREPROCESSOR_DEFINITIONS: CLANG_COVERAGE_MAPPING: TARGET_NAME: nanopb CONFIGURATION: Debug PLATFORM_NAME: iphonesimulator XCODE_PRODUCT_BUILD_VERSION: 13C100 CURRENT_PROJECT_VERSION: DYLIB_COMPATIBILITY_VERSION: DYLIB_CURRENT_VERSION: PRODUCT_MODULE_NAME: nanopb ARCHS: arm64
With the exception of one module that defines CURRENT_PROJECT_VERSION: 100
all others look like that.
In the failed build RC log I noticed both the 404 errors and local fingerprint fingerprint mismatch errors.
I picked one of the modules (nanopb
) with 404 error and verified there were 2 meta requests in the build log for xcprebuild
:
2022-05-03 11:17:22.806 I xcprebuild[30279:817eb6] (nanopb) Making request https://{url}/rc/meta/8b3c37c27dea9796aefde9a8a4f4c54805def119-nanopb-Debug-iphonesimulator-13C100-6ea319252a122f193b59ab30ab187422.json .... 2022-05-03 11:19:40.252 I xcprebuild[37233:81edb3] (nanopb) Making request https://{url}/rc/meta/8b3c37c27dea9796aefde9a8a4f4c54805def119-nanopb-Debug-iphonesimulator-13C100-262669077fa61748b755747279b40903.json
The nanopb
target shows only one prebuild
and one postbuild
steps and the build log shows only one execution per step.
I though I maybe accumulated the RC log for two different builds, so I deleted everything and built again.
To my surprise I got a successful build with 100% cache hit rate AND again two meta requests for the nanopb
target (and others too). One of them succeeded and the other did not:
2022-05-03 11:34:44.086 I xcprebuild[61908:838bdc] (nanopb) Network request failed with unsuccessful code 404 2022-05-03 11:34:44.086 E xcprebuild[61908:838bae] (nanopb) Prebuild step failed with error: unsuccessfulResponse(status: 404)
But on Xcode I see one execution per step.
Here is all the nanopb
RC logs for the same successful app build (158.6s long):
2022-05-03 11:33:28.713 E xcprebuild[53583:832ec2] (nanopb) Couldn't verify if should disable RC for 8b3c37c27dea9796aefde9a8a4f4c54805def119. 2022-05-03 11:33:28.720 I xcprebuild[53583:832ec2] (nanopb) Found url to remapp: file:///Users/csignori/Library/Developer/Xcode/DerivedData/{app}-bkgyochpbymbjodvowqymxzemazn/Build/Products/. Remapping: /Users/csignori/Library/Developer/Xcode/DerivedData/{app}-bkgyochpbymbjodvowqymxzemazn/Build/Products 2022-05-03 11:33:28.720 I xcprebuild[53583:832ec2] (nanopb) Found url to remapp: file:///Users/csignori/Projects/iphone/Pods/nanopb/. Remapping: /Users/csignori/Projects/iphone/Pods/nanopb 2022-05-03 11:33:28.720 I xcprebuild[53583:832ec2] (nanopb) Found url to remapp: file:///Users/csignori/Projects/iphone/. Remapping: /Users/csignori/Projects/iphone 2022-05-03 11:33:28.721 I xcprebuild[53583:832ec2] (nanopb) Making request https://{url}/rc/meta/8b3c37c27dea9796aefde9a8a4f4c54805def119-nanopb-Debug-iphonesimulator-13C100-6ea319252a122f193b59ab30ab187422.json 2022-05-03 11:33:31.335 I xcprebuild[53583:832ec2] (nanopb) Downloading artifact to file:///Users/csignori/Library/Developer/Xcode/DerivedData/{app}-bkgyochpbymbjodvowqymxzemazn/Build/Intermediates.noindex/nanopb.build/Debug-iphonesimulator/nanopb.build/xccache/81d6178e5cfd5d84381c7d67ee340208.zip 2022-05-03 11:33:31.502 I xcprebuild[53583:832ec2] (nanopb) Artifact unzipped to file:///Users/csignori/Library/Developer/Xcode/DerivedData/{app}-bkgyochpbymbjodvowqymxzemazn/Build/Intermediates.noindex/nanopb.build/Debug-iphonesimulator/nanopb.build/xccache/81d6178e5cfd5d84381c7d67ee340208 2022-05-03 11:33:37.289 I xcpostbuild[54106:833b51] (nanopb) Found url to remapp: file:///Users/csignori/Library/Developer/Xcode/DerivedData/{app}-bkgyochpbymbjodvowqymxzemazn/Build/Products/. Remapping: /Users/csignori/Library/Developer/Xcode/DerivedData/{app}-bkgyochpbymbjodvowqymxzemazn/Build/Products 2022-05-03 11:33:37.289 I xcpostbuild[54106:833b51] (nanopb) Found url to remapp: file:///Users/csignori/Projects/iphone/Pods/nanopb/. Remapping: /Users/csignori/Projects/iphone/Pods/nanopb 2022-05-03 11:33:37.289 I xcpostbuild[54106:833b51] (nanopb) Found url to remapp: file:///Users/csignori/Projects/iphone/. Remapping: /Users/csignori/Projects/iphone 2022-05-03 11:34:41.885 E xcprebuild[61908:838bae] (nanopb) Couldn't verify if should disable RC for 8b3c37c27dea9796aefde9a8a4f4c54805def119. 2022-05-03 11:34:41.894 I xcprebuild[61908:838bae] (nanopb) Found url to remapp: file:///Users/csignori/Library/Developer/Xcode/DerivedData/{app}-bkgyochpbymbjodvowqymxzemazn/Index/Build/Products/. Remapping: /Users/csignori/Library/Developer/Xcode/DerivedData/{app}-bkgyochpbymbjodvowqymxzemazn/Index/Build/Products 2022-05-03 11:34:41.894 I xcprebuild[61908:838bae] (nanopb) Found url to remapp: file:///Users/csignori/Projects/iphone/Pods/nanopb/. Remapping: /Users/csignori/Projects/iphone/Pods/nanopb 2022-05-03 11:34:41.894 I xcprebuild[61908:838bae] (nanopb) Found url to remapp: file:///Users/csignori/Projects/iphone/. Remapping: /Users/csignori/Projects/iphone 2022-05-03 11:34:41.894 I xcprebuild[61908:838bae] (nanopb) Making request https://{url}/rc/meta/8b3c37c27dea9796aefde9a8a4f4c54805def119-nanopb-Debug-iphonesimulator-13C100-262669077fa61748b755747279b40903.json 2022-05-03 11:34:44.086 I xcprebuild[61908:838bdc] (nanopb) Network request failed with unsuccessful code 404 2022-05-03 11:34:44.086 E xcprebuild[61908:838bae] (nanopb) Prebuild step failed with error: unsuccessfulResponse(status: 404) 2022-05-03 11:34:47.461 I xcpostbuild[62385:8392eb] (nanopb) Found url to remapp: file:///Users/csignori/Library/Developer/Xcode/DerivedData/{app}-bkgyochpbymbjodvowqymxzemazn/Index/Build/Products/. Remapping: /Users/csignori/Library/Developer/Xcode/DerivedData/{app}-bkgyochpbymbjodvowqymxzemazn/Index/Build/Products 2022-05-03 11:34:47.461 I xcpostbuild[62385:8392eb] (nanopb) Found url to remapp: file:///Users/csignori/Projects/iphone/Pods/nanopb/. Remapping: /Users/csignori/Projects/iphone/Pods/nanopb 2022-05-03 11:34:47.461 I xcpostbuild[62385:8392eb] (nanopb) Found url to remapp: file:///Users/csignori/Projects/iphone/. Remapping: /Users/csignori/Projects/iphone
The second request comes from the indexing build, more details here: https://github.com/spotify/XCRemoteCache/issues/109#issuecomment-1079640933
Xcode 13 changed the indexing flow and Xcode in the background builds everything twice. If you are curious what happens there, unlock the Indexing build log summary:
To confirm, you could disable indexing globally: defaults write com.apple.dt.Xcode IDEIndexEnable -bool YES
and restart Xcode. There should be only one xcprebuild
and one xcpostbuild
per target.
Do you clean DerivedData (or call xcodebuild clean build... on the CI prior to the producer build? That is required.
Initially I didn't. But since yesterday I do. I suspected the uploaded content could be dirty. I still see the errors.
This is how I run the producer:
#!/bin/bash -e # This is interrupts the script in case of errors (so mark does not run if xcodebuild fails) git clean -ffdx # This is because DerivedData stays in $WORKSPACE. See below. rm -rf ~/Library/Caches/XCRemoteCache ./buckw clean # This step generates xcodeproj with RC steps, and automaticaly creates .rcinfo and user.rcinfo files. ./buckw project GuestApp -c schema.no-swift-time-checks=true -c project.rc-mode=producer xcodebuild -workspace "$WORKSPACE/$XC_WORKSPACE_FILE" \ -scheme "$WORKSPACE_SCHEME" \ -configuration "Debug" \ -destination "platform=iOS Simulator,name=iPhone 13 Pro,OS=15.2" \ -derivedDataPath "$WORKSPACE/deriveddata" \ -disableAutomaticPackageResolution | tee ./build_artifacts/xcodebuild.log | bundle exec xcpretty # This will call RC mark once per repo ./Scripts/Utils/xcremotecache.py --mark 1
But does RC create duplicated filenames across different commits?. Yes.
I don't think there is a problem with overriding files, unless they have different content and same name. In those cases, we probably want a different name (not to skip a PUT call). So to confirm, do you mean RC can generate the same file name for files with different content (e.g. because different commit)? I think that could be an issue.
Can you compare meta files for the ApplicationServices target generated for the commit you built on Friday and the commit built on Monday morning? Do you see ImageNameConstants.h listed as a dependency only for the Friday one and not for the Monday one?
Unfortunately not. I don't have the files and both builds ran on the same commit. I could rebuild that commit and see what I get. But it seems like we should park that one for now and focus on the multiple request and fingerprint difference issue. Perhaps on the files with different content but same name too. Those are much more common and impactful.
Confirmed 404 error is due to Indexing. Turns out indexing builds with ARCHS=x86_64.
export ARCHS\=x86_64 export ARCHS_STANDARD\=arm64\ x86_64 export ARCHS_STANDARD_32_64_BIT\=arm64\ i386\ x86_64 export ARCHS_STANDARD_32_BIT\=i386 export ARCHS_STANDARD_64_BIT\=arm64\ x86_64 export ARCHS_STANDARD_INCLUDING_64_BIT\=arm64\ x86_64 export ARCHS_UNIVERSAL_IPHONE_OS\=arm64\ i386\ x86_64 export NATIVE_ARCH\=arm64 export NATIVE_ARCH_32_BIT\=arm export NATIVE_ARCH_64_BIT\=arm64 export NATIVE_ARCH_ACTUAL\=arm64 export PLATFORM_PREFERRED_ARCH\=x86_64 export VALID_ARCHS\=arm64\ arm64e\ i386\ x86_64 export XCRC_PLATFORM_PREFERRED_ARCH\=x86_64 export arch\=undefined_arch export CURRENT_ARCH\=undefined_arch
Whereas a regular build is:
export ARCHS\=arm64 export ARCHS_STANDARD\=arm64\ x86_64 export ARCHS_STANDARD_32_64_BIT\=arm64\ i386\ x86_64 export ARCHS_STANDARD_32_BIT\=i386 export ARCHS_STANDARD_64_BIT\=arm64\ x86_64 export ARCHS_STANDARD_INCLUDING_64_BIT\=arm64\ x86_64 export ARCHS_UNIVERSAL_IPHONE_OS\=arm64\ i386\ x86_64 export NATIVE_ARCH\=arm64 export NATIVE_ARCH_32_BIT\=arm export NATIVE_ARCH_64_BIT\=arm64 export NATIVE_ARCH_ACTUAL\=arm64 export PLATFORM_PREFERRED_ARCH\=x86_64 export VALID_ARCHS\=arm64\ arm64e\ i386\ x86_64 export XCRC_PLATFORM_PREFERRED_ARCH\=arm64 export arch\=undefined_arch export CURRENT_ARCH\=undefined_arch
XCRC_PLATFORM_PREFERRED_ARCH
is set as "XCRC_PLATFORM_PREFERRED_ARCH": "$(LINK_FILE_LIST_$(CURRENT_VARIANT)_$(PLATFORM_PREFERRED_ARCH):dir:standardizepath:file:default=arm64)",
.
But for Indexing builds $(LINK_FILE_LIST_$(CURRENT_VARIANT)_$(PLATFORM_PREFERRED_ARCH)
resolves as LINK_FILE_LIST_normal_x86_64
while for regular builds it resolves to LINK_FILE_LIST_normal_arm64
.
Strange thing is that PLATFORM_PREFERRED_ARCH
is x86_64
in both cases at the times I can print it.
I'm investigating why and how to change that.
Could indexing builds be somehow breaking regular builds? If that's not the case, it might be all the issues I saw on logs so far are just noise (not the actual problem that breaks the build). I am going to also run a few tests with Indexing disabled.
After disabling indexing I was able to build sucessfully the commit I was at consistently (even after regenerating the project and calling xcprepare
) as expected.
Then I pulled from master, regenerated the project, called xcprepare
, opened Xcode and hit build (without cleaning up previous build files).
Like before, the build failed with a module (SearchResultsList
) not being able to load underlying module for "ApplicationServices"
. This error is shown for several files in SearchResultsList
always pointing to import ApplicationServices
line. No additional context is available.
In the Xcode build log I see ApplicationServices
built successfully with [RC] Cached build for ApplicationServices target
message.
RC logs show (SearchResultsList) Prebuild step failed with error: missingFile
for a dependency which failed with Local fingerprint Fingerprint(raw: "83a05da6423b11af6257225b1c429a87", contextSpecific: "7c6acec80ba61a984ba01d9784cca64f") does not match with remote one ce81524e4068217e579b50efdf586364.
.
The dependency with local fingerprint mismatch error had the expected values for defaultEnvFingerprintKeys
on the consumer side.
In this case, I did not expect 100% cache hit rate as not all commits I fetched had been built by CI in producer mode, but I also don't expect a build failure as outcome.
Since ApplicationServices
built fine and the issue with SearchResultsList
is with a different dependency that cached missed, I don't know why Xcode shows the ApplicationServices
could not be found error.
Also I am not sure if the local fingerprint mismatch error is the error I should expect to see in RC logs. The dependency in question was changed in one of those commits CI didn't pick up yet.. but from your (@polac24's) message should that error only show up when producer had different defaultEnvFingerprintKeys
than consumer? (provided same RC binaries are used).
PS: Haven't found how to fix Indexing yet.
So to confirm, do you mean RC can generate the same file name for files with different content (e.g. because different commit)?
No, artifact filename is unique for the content so all target builds with the same filename are equivalent. Of course, assuming we don't include some input files or ENVs in the fingerprint hash. My initial concern was regarding the very short period of time when PUT overwrites a file. I read Amazon's docs again and they only mention that some content will be always served.
should that error only show up when producer had different (...)
No, there is something off with ApplicationServices
, because XCRemoteCache uses remotely available artifacts with best-effort. If nothing is available yet, a locally built product is fully compatible with the rest of the XCRemoteCache flow. Sidenote: One of our requirements was to be resilient in case of disconnected VPN. If a connection comes back, XCRemoteCache should continue using remote artifacts, even some of its dependencies were compiled locally.
I think we should focus on ApplicationServices
error. Seems that its product (.a, .swiftmodule etc.) are not correctly constructed from the artifact.
.zip
. The simples way is to trigger an error and inspect the unzipped content, e.g. in ..../DerivedData/Project/Build/Intermediates.noindex/Project.build/Debug-iphonesimulator/ApplicationServices.build/xccache/active
. Can you see there .swiftmodule
and include
dirs? What files are available there? I would also review meta's dependencies
field in some_hash.json
. Does the list makes sense - as the name suggests, the content of these files are used to generate a fingerprint that identifies an artifact.Is it an ObjC, Swift or mixed target?
Mixed.
If .modulemap is used, how is it created? Is it auto-generated by Xcode or commited-in to the repo?
The .modulemap
is auto-generated when buck
generates the Xcode project.
It looks like this:
module ApplicationServices { umbrella header "ApplicationServices.h" export * module * { export * } } module ApplicationServices.Swift { header "ApplicationServices-Swift.h" requires objc }
And the objc.modulemap
file looks like:
module ApplicationServices { umbrella header "ApplicationServices.h" export * module * { export * } } module ApplicationServices.__Swift { exclude header "ApplicationServices-Swift.h" }
(Extra) I would inspect the content of the artifact .zip.
I fetched the meta and file zip from a previous log where the error happened. In the zip file I find:
And the meta.json
field makes sense:
{"inputs":[],"generationCommit":"1e3df9044505d06a3a859e45428831af75423c88","pluginsKeys":{},"targetName":"ApplicationServices","platform":"iphonesimulator","configuration":"Debug","rawFingerprint":"01aa667906554640e3790c60358b33d1","fileKey":"4c6091b5e2e6151cc1dc85e89f1e7038","dependencies":["$(SRCROOT)\/ApplicationServices\/App Updates\/BCAppGetUpdateService.h","$(SRCROOT)\/ApplicationServices\/App Updates\/BCAppGetUpdateService.m","$(SRCROOT)\/ApplicationServices\/App Updates\/BCGetAppUpdateHandlers.h","$(SRCROOT)\/ApplicationServices\/App Updates\/BCGetAppUpdateHandlers.m","$(SRCROOT)\/ApplicationServices\/BCCreditCardConfigurator.h","$(SRCROOT)\/ApplicationServices\/BCCreditCardConfigurator.m","$(SRCROOT)\/ApplicationServices\/BCDataModelsInitializing.h","$(SRCROOT)\/ApplicationServices\/BCDataModelsInitializing.m","$(SRCROOT)\/ApplicationServices\/BCNetworkSecurityCheckService.h","$(SRCROOT)\/ApplicationServices\/BCNetworkSecurityCheckService.m","$(SRCROOT)\/ApplicationServices\/BCPropertyDataProviderImplementation.h","$(SRCROOT)\/ApplicationServices\/BCPropertyDataProviderImplementation.m","$(SRCROOT)\/ApplicationServices\/BCWechatController.h","$(SRCROOT)\/ApplicationServices\/BCWechatController.m","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/BCBugReport.h","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/BCBugReport.m","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/BCBugReportAttachment.h","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/BCBugReportAttachment.m","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/BCBugReporter.h","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/BCBugReporter.m","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/BCScreenRecorder.h","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/BCScreenRecorder.m","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/BCScreenshotTool.h","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/BCScreenshotTool.m","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/Networking\/BCBugSubmitter.h","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/Networking\/BCBugSubmitter.m","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/Networking\/BCJSONSubmittedBug.h","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/Networking\/BCJSONSubmittedBug.m","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/UI\/BCBugReporterViewController.h","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/UI\/BCBugReporterViewController.m","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/UI\/BugshotAnnotation\/JBSKAnnotationArrowView.h","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/UI\/BugshotAnnotation\/JBSKAnnotationArrowView.m","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/UI\/BugshotAnnotation\/JBSKAnnotationBlurView.h","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/UI\/BugshotAnnotation\/JBSKAnnotationBlurView.m","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/UI\/BugshotAnnotation\/JBSKAnnotationBoxView.h","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/UI\/BugshotAnnotation\/JBSKAnnotationBoxView.m","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/UI\/BugshotAnnotation\/JBSKAnnotationView.h","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/UI\/BugshotAnnotation\/JBSKAnnotationView.m","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/UI\/BugshotAnnotation\/JBSKCheckerboardView.h","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/UI\/BugshotAnnotation\/JBSKCheckerboardView.m","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/UI\/BugshotAnnotation\/JBSKScreenshotViewController.h","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/UI\/BugshotAnnotation\/JBSKScreenshotViewController.m","$(SRCROOT)\/ApplicationServices\/Core\/DebugViewControllerFactory.h","$(SRCROOT)\/ApplicationServices\/Core\/DebugViewControllerFactory.m","$(SRCROOT)\/ApplicationServices\/Core\/Experimentation\/BCMainApplicationContext.h","$(SRCROOT)\/ApplicationServices\/Core\/Experimentation\/BCMainApplicationContext.m","$(SRCROOT)\/ApplicationServices\/Core\/Experimentation\/BCMainExperimentLoader.h","$(SRCROOT)\/ApplicationServices\/Core\/Experimentation\/BCMainExperimentLoader.m","$(SRCROOT)\/ApplicationServices\/Core\/Experimentation\/ETExperimentToolConnector.h","$(SRCROOT)\/ApplicationServices\/Core\/Experimentation\/ETExperimentToolConnector.m","$(SRCROOT)\/ApplicationServices\/Core\/Experimentation\/ExperimentationObserver.h","$(SRCROOT)\/ApplicationServices\/Core\/Experimentation\/ExperimentationObserver.m","$(SRCROOT)\/ApplicationServices\/Core\/Localization\/BCLocalizationInitializer.h","$(SRCROOT)\/ApplicationServices\/Core\/Localization\/BCLocalizationInitializer.m","$(SRCROOT)\/ApplicationServices\/Core\/Localization\/BCLocalizeNumberFormattingHelper.h","$(SRCROOT)\/ApplicationServices\/Core\/Localization\/BCLocalizeNumberFormattingHelper.m","$(SRCROOT)\/ApplicationServices\/Core\/Localization\/I18N\/BCI18NFactory.h","$(SRCROOT)\/ApplicationServices\/Core\/Localization\/I18N\/BCI18NFactory.m","$(SRCROOT)\/ApplicationServices\/Core\/Localization\/I18N\/BKCTranslationABTesting.h","$(SRCROOT)\/ApplicationServices\/Core\/Localization\/I18N\/BKCTranslationABTesting.m","$(SRCROOT)\/ApplicationServices\/Core\/Service Provider\/BCServiceProvider+CommonServices.h","$(SRCROOT)\/ApplicationServices\/Core\/Service Provider\/BCServiceProvider+CommonServices.m","$(SRCROOT)\/ApplicationServices\/Core\/Service Provider\/BCServiceProvider+CommonServices_Private.h","$(SRCROOT)\/ApplicationServices\/Core\/Service Provider\/BCServiceProviderInjectable.h","$(SRCROOT)\/ApplicationServices\/Core\/Support\/BCApplicationDevelopmentServices.h","$(SRCROOT)\/ApplicationServices\/Core\/Support\/BCApplicationDevelopmentServices.m","$(SRCROOT)\/ApplicationServices\/Core\/Support\/BCRemoveUnusedDataService.h","$(SRCROOT)\/ApplicationServices\/Core\/Support\/BCRemoveUnusedDataService.m","$(SRCROOT)\/ApplicationServices\/Core\/Support\/BCSessionTimeMeasurer.h","$(SRCROOT)\/ApplicationServices\/Core\/Support\/BCSessionTimeMeasurer.m","$(SRCROOT)\/ApplicationServices\/Core\/Support\/BCSettings.h","$(SRCROOT)\/ApplicationServices\/Core\/Support\/BCSettings.m","$(SRCROOT)\/ApplicationServices\/Core\/Support\/CrashReporter\/BCCrashlyticsLogger.h","$(SRCROOT)\/ApplicationServices\/Core\/Support\/CrashReporter\/BCCrashlyticsLogger.m","$(SRCROOT)\/ApplicationServices\/Core\/Support\/Log\/BCDebugLogConfiguration.h","$(SRCROOT)\/ApplicationServices\/Core\/Support\/Log\/BCDebugLogConfiguration.m","$(SRCROOT)\/ApplicationServices\/Core\/Support\/Log\/BCDebugLogger.h","$(SRCROOT)\/ApplicationServices\/Core\/Support\/Log\/BCDebugLogger.m","$(SRCROOT)\/ApplicationServices\/Core\/Support\/Log\/BCDebugLoggingService.h","$(SRCROOT)\/ApplicationServices\/Core\/Support\/Log\/BCDebugLoggingService.m","$(SRCROOT)\/ApplicationServices\/Core\/Support\/Log\/BCLoggingService.h","$(SRCROOT)\/ApplicationServices\/Core\/Support\/Log\/BCLoggingService.m","$(SRCROOT)\/ApplicationServices\/Core\/Support\/Log\/BCMutableDebugLogConfiguration.h","$(SRCROOT)\/ApplicationServices\/Core\/Support\/Log\/BCMutableDebugLogConfiguration.m","$(SRCROOT)\/ApplicationServices\/Core\/Support\/Log\/BCTTYDebugLogger.h","$(SRCROOT)\/ApplicationServices\/Core\/Support\/Log\/BCTTYDebugLogger.m","$(SRCROOT)\/ApplicationServices\/Core\/Support\/Log\/BCUITestFileLogger.h","$(SRCROOT)\/ApplicationServices\/Core\/Support\/Log\/BCUITestFileLogger.m","$(SRCROOT)\/ApplicationServices\/Core\/Support\/Screenshotter.h","$(SRCROOT)\/ApplicationServices\/Core\/Support\/Screenshotter.m","$(SRCROOT)\/ApplicationServices\/Core\/User\/Authentication Providers\/BCAuthenticationProvider.h","$(SRCROOT)\/ApplicationServices\/Core\/User\/Authentication Providers\/BCAuthenticationProvider.m","$(SRCROOT)\/ApplicationServices\/Core\/User\/Authentication Providers\/BCAuthenticationProviderRegistry.h","$(SRCROOT)\/ApplicationServices\/Core\/User\/Authentication Providers\/BCAuthenticationProviderRegistry.m","$(SRCROOT)\/ApplicationServices\/Core\/User\/Payments Provider\/BCPaymentProvider.h","$(SRCROOT)\/ApplicationServices\/Core\/User\/Payments Provider\/BCPaymentProviderRegistry.h","$(SRCROOT)\/ApplicationServices\/Core\/User\/Payments Provider\/BCPaymentProviderRegistry.m","$(SRCROOT)\/ApplicationServices\/Core\/User\/Payments Provider\/BCWechatPaymentProvider.h","$(SRCROOT)\/ApplicationServices\/Core\/User\/Payments Provider\/BCWechatPaymentProvider.m","$(SRCROOT)\/ApplicationServices\/CountryCode+Flags.h","$(SRCROOT)\/ApplicationServices\/CountryCode+Flags.m","$(SRCROOT)\/ApplicationServices\/Fabric\/BCFabricService.h","$(SRCROOT)\/ApplicationServices\/Fabric\/BCFabricService.m","$(SRCROOT)\/ApplicationServices\/Fabric\/BCFirebaseSessionExperiments.h","$(SRCROOT)\/ApplicationServices\/Fabric\/BCFirebaseSessionExperiments.m","$(SRCROOT)\/ApplicationServices\/Network\/Requests\/UpdateDeviceInfoRequestConfig+Defaults.swift","$(SRCROOT)\/ApplicationServices\/Network\/Webcalls\/BCMobileBackendServiceContext.h","$(SRCROOT)\/ApplicationServices\/Network\/Webcalls\/BCMobileBackendServiceContext.m","$(SRCROOT)\/ApplicationServices\/PayInSDK\/PayInSDKBuilder.swift","$(SRCROOT)\/ApplicationServices\/PayInSDK\/Private\/PayInABTestingImp.swift","$(SRCROOT)\/ApplicationServices\/PayInSDK\/Private\/PayInLocalizer.swift","$(SRCROOT)\/ApplicationServices\/PayInSDK\/Private\/PayInfoLoader.swift","$(SRCROOT)\/ApplicationServices\/PayInSDK\/Private\/PayInfoPersistenceProviderImpl.swift","$(SRCROOT)\/ApplicationServices\/PaymentComponent\/PaymentComponentBuilder.h","$(SRCROOT)\/ApplicationServices\/PaymentComponent\/PaymentComponentBuilder.m","$(SRCROOT)\/ApplicationServices\/Phone\/BCPhoneConfirmationService.h","$(SRCROOT)\/ApplicationServices\/Phone\/BCPhoneConfirmationService.m","$(SRCROOT)\/ApplicationServices\/Phone\/BCPhoneService.h","$(SRCROOT)\/ApplicationServices\/Phone\/BCPhoneService.m","$(SRCROOT)\/ApplicationServices\/Phone\/BKPJSONStatusWithError.h","$(SRCROOT)\/ApplicationServices\/Phone\/BKPJSONStatusWithError.m","$(SRCROOT)\/ApplicationServices\/Routing\/BCServiceProvider+Routing.h","$(SRCROOT)\/ApplicationServices\/Routing\/BCServiceProvider+Routing.m","$(SRCROOT)\/ApplicationServices\/Routing\/BCUIOpenerProxy.h","$(SRCROOT)\/ApplicationServices\/Routing\/BCUIOpenerProxy.m","$(SRCROOT)\/ApplicationServices\/Routing\/BCUIOpenerSource.h","$(SRCROOT)\/ApplicationServices\/Routing\/BCUIOpenerSource.m","$(SRCROOT)\/ApplicationServices\/Routing\/ServiceProvider+Routing.swift","$(SRCROOT)\/ApplicationServices\/Routing\/UIBaseOpener.h","$(SRCROOT)\/ApplicationServices\/Routing\/UIBaseOpener.m","$(SRCROOT)\/ApplicationServices\/Support\/Assertion\/BCAssertionService.h","$(SRCROOT)\/ApplicationServices\/Support\/Assertion\/BCAssertionService.m","$(SRCROOT)\/ApplicationServices\/Support\/Assertion\/Listeners\/BCAlertAssertionListener.h","$(SRCROOT)\/ApplicationServices\/Support\/Assertion\/Listeners\/BCAlertAssertionListener.m","$(SRCROOT)\/ApplicationServices\/Support\/Assertion\/Listeners\/BCBCGoalAssertListener.h","$(SRCROOT)\/ApplicationServices\/Support\/Assertion\/Listeners\/BCBCGoalAssertListener.m","$(SRCROOT)\/ApplicationServices\/Support\/Assertion\/Listeners\/BCFabricAssertionListener.h","$(SRCROOT)\/ApplicationServices\/Support\/Assertion\/Listeners\/BCFabricAssertionListener.m","$(SRCROOT)\/DeallocDebugging\/UIViewController+DeallocDebugging.h","$(SRCROOT)\/DeallocDebugging\/UIViewController+DeallocDebugging.m","$(PROJECT_ROOT)\/.com\/Modules\/CoreModules\/buck-out\/gen\/_p\/6tEQmabZZE-pub\/StartupExperiments\/module.modulemap","$(PROJECT_ROOT)\/.com\/Modules\/CoreModules\/buck-out\/gen\/_p\/9aIHa5GMKK-pub\/Core\/module.modulemap","$(PROJECT_ROOT)\/.com\/Modules\/CoreModules\/buck-out\/gen\/_p\/9aIHa5GMKK-pub\/Core\/objc.modulemap","$(PROJECT_ROOT)\/.com\/Modules\/CoreModules\/buck-out\/gen\/_p\/A8iAWHPFeI-pub\/Localization\/module.modulemap","$(PROJECT_ROOT)\/.com\/Modules\/CoreModules\/buck-out\/gen\/_p\/Gze0c3Q_xF-pub\/Analytics\/module.modulemap","$(PROJECT_ROOT)\/.com\/Modules\/CoreModules\/buck-out\/gen\/_p\/Gze0c3Q_xF-pub\/Analytics\/objc.modulemap","$(PROJECT_ROOT)\/.com\/Modules\/CoreModules\/buck-out\/gen\/_p\/cF8V00mifi-pub\/SwiftUICore\/module.modulemap","$(PROJECT_ROOT)\/.com\/Modules\/CoreModules\/buck-out\/gen\/_p\/cg7NxCVKml-pub\/Networking\/module.modulemap","$(PROJECT_ROOT)\/.com\/Modules\/CoreModules\/buck-out\/gen\/_p\/cg7NxCVKml-pub\/Networking\/objc.modulemap","$(PROJECT_ROOT)\/.com\/Modules\/CoreModules\/buck-out\/gen\/_p\/gBnVtj5naS-pub\/Persistence\/module.modulemap","$(PROJECT_ROOT)\/.com\/Modules\/CoreModules\/buck-out\/gen\/_p\/tR9MA1CY0w-pub\/DependencyInjection\/module.modulemap","$(PROJECT_ROOT)\/.com\/Modules\/Platform\/Bdux\/buck-out\/gen\/_p\/CyhVfx8_xO-pub\/Bdux\/module.modulemap","$(PROJECT_ROOT)\/Pods\/AppsFlyerFramework\/AppsFlyerLib.xcframework\/ios-arm64_i386_x86_64-simulator\/AppsFlyerLib.framework\/Headers\/AppsFlyerCrossPromotionHelper.h","$(PROJECT_ROOT)\/Pods\/AppsFlyerFramework\/AppsFlyerLib.xcframework\/ios-arm64_i386_x86_64-simulator\/AppsFlyerLib.framework\/Headers\/AppsFlyerDeepLink.h","$(PROJECT_ROOT)\/Pods\/AppsFlyerFramework\/AppsFlyerLib.xcframework\/ios-arm64_i386_x86_64-simulator\/AppsFlyerLib.framework\/Headers\/AppsFlyerDeepLinkResult.h","$(PROJECT_ROOT)\/Pods\/AppsFlyerFramework\/AppsFlyerLib.xcframework\/ios-arm64_i386_x86_64-simulator\/AppsFlyerLib.framework\/Headers\/AppsFlyerLib.h","$(PROJECT_ROOT)\/Pods\/AppsFlyerFramework\/AppsFlyerLib.xcframework\/ios-arm64_i386_x86_64-simulator\/AppsFlyerLib.framework\/Headers\/AppsFlyerLinkGenerator.h","$(PROJECT_ROOT)\/Pods\/AppsFlyerFramework\/AppsFlyerLib.xcframework\/ios-arm64_i386_x86_64-simulator\/AppsFlyerLib.framework\/Headers\/AppsFlyerShareInviteHelper.h","$(PROJECT_ROOT)\/Pods\/AppsFlyerFramework\/AppsFlyerLib.xcframework\/ios-arm64_i386_x86_64-simulator\/AppsFlyerLib.framework\/Modules\/module.modulemap","$(PROJECT_ROOT)\/Pods\/FirebaseAnalytics\/Frameworks\/FirebaseAnalytics.xcframework\/ios-arm64_i386_x86_64-simulator\/FirebaseAnalytics.framework\/Modules\/module.modulemap","$(PROJECT_ROOT)\/Pods\/GoogleAnalytics\/Frameworks\/GoogleAnalytics.xcframework\/ios-arm64_i386_x86_64-simulator\/GoogleAnalytics.framework\/Headers\/GAI.h","$(PROJECT_ROOT)\/Pods\/GoogleAnalytics\/Frameworks\/GoogleAnalytics.xcframework\/ios-arm64_i386_x86_64-simulator\/GoogleAnalytics.framework\/Headers\/GAIDictionaryBuilder.h","$(PROJECT_ROOT)\/Pods\/GoogleAnalytics\/Frameworks\/GoogleAnalytics.xcframework\/ios-arm64_i386_x86_64-simulator\/GoogleAnalytics.framework\/Headers\/GAIEcommerceFields.h","$(PROJECT_ROOT)\/Pods\/GoogleAnalytics\/Frameworks\/GoogleAnalytics.xcframework\/ios-arm64_i386_x86_64-simulator\/GoogleAnalytics.framework\/Headers\/GAIEcommerceProduct.h","$(PROJECT_ROOT)\/Pods\/GoogleAnalytics\/Frameworks\/GoogleAnalytics.xcframework\/ios-arm64_i386_x86_64-simulator\/GoogleAnalytics.framework\/Headers\/GAIEcommerceProductAction.h","$(PROJECT_ROOT)\/Pods\/GoogleAnalytics\/Frameworks\/GoogleAnalytics.xcframework\/ios-arm64_i386_x86_64-simulator\/GoogleAnalytics.framework\/Headers\/GAIEcommercePromotion.h","$(PROJECT_ROOT)\/Pods\/GoogleAnalytics\/Frameworks\/GoogleAnalytics.xcframework\/ios-arm64_i386_x86_64-simulator\/GoogleAnalytics.framework\/Headers\/GAIFields.h","$(PROJECT_ROOT)\/Pods\/GoogleAnalytics\/Frameworks\/GoogleAnalytics.xcframework\/ios-arm64_i386_x86_64-simulator\/GoogleAnalytics.framework\/Headers\/GAILogger.h","$(PROJECT_ROOT)\/Pods\/GoogleAnalytics\/Frameworks\/GoogleAnalytics.xcframework\/ios-arm64_i386_x86_64-simulator\/GoogleAnalytics.framework\/Headers\/GAITrackedViewController.h","$(PROJECT_ROOT)\/Pods\/GoogleAnalytics\/Frameworks\/GoogleAnalytics.xcframework\/ios-arm64_i386_x86_64-simulator\/GoogleAnalytics.framework\/Headers\/GAITracker.h","$(PROJECT_ROOT)\/Pods\/GoogleAnalytics\/Frameworks\/GoogleAnalytics.xcframework\/ios-arm64_i386_x86_64-simulator\/GoogleAnalytics.framework\/Headers\/GoogleAnalytics-umbrella.h","$(PROJECT_ROOT)\/Pods\/GoogleAnalytics\/Frameworks\/GoogleAnalytics.xcframework\/ios-arm64_i386_x86_64-simulator\/GoogleAnalytics.framework\/Modules\/module.modulemap","$(PROJECT_ROOT)\/Pods\/OTPublishersHeadlessSDK\/OTPublishersHeadlessSDK.xcframework\/ios-arm64_x86_64-simulator\/OTPublishersHeadlessSDK.framework\/Headers\/OTPublishersHeadlessSDK-Swift.h","$(PROJECT_ROOT)\/Pods\/OTPublishersHeadlessSDK\/OTPublishersHeadlessSDK.xcframework\/ios-arm64_x86_64-simulator\/OTPublishersHeadlessSDK.framework\/Headers\/OTPublishersHeadlessSDK.h","$(PROJECT_ROOT)\/Pods\/OTPublishersHeadlessSDK\/OTPublishersHeadlessSDK.xcframework\/ios-arm64_x86_64-simulator\/OTPublishersHeadlessSDK.framework\/Headers\/TCF2Encoder.h","$(PROJECT_ROOT)\/Pods\/OTPublishersHeadlessSDK\/OTPublishersHeadlessSDK.xcframework\/ios-arm64_x86_64-simulator\/OTPublishersHeadlessSDK.framework\/Modules\/OTPublishersHeadlessSDK.swiftmodule\/arm64-apple-ios-simulator.swiftinterface","$(PROJECT_ROOT)\/Pods\/OTPublishersHeadlessSDK\/OTPublishersHeadlessSDK.xcframework\/ios-arm64_x86_64-simulator\/OTPublishersHeadlessSDK.framework\/Modules\/module.modulemap","$(PROJECT_ROOT)\/Pods\/buck-out\/gen\/_p\/0dRqK4SfsV-pub\/Promises\/module.modulemap","$(PROJECT_ROOT)\/Pods\/buck-out\/gen\/_p\/15TFc8Szm6-pub\/FLEX\/module.modulemap","$(PROJECT_ROOT)\/Pods\/buck-out\/gen\/_p\/2fbdIKI0gS-pub\/FBSDKCoreKit\/objc.modulemap","$(PROJECT_ROOT)\/Pods\/buck-out\/gen\/_p\/CQw1YVPsxj-pub\/Identity\/module.modulemap","$(PROJECT_ROOT)\/Pods\/buck-out\/gen\/_p\/OZ9vhJNhFI-pub\/Valet\/module.modulemap","$(PROJECT_ROOT)\/Pods\/buck-out\/gen\/_p\/SGT46c7cUG-pub\/GoogleUtilities\/module.modulemap","$(PROJECT_ROOT)\/Pods\/buck-out\/gen\/_p\/lmGQiP_3Pb-pub\/UI\/module.modulemap","$(PROJECT_ROOT)\/Pods\/buck-out\/gen\/_p\/lmGQiP_3Pb-pub\/UI\/objc.modulemap","$(PROJECT_ROOT)\/Pods\/buck-out\/gen\/_p\/vB7q01z9MO-pub\/TSMessages\/module.modulemap","$(PROJECT_ROOT)\/Pods\/buck-out\/gen\/_p\/y7WPAZHCFl-pub\/AFNetworking\/module.modulemap","$(PROJECT_ROOT)\/Submodules\/buck-out\/gen\/_p\/EMMODyN0oP-pub\/PerformanceSuite\/module.modulemap","$(PROJECT_ROOT)\/Submodules\/buck-out\/gen\/_p\/FjZPYUHybM-pub\/PrivacyUI\/module.modulemap","$(PROJECT_ROOT)\/Submodules\/buck-out\/gen\/_p\/XrDcAievMX-pub\/CocoaHTTPServer2\/module.modulemap","$(PROJECT_ROOT)\/Submodules\/buck-out\/gen\/_p\/cZAo5JftXV-pub\/Privacy\/module.modulemap","$(PROJECT_ROOT)\/Submodules\/buck-out\/gen\/_p\/nek6SbMbef-pub\/ABTesting\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/0OKxluvWCI-pub\/LocationServices\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/1dvFki_alS-pub\/PerformanceTrackingServices\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/1dvFki_alS-pub\/PerformanceTrackingServices\/objc.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/2dzer-8x1e-pub\/Pricing\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/2vVGOTBOsy-pub\/MarketplaceAppsUI_Deprecated\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/3cAxWrNmBX-pub\/BKX\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/5x3KPSaE1B-pub\/Messaging\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/7uZ55b6nBZ-pub\/PoliciesDataModel\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/7uZ55b6nBZ-pub\/PoliciesDataModel\/objc.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/8o-TgZBMsX-pub\/BasicUI\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/C_i4L1yqQR-pub\/PostDataModel\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/Eyfl32kr0X-pub\/AppStartsServices\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/FI4P-wg1FM-pub\/LegacyUI\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/FNCOEm0EEp-pub\/SearchDestinationServices\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCAlertAssertionListener.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCAppGetUpdateService.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCApplicationDevelopmentServices.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCApplicationLaunchMethod.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCAuthenticationProvider.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCAuthenticationProviderRegistry.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCBCGoalAssertListener.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCAssertionService.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCI18NFactory.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCBugReport.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCBugReportAttachment.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCBugReporter.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCBugReporterViewController.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCBugSubmitter.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCCrashlyticsLogger.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCCreditCardConfigurator.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCDataModelsInitializing.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCDebugLogConfiguration.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCDebugLogger.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCDebugLoggingService.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCFabricAssertionListener.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCFabricService.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCFirebaseSessionExperiments.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCGetAppUpdateHandlers.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCJSONSubmittedBug.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCLocalizationInitializer.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCLocalizeNumberFormattingHelper.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCLoggingService.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCMainApplicationContext.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCMainExperimentLoader.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCMobileBackendServiceContext.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCMutableDebugLogConfiguration.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCNetworkSecurityCheckService.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCPaymentProvider.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCPaymentProviderRegistry.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCPhoneConfirmationService.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCPhoneService.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCPropertyDataProviderImplementation.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCRemoveUnusedDataService.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCScreenRecorder.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCScreenshotTool.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCServiceProvider+CommonServices.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCServiceProvider+CommonServices_Private.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCServiceProvider+Routing.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCServiceProviderInjectable.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCSessionTimeMeasurer.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCSettings.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCTTYDebugLogger.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCUIOpenerProxy.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCUIOpenerSource.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCUITestFileLogger.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCWechatController.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCWechatPaymentProvider.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BKCTranslationABTesting.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BKPJSONStatusWithError.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/ApplicationServices.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/CountryCode+Flags.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/DebugViewControllerFactory.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/ETExperimentToolConnector.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/ExperimentationObserver.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/JBSKAnnotationArrowView.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/JBSKAnnotationBlurView.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/JBSKAnnotationBoxView.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/JBSKAnnotationView.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/JBSKCheckerboardView.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/JBSKScreenshotViewController.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/PaymentComponentBuilder.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/Screenshotter.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/UIBaseOpener.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/UIViewController+DeallocDebugging.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/objc.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/KKK0l284C6-pub\/PropertyResponses\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/PxhODn6zhP-pub\/C360TrackerService\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/Y9ZCi6HS2l-pub\/FilteringServices\/objc.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/_BVlEto_GS-pub\/BEDrivenComponents\/objc.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/eXDT_yFxGS-pub\/DataModels\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/eXDT_yFxGS-pub\/DataModels\/objc.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/fba5cLfzYh-pub\/EMKUI\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/i1btfxAVkF-pub\/NotificationCenterService\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/j2Aczs_jbP-pub\/WishlistUIExperiments\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/kYucPv7jLf-pub\/BCNotifications\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/nWampcnrgg-pub\/PaymentDependencies\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/opjNULo7Dc-pub\/GeneratedResources\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/s-r4uCF9vZ-pub\/Utils\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/s-r4uCF9vZ-pub\/Utils\/objc.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/vzjHohEuGE-pub\/SegmentsServices\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/vzjHohEuGE-pub\/SegmentsServices\/objc.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/wdutdtfu5K-pub\/DirectionsSheet\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/yhW2CPbkdY-pub\/TransportDiscovery\/module.modulemap","$(BUILD_DIR)\/Debug-iphonesimulator\/Apollo.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/ApolloAPI.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/ApolloCombine.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/ApolloUtils.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/BUIAssets.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/BUIFoundations.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/Blitzen.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/Analytics.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/Core.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/Networking.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/Privacy.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/StartupExperiments.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/UI.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/DMLClient.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/DSAssets.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/DataModels.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/DependencyInjection.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/FilteringServices.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/GeneratedResources.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/NetworkServices.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/PayInSDK.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/PoliciesDataModel.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/PostDataModel.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/Pricing.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/Promises.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/SwiftBUI.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5"],"xcode":"13C100"}
The modulemap format you posted suggests that ApplicationServices
is a framework built using Xcode. Otherwise the zip artifact wouldn't be created.
In general, when a framework is created in Xcode, Xcode creates some intermediate module maps and passes their paths to both compilers using virtualFileSystem (-ivfsoverlay
). These modulemaps, unextended-module.modulemap
and module.modulemap
, have very similar format to your .modulemap
and objc.modulemap
. I didn't get why buck generates them. Does Xcode consume these files? If so, which build setting is used for that, MODULEMAP_FILE
? Is the target compiled partially in buck and partially in Xcode?
When a build is failing with a missing module, can you copy the entire directory ApplicationServices.framework
in the DerviedData's Product dir and compare its content after introducing a dummy change in the ApplicationServices
target (e.g. a new line)? Obviously a binary file will differ but module.modulemap
, PPP-Swift.h
(and maybe other .h files in Headers
dir) should be identical. *.swiftmodule
files structure should be the same, except .swiftsourceinfo
.
Sample .framework content:
ApplicationServices
is not built as a .framework
but as a .a
library. In DerivedData's Product dir, ApplicationServices
is found as ApplicationServices.swiftmodule
and ApplicationServices.a
.
Anyways, the ApplicationServices
error now seems to be resolved. Haven't seen it anymore. The SearchResultsList
module was not declaring ApplicationServices
as a dependency and was imported under a canImport
condition. Removing that and declaring the dependency seems to have fixed the build failure.
With that out of the way and Indexing disabled, there is less noise.
Now, I can confirm that checking out to a specific commit CI already produced does not consistently result in 100% cache hit rate. This is what I am experiencing:
e717bd30
. Verified CI job finished successfully and that it was the last RC CI job run. Deleted everything and regenerated the project (automatically triggering xcprepare
). The build succeeded with 100% cache hit rate.b8768420
instead. This was the commit with the SearchResultsList
fix. The build succeeded with many cache misses. The reason were fingerprint mismatch errors
. I see another RC CI job was running at the time for 8263d68c
.8263d68c
instead. This time I got one cache miss for PrivacyUI
(and reverse dependencies). PrivacyUI
failed due to a fingerprint mismatch error
.4dde5913
. Again, I verified the CI job finished successfully for that commit and that it was the last RC CI job run. This time I did not delete anything locally. Just regenerated the project (automatically triggering xcprepare
). Again I got many cache misses due to modules with high number of reverse dependencies failing with a fingerprint mismatch error
.My expectation is that whenever I check out to a commit in master that was successfully produced by CI, as long as the commit is within the last 10 limit and I have no local changes, with or without a local DerivedData
and Caches
directory, I should always get 100% cache hit rate when building. Is that correct?
Unfortunately I can't call log show --predicate 'sender BEGINSWITH "xc"' --style compact --info --debug
due to lack of permissions on CI, so I don't have Producer logs for every commit. But I can tell all these commits built successfully (indicating no RC networking errors as those break producer builds).
When running the same CI commands in my local machine and capturing both Xcode and RC logs I did not find any defaultEnvFingerprintKeys
with different values. I will do some more testing mimic-ing the CI behaviour across commits to see if I can reproduce.
Any other ideas?
My expectation is that whenever I check out to a commit in master that was successfully produced by CI, as long as the commit is within the last 10 limit and I have no local changes, with or without a local DerivedData and Caches directory, I should always get 100% cache hit rate when building. Is that correct?
That is correct. Moreover, there is no 10 commits limit. It is enough that CI has ever generated artifacts for that commit. Of course, assuming you don't have any eviction policy on the HTTP server.
I recall such inconsistency when code generation is not deterministic. If you have some .swift, .m, .h (or anything else) sources generated on-fly, their char-level difference can introduce random cache miss every now and then.
To troubleshoot it faster, there is an extra fields rawFingerprint
in meta .json that is generated only based on all sources content. Contrary to fileKey
, it doesn't interleave any ENVs like ARCHS
, XCODE_VERSION
etc. so locally you can quicker say if a miss is caused by a source file.
Hey Bartosz,
We have found out one of issues that was causing fingerprint mismatch. Our main application repo was pointing to a commit that is not on master of one of our sub-repos Privacy
.
merging changes of Privacy
sub-repo to master and updating the main app repo to point to latest master commit of Privacy
sub-repo had fixed the mismatch issue and cache misses with PrivacyUI
module.
P.S. We are still investigating and validating other incidents of weird cache misses and fingerprint mismatch.
Although now, we are noting an issue related to build time for app in cases of cache misses happens.
For example, pulling latest commit on master of main application repo, and trying to build it, before the producer_ci_job completes for that commit, this is expected to cause cache misses for modules touched in that latest commit and all reverse dependencies. I can see this is normal, but the issue is with build time itself. it sometimes exceeds by far (1.5x to 2x) the regular app build time without XC Remote Cache support. This mainly happens in case building that latest commit in consumer mode without cleaning the DerivedData dir.
Steps to produce:
1) pull a recent commit on master that has producer job completed for.
2) build the app for that commit achieves 100% cache hits
3) pull latest commit on master that that producer job did not ran for it
4) re-build the app for that commit without cleaning DerivedData
nor XCRemoteCache
dirs
here this app build could take 1.5x to 2x to the regular app build time
5) cleaning DerivedData and rebuild again
here build time reduced to to around 0,66x the regular app build
Do we need to clean DerivedData
on before building on local machines as consumers?
I also just want to highlight that we are still seeing weird 404 errors on the local consumer machines logs, and did not verify that all issues related to cache misses are solved yet. We are still working on that, but trying to handle issues one by one.
Do we need to clean DerivedData on before building on local machines as consumers?
No, XCRemoteCache is optimized for incremental builds, including scenario after checking out to a different sha. 1.5x to 2x the regular app build time is off.
There is an extra optimization that could help achieving the incremental build with XCRemoteCache:
defaults write com.apple.dt.XCBuild IgnoreFileSystemDeviceInodeChanges -bool YES
That command modifies Xcode global config and shouldn't have any side effects - at least I haven't noticed any.
hey Bartosz,
Thanks for your updates on this. As status of today we got one step back, as we found out that finger print mismatch for PrivacyUI
is not solved as we thought yesterday. We can still see it randomly happens with different modules but it is very common to happen with PrivacyUI
.
I had changed my cache server to a local docker image instead of amazon S3, so that I can easily traverse and investigate the cache files produced from multiple producer jobs.
From investigation I can see that the same meta file with exact same name of PrivacyUI
module is being overwritten with different content, for every producer build executed.
Step to produce:
41e57bbf9c49f1cfb44ac930f374b57a23e263f6
on master repo. This commit points to the following commit hash 6f524499a878ad213c02f81eb36f5969996579ec
on Privacy
sub-repo that has PrivacyUI
module.I can see the following meta file generated 6f524499a878ad213c02f81eb36f5969996579ec-PrivacyUI-Debug-iphonesimulator-13C100-867e80b75d883611ed817cb2686c05de.json
for PrivacyUI
module with content `{
"inputs":[
], "generationCommit":"6f524499a878ad213c02f81eb36f5969996579ec", "pluginsKeys":{
}, "targetName":"PrivacyUI", "platform":"iphonesimulator", "configuration":"Debug", "rawFingerprint":"7229c23320e5a9cc2a66e9adefa8a5d5", "fileKey":"5ef3c61d6aa040a484ded6fcc0750c17", "dependencies":[......`
git clean -ffdx git submodule foreach git clean -ffdx rm -rf ~/Library/Caches/XCRemoteCache rm -rf ~/Library/Developer/Xcode/DerivedData
./buck project App -c project.rc-mode=producer
I can see the meta file 6f524499a878ad213c02f81eb36f5969996579ec-PrivacyUI-Debug-iphonesimulator-13C100-867e80b75d883611ed817cb2686c05de.json
for PrivacyUI
module had been overwritten with different rawFingerprint
and fileKey
as follow `{
"inputs":[
], "generationCommit":"6f524499a878ad213c02f81eb36f5969996579ec", "pluginsKeys":{
}, "targetName":"PrivacyUI", "platform":"iphonesimulator", "configuration":"Debug", "rawFingerprint":"e2d5a803cc5e82355bb1d6ab0b55e270", "fileKey":"eced69e44829c0b6eb24449e58fa000b", "dependencies":[......`
rawFingerprint
& fileKey
values (sometimes old values repeated).So, if rawFingerprint
is a hash for source files content, then this confirms a change happens between builds, although there is not code generated except the project files generated by buck, and modulemap files that are being generated by buck too. Working now to investigate what could be the potential reason for that source change or rawFingerprint
mismatch
Hi Bartosz,
I am noting that list of dependencies in the meta file of PrivacyUI
module, changes between different producer builds for the same commit, with even re-generating the project with buck. So I generated the project once with buck and ran multiple producer builds with just cleaning DerivedData dir in between
Attaching sample of these meta files generated with the same name and different content for the same module [PrivacyUI
]
31d760b57ddd53f99a4373139aa4d298b073fae7-PrivacyUI-Debug-iphonesimulator-13C100-867e80b75d883611ed817cb2686c05de.zip
so what changed between these builds is only clean DerivedData and no code generation happened
Are ‘BUIAssets’, ‘PayInSDK’ prebuilt or generated on-fly when xcodebuild happens by buck (or any other tool)? In other words, does ‘ \/Pods\/buck-out\/gen\/_p\/MRmmgKh-hU-pub\/BUIAssets\/objc.modulemap’ exist prior to the Xcode build (after project generation)?
/Pods/buck-out/gen/_p/MRmmgKh-hU-pub/BUIAssets/objc.modulemap
is generated as part of xcode project generation by buck. So at build time there is no code generated at all, neither forBUIAssets
nor for PayInSDK
. Both modules are NOT prebuilt and are compiled during xcode build. Also both of them achieve cache hit on consumer side.
On other side, is it expected that XCRemoteCache to include /Pods/buck-out/gen/_p/MRmmgKh-hU-pub/BUIAssets/objc.modulemap
as part of dependencies for PrivacyUI
? and given these module map files are already existing before build time, any clue why sometimes could XCRemoteCache include them and sometimes not?
I fixed the project generation state (generated project once, and only cleaning derived data in between), and I can still see different rawFingerprint each time I run the producer.
A list of dependencies is takes directly from clang/swift compiler .d files and assuming all these .modulemap files are present on a disk when PrivacyUI
target is compiled, their locations are passed into header search paths (or similar), compilers should always include them.
I would analyze isolated PrivacyUI
compilation and inspect why modulemaps are placed in any of DerivedData/Proj/Build/Intermediates.noindex/Proj.build/Debug-iphonesimulator/PrivacyUI.build/Objects-normal/arm64/*.d
:
PrivacyUI
import BUIAssets
? (Directly/indirectly)? /Pods/buck-out/gen/_p/MRmmgKh-hU-pub/BUIAssets/objc.modulemap
should be analyzed (e.g. -Xcc -fmodule-map-file=BUIAssets/obj.modulemap
? -fmodule-map-file=...BUIAssets/objc.modulemap
is used, does the generated project always contain that argument?On the other hand, if /Pods/buck-out/gen/_p/MRmmgKh-hU-pub/BUIAssets/objc.modulemap
always exists on a disk (and its content is byte-level identical), including/not including it should not cause any cache miss on a consumer side, right? I agree it is not optimal as consumers can download several artifact zips of the same library (thus finding a reason should be prioritized) but that would only affect performance, not cache hit rate.
Hi @polac24!
I will first address your questions and then add some more detail and questions from our side.
Addressing your questions
- Does PrivacyUI import BUIAssets? (Directly/indirectly)?
Yes, BUIAssets
is directly referenced as a dependency of PrivacyUI
in the BUCK
file.
- How the compiler knows that
/Pods/buck-out/gen/_p/MRmmgKh-hU-pub/BUIAssets/objc.modulemap
should be analyzed (e.g.-Xcc -fmodule-map-file=BUIAssets/obj.modulemap
)?
-fmodule-map-file
is not used. The compiler knows that /Pods/buck-out/gen/_p/MRmmgKh-hU-pub
should be analyzed via HEADER_SEARCH_PATHS
, SWIFT_INCLUDE_PATHS
and isystem
flags on OTHER_CFLAGS
, OTHER_CPLUSPLUSFLAGS
and OTHER_SWIFT_FLAGS
. In the test I ran, BUIAssets
wasn't directly referenced but BUIAssets
was the only directory under that path and the meta
file did not contain any references to it but $(BUILD_DIR)/Debug-iphonesimulator/BUIAssets.swiftmodule/arm64-apple-ios-simulator.swiftmodule.md5
.
On the other hand, if /Pods/buck-out/gen/_p/MRmmgKh-hU-pub/BUIAssets/objc.modulemap always exists on a disk (and its content is byte-level identical), including/not including it should not cause any cache miss on a consumer side, right?
I think that depends on whether it is consistently listed as a dependency. In the above mentioned test, the objc.modulemap
was there but it was not referenced in the meta
file so I believe not included in the rawFingerprint
.
Details and questions from our side
Using the meta files produced by different producer runs for the same commit (6f524499a878ad213c02f81eb36f5969996579ec
) on the PrivacyUI
git submodule we identified two issues:
Issue 1. The dependency list changes; Issue 2. The raw fingerprint changes even if the dependencies are the same;
We are investing both our setup (focused on the related modules) as well as RC
to understand what is going on.
I reviewed RC's code on identifying dependencies (Issue 1) and generating the raw fingerprint (Issue 2).
Regarding Issue 1 / identifying dependencies
:
a) isRelevantDependency
excludes modulemap
files in the $TARGET_BUILD_DIR
directory (~/Library/Developer/Xcode/DerivedData/{app}/Build/Products/Debug-iphonesimulator
). Should instead modulemap
files be excluded regardless of location?
In our case,
PrivacyUI
listsmodulemap
files (e.g.,BUIAssets/objc.modulemap
) under/buck-out/gen
. That could fix the dependency differences regardingmodulemap
files, though it is still unclear why they randomly show up in the/buck-out/gen
directory (likely a problem unrelated toRC
).
b) Otherwise isRelevantDependency
excludes any .xcode
, .intermediate
, .ownProduct
, .derivedFile
.
But in your setup: b.1)
RC 0.3.11
excludes all$DERIVED_FILE_DIR
files from dependency list. ForPrivacyUI
the value of$DERIVED_FILE_DIR
is~/Projects/iphone/Submodules/buck-out/xcode/derived-sources/PrivacyUI-16364f6141f26cc02c2e3f099242c1c0ece88448
. And there are found only-Swift.h
files (in this case onlyPrivacyUI-Swift.h
) file. b.2) Dependencies are read from$OBJECT_FILE_DIR_normal/arm64
. In our case$OBJECT_FILE_DIR_normal=~/Library/Developer/Xcode/DerivedData/{app}/Build/Intermediates.noindex/Privacy.build/Debug-iphonesimulator/PrivacyUI.build/Objects-normal
and we find files:.d
,.dia
,.o
,.swiftdeps
,~partial.swiftdoc
,~partial.swiftmodule
and~partial.swiftsourceinfo
Are those values matching RC
expectations? If not, can you clarify what files RC
expects to be where?
c) Why is fingerprintOverrideManager.getFingerprintFile
run over the dependency output and when?
Regarding Issue 2 / generating rawFingerprint
:
a) The piece that is unclear to me to is Postbuild.generateFingerprintOverrides
. Why and when is that run? Could it be causing Category 3 issues?
Hi @cezarsignori!
The compiler knows that /Pods/buck-out/gen/_p/MRmmgKh-hU-pub should be analyzed via HEADER_SEARCH_PATHS, SWIFT_INCLUDE_PATHS and isystem flags on OTHER_CFLAGS, OTHER_CPLUSPLUSFLAGS and OTHER_SWIFT_FLAGS.
I tried to reproduce that in a simple project but no matter which build setting I define, only module.modulemap
files are read by the swift compiler and obj.modulemap
is just skipped. The only way to force objc.modulemap
processing was passing -Xcc -fmodule-map-file=BUIAssets/obj.modulemap
. In the example project attached below, importing BUIAssetsInObj
always fails because it is not defined in module.modulemap
:
ModuleDemo.zip
Am I missing something? Do you know why does you project process obj.modulemap
?
isRelevantDependency excludes modulemap files in the $TARGET_BUILD_DIR directory (~/Library/Developer/Xcode/DerivedData/{app}/Build/Products/Debug-iphonesimulator). Should instead modulemap files be excluded regardless of location?
To be honest, isRelevantDependency
function is not perfect (thus TODO was added) and it unnecessary skips all .modulemap
s in TARGET_BUILD_DIR
. As a comment says, the method should precisely inspect if a given modulemap
was actually used and then include it in a list of dependencies. It is still an open question how to decide if a modulemap was relevant or not from that place so to play safe, XCRemoteCache just excludes all modulemap
s. That could lead to an edge case and overcache a target if someone changes .modulemap
only. Technically, a change in a .modulemap
that defines module A
should invalidate all its dependants (reverse dependencies). _I think it is not a problem for you as you don't have any modulemap
s in TARGET_BUILD_DIR
._
To conclude, I don't think we should exclude all modulemaps, quite opposite - we should consider .modulemap
in a fingerprint calculation even for some modules in TARGET_BUILD_DIR
.
RC 0.3.11 excludes all $DERIVED_FILE_DIR files from dependency list. For PrivacyUI the value of $DERIVED_FILE_DIR is ~/Projects/iphone/Submodules/buck-out/xcode/derived-sources/PrivacyUI-16364f6141f26cc02c2e3f099242c1c0ece88448. And there are found only -Swift.h files (in this case only PrivacyUI-Swift.h) file.
I think it is correct - PrivacyUI-Swift.h
shouldn't be defined as a dependency because this file is generated as a target compilation product. In other words, prior to PrivacyUI
compilation, when xcprebuild
is called, PrivacyUI-Swift.h
doesn't exists so if we were including it in a dependencies
list, a target wouldn't be cached.
Dependencies are read from $OBJECT_FILE_DIR_normal/arm64. In our case $OBJECT_FILE_DIR_normal=~/Library/Developer/Xcode/DerivedData/{app}/Build/Intermediates.noindex/Privacy.build/Debug-iphonesimulator/PrivacyUI.build/Objects-normal and we find files: .d, .dia, .o, .swiftdeps, ~partial.swiftdoc, ~partial.swiftmodule and ~partial.swiftsourceinfo. Are those values matching RC expectations? If not, can you clarify what files RC expects to be where?
That is fine. Dependencies are read only from .d
files, all other file extensions can sit next to it:
https://github.com/spotify/XCRemoteCache/blob/3b614c6172ffa50b39163ee25325533beca93029/Sources/XCRemoteCache/Dependencies/TargetDepdenciesReader.swift#L48
Why is fingerprintOverrideManager.getFingerprintFile run over the dependency output and when?
This is required when a byte-level fingerprinting of some fileextension is not portable. In the blogpost, section "Build artifacts portability" you can find a problem statement. In essence, we cannot generate a fingerprint from a binary file .swiftmodule
because it contains absolute paths. To overcome that, XCRemoteCache always generates an extra file next to xxx.swiftmodule
(default xxx.swiftmodule.md5
) that contains a normalized fingerprint of that target. By normalized I mean SRCROOT-agnostic. In the fingerprint generation, we should always consider yyy.swiftmodule.md5
content, not yyy.swiftmodule
. Sidenote: that xxx.md5
file is called "an override".
The piece that is unclear to me to is Postbuild.generateFingerprintOverrides. Why and when is that run? Could it be causing Category 3 issues?
generateFingerprintOverrides
is a step that generates an override (see above). It has to be called on a producer side (all Swift targets) and consumer side (only for Swift targets that have been compiled locally).
The aim of calling that on a consumer side is to make locally built .swiftmodule
compatible with XCRemoteCache.
Imagine a project with 2 swift targets A and B, where B depends on A. Let's assume we didn't introduce any changes locally. First we build A, but for some reason we had to compile it locally (e.g. VPN issue). When it comes a turn for target B compilation, we call xcprebuild
and generate fingerprint from all B's dependencies - including A.swiftmodule.md5
. If we weren't generating an override file for A.swiftmodule, B would have been compiled locally (cache miss because a dependency doesn't exist) even remotely available artifact could be used.
You are asking if that could be a reason of the issue 3? Yes. If your list of dependencies is non-deterministic, some other dependency of PrivacyUI
(e.g. Privacy
) could get a different "normalized" fingerprint - Privacy.swiftmodule.md5
and even the list of PrivacyUI
dependencies is the same - their content is different and a final fingerprint of the PrivacyUI
target is different.
So to make the fully stable fingerprinting, the list of dependencies for all targets has to be stable too.
I think that depends on whether it is consistently listed as a dependency. In the above mentioned test, the objc.modulemap was there but it was not referenced in the meta file so I believe not included in the rawFingerprint.
Right.
Again, having inconsistent list of dependencies in a meta is annoying and could lead to excessive artifact download on a consumer side, that shouldn't affect consumer's cache hit rate.
Am I missing something? Do you know why does you project process obj.modulemap?
Comparing to ModuleDemo
, BUIAssets
(dependency of PrivacyUI
) is a mixed module that depends on a bundle (BUIAssets-Bundle
), on a pure Swift module (BUIFoundations
), and on another mixed module (DSAssets
) that also depends on a bundle (DSAssets-Bundle
).
About the compiler flags, I do not know why it is included. I can confirm none of our .bzl
or BUCK
files contain the -fmodule-map-file
flag. In fact, the only fmodule*
flag we use is -fmodules
. And so far I have not seen it in the build logs. I will keep an eye for that.
I don't think we should exclude all modulemaps, quite opposite - we should consider .modulemap in a fingerprint calculation even for some modules in TARGET_BUILD_DIR.
My reasoning here is that if modulemap
files are excluded because their are in TARGET_BUILD_DIR
, they should be also excluded if they are in buck-out
. I mean, RC
expects modulemap
files to be in the DerivedData
directory, but in the buck
setup, they are found elsewhere. Until we find how to identify whether a modulemap
was actually used, it stands to reason RC
should consistently ignore them, as that was the original intent of the current code. Right?
Of course, making that change to RC
would not make it so we no longer need to have a consistent list of dependencies for our modules, but at the same time, my understanding is that we can't say RC
works as designed given in our setup buck
places modulemap
files outside of the TARGET_BUILD_DIR
. At the very least, that change should reduce the frequency of inconsistencies in the list of dependencies on our meta files (no more objc.modulemap
listed).
Makes sense?
You are asking if that could be a reason of the issue 3? Yes. If your list of dependencies is non-deterministic, some other dependency of PrivacyUI (e.g. Privacy) could get a different "normalized" fingerprint - Privacy.swiftmodule.md5 and even the list of PrivacyUI dependencies is the same - their content is different and a final fingerprint of the PrivacyUI target is different.
The Category 3 error is happening with PrivacyUI
but all dependencies have successful cache hits, meaning their fingerprint was a match and therefore no content difference. I believe that is a different behaviour than what you described. Correct?
Again, having inconsistent list of dependencies in a meta is annoying and could lead to excessive artifact download on a consumer side, that shouldn't affect consumer's cache hit rate.
I do not follow. Since the raw fingerprint is computed from the list of dependencies, inconsistencies in that list do affect hit rate (due to fingerprint mismatch between local and remote). That is what we are experiencing. Am I missing something?
I will reply for other questions, but for now have a draft PR that could unblock you (until you find out why you have unstable dependency list).
Until we find how to identify whether a modulemap was actually used, it stands to reason RC should consistently ignore them, as that was the original intent of the current code. Right?
Once #130 is merged, you could specify something like irrelevant_dependencies_paths: "Pods/buck-out/gen/.*\\.modulemap$"
to exclude buck's modulemaps only for your project. That potentially could lead to target overcaching if one introduces a change to a target's modulemap without modifying any other sources of the target - chances of happening that are low though. Please let me know if such property would unblock you and then I will create a PR.
@polac24 , by declaring irrelevant_dependencies_paths: ["\\.modulemap$"]
across all .rcinfo
files I was able to run 12 (local) producer builds where all modules consistently generated the same rawFingerprint
for the same commit (Thanks!). I am rolling out the change (still in a controlled environment) so CI starts to automatically build commits as they come in to increase the amount of samples.
My reasoning here is that if modulemap files are excluded because their are in TARGET_BUILD_DIR, they should be also excluded if they are in buck-out. I mean, RC expects modulemap files to be in the DerivedData directory, but in the buck setup, they are found elsewhere.
Excluding modulemaps in TARGET_BUILD_DIR
has another reason. Contrary to your scenario, we have a project where modulemaps are copied (ditto
ed) there progressively, as a postbuild phase step for each Xcode target. So these *.modulemap files do not exist prior to the Xcode build. Because an order of building Xcode targets is random, it may happen that some modulemaps that were added to a dependencies
list to a meta file on a producer, do not exists on a consumer side (because that target has not been finished yet there). The simplest workaround was to skip all modulemaps in TARGET_BUILD_DIR
, but the valid behaviour should only exclude modulemaps that are not actually used (there is TODO
in the code for that).
that we can't say RC works as designed given in our setup buck places modulemap files outside of the TARGET_BUILD_DIR.
Hard to say what is a reason of that and I wouldn't blame XCRemoteCache yet. We cannot reproduce in a sample project processing obj.modulemap
so until we know what is happening, we are blind.
At the very least, that change should reduce the frequency of inconsistencies in the list of dependencies on our meta files (no more objc.modulemap listed).
That is correct. Introducing irrelevant_dependencies_paths
can improve incremental build times after rebasing/merging.
Since the raw fingerprint is computed from the list of dependencies, inconsistencies in that list do affect hit rate (due to fingerprint mismatch between local and remote). That is what we are experiencing. Am I missing something?
Consumer always computes a fingerprint from the same set of files as a producer so there will be no fingerprint mismatch. The very first step in a xcprebuild
is downloading meta json for a given commit sha, which contains a list of dependencies. There always is a parity of local & remote files used to computer a fingerprint.
So I think this issue could be safely closed after the fix of adding irrelevant_dependencies_paths
config to exclude modulemap files that are generated by buck outside the DerivedData Dir.
Context We have managed integrating XCRemoteCache with our project, using manual integration steps that had been added as part of project generation rules using BUCK.
Desired output had been achieved of saving around 70% of build time.
End to end integration tested for producer and consumer on different machines where cache was being pushed to remote cache server.
During testing we were setting our feature branch (for XCRemoteCache Integration) as main XCRemoteCache branch and running the following steps:
rm -rf ~/Library/Developer/Xcode/DerivedData
andrm -rf ~/Library/Caches/XCRemoteCache;
on machine that with be used as consumer.Output: On consumer builds 100% cache hit achieved and around 70% in build time is being saved.
Problem After merging our feature branch to master, setting up master as main XC-RC branch, we integrated xc cache generation (building the project in producer mode) as part of our CI pipeline that runs immediately and automatically after merging any feature MR (merge request) to master.
By this setup for xc-rc cache generation, we started randomly seeing project build fails on local machines that builds the project with XC-RC integration in consumer mode.
On Xcode, we see errors like the following:
Showing All Messages /Users/anafei/git_tree/iosapp/Modules/Presentation/UI/SearchResultsList/Sources/ViewControllers/SearchResultsViewController.swift:21:12: Cannot load underlying module for 'ApplicationServices'
And on machine logs we see errors like:
(DMLScalarTypes) Local fingerprint Fingerprint(raw: "ddd713e9d85f977383c244f69787e55e", contextSpecific: "0fb9aac67d84f8e2ed5bc27bdc678489") does not match with remote one 9f8e9e9907f734f02b826d413d73905b.
And / Or: Tons of 404 fails to fetch the cache meta filesThe project build failure issue is some time being solved by just rebuilding the project again, and sometimes needed to pull latest master commit and clean the DerivedData, then rebuild again.
It worth to mention that, as our project is being extensively developed by multiple developers, so it happens that two merge requests or more are being merged to master with very small time difference and based on that CI pipeline would be running multiple times in parallel (one for each MR merged) and on different CI runner. So it happens that more than one producer job would be running in parallel on different CI runners, and generating cache for different commits at the same time.
Also as our CI pipeline for every MR takes considerable amount of time and cache generation is just one step of it, and also it runs after the MR is being merged to master so the following scenarios are expected too:
Also worth to mention and as highlighted above, during integration test we were always building the app in producer mode first, wait until producer build finishes and cache generation completes, then start testing to build the project in consumer mode and this was always succeeding with 100% cache hits. Same happens in our real integration, if we waited until all CI jobs completes, pulled the latest master commit on local machine and tried to build the project with consumer mode. This also always succeeds with 100% cache hit.
Environment