search

spotify / XCRemoteCache

Other
837 stars 52 forks source link

Remote cache cannot be used in consumer mode - local fingerprint does not match with remote one #174

Open ghost opened 1 year ago

ghost commented 1 year ago

My integration setup

Expected/desired behavior

The remote build cache is found and can be used in consumer mode. The local fingerprint should match the remote one.

Minimal reproduction of the problem with instructions

NOTE: If there are no changes to the iOS project code (regardless if this is a change to Swift or other filles), the cache is used and it gives a reasonable performance boost to Debug and AdHoc builds.

Producer Logs

 Filtering the log data using "sender == "xcprepare""
Timestamp               Ty Process[PID:TID]
2022-11-11 10:16:00.943 I  xcprepare[2899:2c42] Extra config override failed with missingConfigurationFile(user.rcinfo -- file:///Users/distiller/project/IOS_PROJECT_NAME/). Skipping extra configuration
2022-11-11 10:21:48.397 I  xcprepare[5853:5359] Making request https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/marker/3309eaa5d149c2c9f770f67b734db377dae83a06-Debug-iphonesimulator-14A400-5
2022-11-11 10:22:14.361 I  xcprepare[6120:5628] Making request https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/marker/3309eaa5d149c2c9f770f67b734db377dae83a06-Debug-iphonesimulator-14A400-5
Filtering the log data using "sender BEGINSWITH "xc""
Timestamp               Ty Process[PID:TID]
2022-11-11 10:16:00.943 I  xcprepare[2899:2c42] Extra config override failed with missingConfigurationFile(user.rcinfo -- file:///Users/distiller/project/IOS_PROJECT_NAME/). Skipping extra configuration
2022-11-11 10:21:00.843 E  xcprebuild[5694:4f48] (Store) Prebuild step disabled, selected mode: producer
2022-11-11 10:21:00.894 Df xcswiftc[5697:4f4c] Fallbacking to compilation using swiftc.
2022-11-11 10:21:37.236 I  xcpostbuild[5787:5231] (Store) Found url to remapp: file:///Users/distiller/project/DerivedData/Build/Products/. Remapping: /Users/distiller/project/DerivedData/Build/Products
2022-11-11 10:21:37.236 I  xcpostbuild[5787:5231] (Store) Found url to remapp: file:///Users/distiller/project/IOS_PROJECT_NAME/. Remapping: /Users/distiller/project/IOS_PROJECT_NAME
2022-11-11 10:21:43.059 I  xcpostbuild[5787:5231] (Store) Making upload request to https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/file/83cb3d728d9fb9c1a4850cb701c9ddb3.zip with 3 retries.
2022-11-11 10:21:46.115 I  xcpostbuild[5787:5231] (Store) Making upload request to https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/meta/3309eaa5d149c2c9f770f67b734db377dae83a06-Store-Debug-iphonesimulator-14A400-eccc143bfd3c1cd4f4c0d2842b7b3f1f.json with 3 retries.
2022-11-11 10:21:48.397 I  xcprepare[5853:5359] Making request https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/marker/3309eaa5d149c2c9f770f67b734db377dae83a06-Debug-iphonesimulator-14A400-5
2022-11-11 10:21:50.488 E  xcprebuild[5864:5384] (IOS_PROJECT_NAMEUITests) Prebuild step disabled, selected mode: producer
2022-11-11 10:21:50.504 Df xcswiftc[5865:5386] Fallbacking to compilation using swiftc.
2022-11-11 10:22:07.296 I  xcpostbuild[5884:54c0] (IOS_PROJECT_NAMEUITests) Found url to remapp: file:///Users/distiller/project/DerivedData/Build/Products/. Remapping: /Users/distiller/project/DerivedData/Build/Products
2022-11-11 10:22:07.296 I  xcpostbuild[5884:54c0] (IOS_PROJECT_NAMEUITests) Found url to remapp: file:///Users/distiller/project/IOS_PROJECT_NAME/. Remapping: /Users/distiller/project/IOS_PROJECT_NAME
2022-11-11 10:22:09.965 I  xcpostbuild[5884:54c0] (IOS_PROJECT_NAMEUITests) Making upload request to https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/file/bdba6cc8d2a19f991b8d0024a9f729d0.zip with 3 retries.
2022-11-11 10:22:11.641 I  xcpostbuild[5884:54c0] (IOS_PROJECT_NAMEUITests) Making upload request to https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/meta/3309eaa5d149c2c9f770f67b734db377dae83a06-IOS_PROJECT_NAMEUITests-Debug-iphonesimulator-14A400-bfd512a3a8ee6ba690a014460dbe617f.json with 3 retries.
2022-11-11 10:22:14.361 I  xcprepare[6120:5628] Making request https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/marker/3309eaa5d149c2c9f770f67b734db377dae83a06-Debug-iphonesimulator-14A400-5

Consumer Logs

 Filtering the log data using "sender == "xcprepare""
Timestamp               Ty Process[PID:TID]
2022-11-11 10:38:52.374 I  xcprepare[2464:215a] Extra config override failed with missingConfigurationFile(user.rcinfo -- file:///Users/distiller/project/IOS_PROJECT_NAME/). Skipping extra configuration
2022-11-11 10:38:52.951 I  xcprepare[2464:215a] Checking HTTP file HEAD for https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-11 10:38:52.951 I  xcprepare[2464:215a] Making request https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-11 10:38:53.173 I  xcprepare[2464:215a] Checking HTTP file HEAD for https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-11 10:38:53.173 I  xcprepare[2464:215a] Making request https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-11 10:38:53.226 I  xcprepare[2464:215a] Checking HTTP file HEAD for https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-11 10:38:53.226 I  xcprepare[2464:215a] Making request https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-11 10:38:53.699 I  xcprepare[2464:215a] Checking HTTP file HEAD for https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/marker/3309eaa5d149c2c9f770f67b734db377dae83a06-Debug-iphonesimulator-14A400-5
2022-11-11 10:38:53.699 I  xcprepare[2464:215a] Making request https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/marker/3309eaa5d149c2c9f770f67b734db377dae83a06-Debug-iphonesimulator-14A400-5
2022-11-11 10:38:53.755 I  xcprepare[2464:215a] ClangWrapperBuilder compiles file at file:///var/folders/bl/wbxjgtzx7j5_mjsmfr3ynlc00000gp/T/xccc.c.
2022-11-11 10:38:55.277 I  xcprepare[2464:215a] Clang compilation output:
2022-11-11 10:38:55.939 I  xcprepare[2696:22b3] Extra config override failed with missingConfigurationFile(user.rcinfo -- file:///Users/distiller/project/IOS_PROJECT_NAME/). Skipping extra configuration
2022-11-11 10:38:56.500 I  xcprepare[2697:22b5] Checking HTTP file HEAD for https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-11 10:38:56.500 I  xcprepare[2697:22b5] Making request https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-11 10:38:56.595 I  xcprepare[2697:22b5] Checking HTTP file HEAD for https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-11 10:38:56.595 I  xcprepare[2697:22b5] Making request https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-11 10:38:56.648 I  xcprepare[2697:22b5] Checking HTTP file HEAD for https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-11 10:38:56.648 I  xcprepare[2697:22b5] Making request https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-11 10:38:57.055 I  xcprepare[2697:22b5] Checking HTTP file HEAD for https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/marker/3309eaa5d149c2c9f770f67b734db377dae83a06-Debug-iphonesimulator-14A400-5
2022-11-11 10:38:57.055 I  xcprepare[2697:22b5] Making request https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/marker/3309eaa5d149c2c9f770f67b734db377dae83a06-Debug-iphonesimulator-14A400-5
2022-11-11 10:38:57.110 I  xcprepare[2697:22b5] ClangWrapperBuilder compiles file at file:///var/folders/bl/wbxjgtzx7j5_mjsmfr3ynlc00000gp/T/xccc.c.
2022-11-11 10:38:57.615 I  xcprepare[2697:22b5] Clang compilation output:
Filtering the log data using "sender BEGINSWITH "xc""
Timestamp               Ty Process[PID:TID]
2022-11-11 10:38:52.374 I  xcprepare[2464:215a] Extra config override failed with missingConfigurationFile(user.rcinfo -- file:///Users/distiller/project/IOS_PROJECT_NAME/). Skipping extra configuration
2022-11-11 10:38:52.951 I  xcprepare[2464:215a] Checking HTTP file HEAD for https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-11 10:38:52.951 I  xcprepare[2464:215a] Making request https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-11 10:38:53.173 I  xcprepare[2464:215a] Checking HTTP file HEAD for https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-11 10:38:53.173 I  xcprepare[2464:215a] Making request https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-11 10:38:53.226 I  xcprepare[2464:215a] Checking HTTP file HEAD for https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-11 10:38:53.226 I  xcprepare[2464:215a] Making request https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-11 10:38:53.699 I  xcprepare[2464:215a] Checking HTTP file HEAD for https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/marker/3309eaa5d149c2c9f770f67b734db377dae83a06-Debug-iphonesimulator-14A400-5
2022-11-11 10:38:53.699 I  xcprepare[2464:215a] Making request https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/marker/3309eaa5d149c2c9f770f67b734db377dae83a06-Debug-iphonesimulator-14A400-5
2022-11-11 10:38:53.755 I  xcprepare[2464:215a] ClangWrapperBuilder compiles file at file:///var/folders/bl/wbxjgtzx7j5_mjsmfr3ynlc00000gp/T/xccc.c.
2022-11-11 10:38:55.277 I  xcprepare[2464:215a] Clang compilation output:
2022-11-11 10:38:55.939 I  xcprepare[2696:22b3] Extra config override failed with missingConfigurationFile(user.rcinfo -- file:///Users/distiller/project/IOS_PROJECT_NAME/). Skipping extra configuration
2022-11-11 10:38:56.500 I  xcprepare[2697:22b5] Checking HTTP file HEAD for https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-11 10:38:56.500 I  xcprepare[2697:22b5] Making request https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-11 10:38:56.595 I  xcprepare[2697:22b5] Checking HTTP file HEAD for https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-11 10:38:56.595 I  xcprepare[2697:22b5] Making request https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-11 10:38:56.648 I  xcprepare[2697:22b5] Checking HTTP file HEAD for https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-11 10:38:56.648 I  xcprepare[2697:22b5] Making request https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-11 10:38:57.055 I  xcprepare[2697:22b5] Checking HTTP file HEAD for https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/marker/3309eaa5d149c2c9f770f67b734db377dae83a06-Debug-iphonesimulator-14A400-5
2022-11-11 10:38:57.055 I  xcprepare[2697:22b5] Making request https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/marker/3309eaa5d149c2c9f770f67b734db377dae83a06-Debug-iphonesimulator-14A400-5
2022-11-11 10:38:57.110 I  xcprepare[2697:22b5] ClangWrapperBuilder compiles file at file:///var/folders/bl/wbxjgtzx7j5_mjsmfr3ynlc00000gp/T/xccc.c.
2022-11-11 10:38:57.615 I  xcprepare[2697:22b5] Clang compilation output:
2022-11-11 10:44:06.823 E  xcprebuild[5657:48d6] (Store) Couldn't verify if should disable RC for 3309eaa5d149c2c9f770f67b734db377dae83a06.
2022-11-11 10:44:06.829 I  xcprebuild[5657:48d6] (Store) Found url to remapp: file:///Users/distiller/project/DerivedData/Build/Products/. Remapping: /Users/distiller/project/DerivedData/Build/Products
2022-11-11 10:44:06.829 I  xcprebuild[5657:48d6] (Store) Found url to remapp: file:///Users/distiller/project/IOS_PROJECT_NAME/. Remapping: /Users/distiller/project/IOS_PROJECT_NAME
2022-11-11 10:44:06.829 I  xcprebuild[5657:48d6] (Store) Making request https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/meta/3309eaa5d149c2c9f770f67b734db377dae83a06-Store-Debug-iphonesimulator-14A400-eccc143bfd3c1cd4f4c0d2842b7b3f1f.json
2022-11-11 10:44:07.383 I  xcprebuild[5657:48d6] (Store)     Local fingerprint Fingerprint(raw: "7f7331398da0aac6f226664cee86d6a5", contextSpecific: "f7de4deca3879dab62427e7dd14593d0") does not match with remote one d96f8009a62cf5041ba1af1f56480fb7.
2022-11-11 10:44:07.383 I  xcprebuild[5657:48d6] (Store) Remote cache cannot be used
2022-11-11 10:44:07.454 I  xcswiftc[5660:48e0] Swiftc marker doesn't exist
2022-11-11 10:44:07.454 Df xcswiftc[5660:48e0] Fallbacking to compilation using swiftc.
2022-11-11 10:44:45.318 I  xcpostbuild[5745:4c85] (Store) Found url to remapp: file:///Users/distiller/project/DerivedData/Build/Products/. Remapping: /Users/distiller/project/DerivedData/Build/Products
2022-11-11 10:44:45.319 I  xcpostbuild[5745:4c85] (Store) Found url to remapp: file:///Users/distiller/project/IOS_PROJECT_NAME/. Remapping: /Users/distiller/project/IOS_PROJECT_NAME
2022-11-11 10:44:48.970 E  xcprebuild[5812:4d96] (IOS_PROJECT_NAMEUITests) Couldn't verify if should disable RC for 3309eaa5d149c2c9f770f67b734db377dae83a06.
2022-11-11 10:44:48.977 I  xcprebuild[5812:4d96] (IOS_PROJECT_NAMEUITests) Found url to remapp: file:///Users/distiller/project/DerivedData/Build/Products/. Remapping: /Users/distiller/project/DerivedData/Build/Products
2022-11-11 10:44:48.977 I  xcprebuild[5812:4d96] (IOS_PROJECT_NAMEUITests) Found url to remapp: file:///Users/distiller/project/IOS_PROJECT_NAME/. Remapping: /Users/distiller/project/IOS_PROJECT_NAME
2022-11-11 10:44:48.979 I  xcprebuild[5812:4d96] (IOS_PROJECT_NAMEUITests) Making request https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/meta/3309eaa5d149c2c9f770f67b734db377dae83a06-IOS_PROJECT_NAMEUITests-Debug-iphonesimulator-14A400-bfd512a3a8ee6ba690a014460dbe617f.json
2022-11-11 10:44:49.287 I  xcprebuild[5812:4d96] (IOS_PROJECT_NAMEUITests)     Local fingerprint Fingerprint(raw: "2907b625f1e944892665a44a3a7856a5", contextSpecific: "688c42fc064e8bcf48b4ce549e52805b") does not match with remote one 55fd71b5a47a3711be8f21fd297de83b.
2022-11-11 10:44:49.287 I  xcprebuild[5812:4d96] (IOS_PROJECT_NAMEUITests) Remote cache cannot be used
2022-11-11 10:44:49.349 I  xcswiftc[5814:4da5] Swiftc marker doesn't exist
2022-11-11 10:44:49.349 Df xcswiftc[5814:4da5] Fallbacking to compilation using swiftc.
2022-11-11 10:45:08.351 I  xcpostbuild[5831:4f15] (IOS_PROJECT_NAMEUITests) Found url to remapp: file:///Users/distiller/project/DerivedData/Build/Products/. Remapping: /Users/distiller/project/DerivedData/Build/Products
2022-11-11 10:45:08.351 I  xcpostbuild[5831:4f15] (IOS_PROJECT_NAMEUITests) Found url to remapp: file:///Users/distiller/project/IOS_PROJECT_NAME/. Remapping: /Users/distiller/project/IOS_PROJECT_NAME

Environment

Post build stats

 
hit_count: 0
miss_count: 2
local_cache_bytes: 118784
indexing_hit_count: 0
indexing_miss_count: 0

Others

I tried to debug an issue with https://github.com/spotify/XCRemoteCache/blob/master/docs/FAQ.md#troubleshooting-cache-misses guide but was no luck.

Also I tried to experiment with cache_commit_history and artifact_maximum_age flags - but no luck. See the latest .rcinfo

primary_repo: git@github.com:PROJECT/IOS_PROJECT_NAME.git
primary_branch: master
cache_addresses:
 - https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/
cache_health_path: bucket-health/
cache_commit_history: 100
artifact_maximum_age: 0
request_custom_headers: 
 Authorization: "Bearer ***"
polac24 commented 1 year ago

Thanks @pp-gtroshin for the extensive logs. I wish we have more such descriptive reports.

Can you look into https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/meta/3309eaa5d149c2c9f770f67b734db377dae83a06-Store-Debug-iphonesimulator-14A400-eccc143bfd3c1cd4f4c0d2842b7b3f1f.json content? There will be a list of files that are used to compute the fingerprint and verify if none of these files are static, deterministically generated and do not contain absolute paths as a content?

In logs I see that there is remapping involved (see file:///Users/distiller/project/IOS_PROJECT_NAME/. Remapping: /Users/distiller/project/IOS_PROJECT_NAME) is that something you manually added in the .rcinfo? Probably not because you haven't included that in the snippet at the bottom so it might be a reason of the vfs used when Xcode builds mixed frameworks. Maybe that is a source of "content mismatch"?

ghost commented 1 year ago

Thanks for the swift response, @polac24!

I checked the metafile, you referenced.

Regarding the remapping. I actually thought this is not important for some reason. Since we found that we have absolute paths of all SPM (SourcePackages) located at /Users/distiller/project/SourcePackages/checkouts/ and other files of DerivedData located at /Users/distiller/project/DerivedData/Build/Intermediates.noindex/GeneratedModuleMaps-iphonesimulator/.

Please, see the screenshots.

SPM DD

/Users/distiller/project is where CI checks out a code.

Should I add the out_of_band_mappings to the .rcinfo file? And should I add both iphonesimulator and iphoneos for DerivedData out of band mappings?

In my specific case, the .rcinfo file will have to look as following (please, confirm if this is right): Note: I did not include the GeneratedModuleMaps-iphonesimulator/ in the paths (hope this is fine).

primary_repo: git@github.com:PROJECT/IOS_PROJECT_NAME.git
primary_branch: master
cache_addresses:
 - https://IOS_PROJECT_NAME-xcode-remote-cache.storage.googleapis.com/
cache_health_path: bucket-health/
cache_commit_history: 100
artifact_maximum_age: 0
request_custom_headers: 
 Authorization: "Bearer ***"
out_of_band_mappings: 
 "SOURCE_PACKAGES": "/Users/distiller/project/SourcePackages/checkouts"
 "DD_INTERMEDIATES": "/Users/distiller/project/DerivedData/Build/Intermediates.noindex"

Also, if this helps, this is a project structure and how we integrate XCRemoteCache at the PROJECT_NAME folder where the main target of the xcworkspace is located, next to PROJECT_NAME.xcodeproj):

mkdir PROJECT_NAME/xcremotecache
curl -L -o PROJECT_NAME/xcremotecache/XCRemoteCache.zip https://github.com/spotify/XCRemoteCache/releases/download/v0.3.17/XCRemoteCache-macOS-x86_64-v0.3.17.zip
cd PROJECT_NAME/xcremotecache/ && unzip XCRemoteCache.zip && rm XCRemoteCache.zip
cd PROJECT_NAME && xcremotecache/xcprepare integrate --input << parameters.xcode_project >> --mode producer --final-producer-target << parameters.final_producer_target >>
.
├── Configurations
├── Docs
├── Frameworks
├── Playgrounds
├── XXX
├── XXX
├── XXX
├── xx_scripts
├── custom_xxfiles
├── fastlane
├── git-submodules
├── PROJECT_NAME
├── PROJECT_NAME.xcworkspace
├── PROJECT_NAMEXX
└── PROJECT_NAMEUITestCore
polac24 commented 1 year ago

Hi! Sorry for the mistake (replying on Friday evening wasn't the best decision). Instead none of these files are static I wanted to say none of these files are dynamic 🤦‍♂️. So if any of your input files (e.g. .swift) are code-generated, make sure the codegen output is stable (produces the same output on a byte-level). Seems that it is not the case in your project.

The out-of-band os is exactly designed for your scenario and the shared .rcinfo looks good. Remember that this is fine for the producer side only. On a consume side, you need to set these SOURCE_PACKAGES and DD_INTERMEDIATES to the absolute path that a consumer (probably a local developer) uses. Thus placing them in user.rcinfo sounds reasonable.

I know this is a bit cumbersome to handle these out-of-band paths in an external script so for such cases, it would be good to investigate when and by who are these files generated and think if there is any better way to reference them in the .xcodeproj. But for now, I suggest going with the out-of-band approach. You mention that the first batch of files come from SPM so supporting such dependencies automatically, without out-of-band mapping, could be something to add in the XCRemoteCache upstream.

ghost commented 1 year ago

@polac24 thanks for the response.

I tried this out_of_band_mappings but after I make any changes to the project .swift files, it causes the same issue. Please, see the logs here, below. Any Idea if I could do something in a different way? Can you confirm that out_of_band_mappings worked in a producer mode and in consumer mode in my case? I also checked all the .json files in meta and for the corresponding commit SHA there are no absolute paths. (I build only on CI machines)

Producer Logs

  Timestamp               Ty Process[PID:TID]
2022-11-15 10:29:59.427 I  xcprepare[2659:2459] Extra config override failed with missingConfigurationFile(user.rcinfo -- file:///Users/distiller/project/IOS_PROJECT_NAME/). Skipping extra configuration
2022-11-15 10:35:47.752 I  xcprepare[5571:4968] Making request https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/marker/65e733e798be3c2e53d837db4f9fd7b20e043c9d-Debug-iphonesimulator-14A400-5
2022-11-15 10:36:12.664 I  xcprepare[5835:4c23] Making request https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/marker/65e733e798be3c2e53d837db4f9fd7b20e043c9d-Debug-iphonesimulator-14A400-5
Filtering the log data using "sender BEGINSWITH "xc""
Timestamp               Ty Process[PID:TID]
2022-11-15 10:29:59.427 I  xcprepare[2659:2459] Extra config override failed with missingConfigurationFile(user.rcinfo -- file:///Users/distiller/project/IOS_PROJECT_NAME/). Skipping extra configuration
2022-11-15 10:35:01.117 E  xcprebuild[5420:45a8] (Store) Prebuild step disabled, selected mode: producer
2022-11-15 10:35:01.146 Df xcswiftc[5423:45ac] Fallbacking to compilation using swiftc.
2022-11-15 10:35:36.891 I  xcpostbuild[5506:484c] (Store) Found url to remapp: file:///Users/distiller/project/DerivedData/Build/Products/. Remapping: /Users/distiller/project/DerivedData/Build/Products
2022-11-15 10:35:36.891 I  xcpostbuild[5506:484c] (Store) Found url to remapp: file:///Users/distiller/project/IOS_PROJECT_NAME/. Remapping: /Users/distiller/project/IOS_PROJECT_NAME
2022-11-15 10:35:36.891 I  xcpostbuild[5506:484c] (Store) Found url to remapp: file:///Users/distiller/project/SourcePackages/checkouts/. Remapping: /Users/distiller/project/SourcePackages/checkouts
2022-11-15 10:35:36.891 I  xcpostbuild[5506:484c] (Store) Found url to remapp: file:///Users/distiller/project/DerivedData/Build/Intermediates.noindex/. Remapping: /Users/distiller/project/DerivedData/Build/Intermediates.noindex
2022-11-15 10:35:42.669 I  xcpostbuild[5506:484c] (Store) Making upload request to https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/file/31d1b7a3b042cc299ff6d2800ca3ef4f.zip with 3 retries.
2022-11-15 10:35:45.576 I  xcpostbuild[5506:484c] (Store) Making upload request to https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/meta/65e733e798be3c2e53d837db4f9fd7b20e043c9d-Store-Debug-iphonesimulator-14A400-eccc143bfd3c1cd4f4c0d2842b7b3f1f.json with 3 retries.
2022-11-15 10:35:47.752 I  xcprepare[5571:4968] Making request https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/marker/65e733e798be3c2e53d837db4f9fd7b20e043c9d-Debug-iphonesimulator-14A400-5
2022-11-15 10:35:49.583 E  xcprebuild[5581:4983] (IOS_PROJECT_NAMEUITests) Prebuild step disabled, selected mode: producer
2022-11-15 10:35:49.599 Df xcswiftc[5582:4985] Fallbacking to compilation using swiftc.
2022-11-15 10:36:05.082 I  xcpostbuild[5599:4aaf] (IOS_PROJECT_NAMEUITests) Found url to remapp: file:///Users/distiller/project/DerivedData/Build/Products/. Remapping: /Users/distiller/project/DerivedData/Build/Products
2022-11-15 10:36:05.082 I  xcpostbuild[5599:4aaf] (IOS_PROJECT_NAMEUITests) Found url to remapp: file:///Users/distiller/project/IOS_PROJECT_NAME/. Remapping: /Users/distiller/project/IOS_PROJECT_NAME
2022-11-15 10:36:05.082 I  xcpostbuild[5599:4aaf] (IOS_PROJECT_NAMEUITests) Found url to remapp: file:///Users/distiller/project/DerivedData/Build/Intermediates.noindex/. Remapping: /Users/distiller/project/DerivedData/Build/Intermediates.noindex
2022-11-15 10:36:05.082 I  xcpostbuild[5599:4aaf] (IOS_PROJECT_NAMEUITests) Found url to remapp: file:///Users/distiller/project/SourcePackages/checkouts/. Remapping: /Users/distiller/project/SourcePackages/checkouts
2022-11-15 10:36:08.126 I  xcpostbuild[5599:4aaf] (IOS_PROJECT_NAMEUITests) Making upload request to https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/file/f97eae5c59336be55594f95bc31ae2dc.zip with 3 retries.
2022-11-15 10:36:09.841 I  xcpostbuild[5599:4aaf] (IOS_PROJECT_NAMEUITests) Making upload request to https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/meta/65e733e798be3c2e53d837db4f9fd7b20e043c9d-IOS_PROJECT_NAMEUITests-Debug-iphonesimulator-14A400-bfd512a3a8ee6ba690a014460dbe617f.json with 3 retries.
2022-11-15 10:36:12.664 I  xcprepare[5835:4c23] Making request https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/marker/65e733e798be3c2e53d837db4f9fd7b20e043c9d-Debug-iphonesimulator-14A400-5

Consumer Logs

  Filtering the log data using "sender == "xcprepare""
Timestamp               Ty Process[PID:TID]
2022-11-15 10:53:01.144 I  xcprepare[2420:1e3e] Extra config override failed with missingConfigurationFile(user.rcinfo -- file:///Users/distiller/project/IOS_PROJECT_NAME/). Skipping extra configuration
2022-11-15 10:53:01.631 I  xcprepare[2420:1e3e] Checking HTTP file HEAD for https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-15 10:53:01.631 I  xcprepare[2420:1e3e] Making request https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-15 10:53:01.840 I  xcprepare[2420:1e3e] Checking HTTP file HEAD for https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-15 10:53:01.840 I  xcprepare[2420:1e3e] Making request https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-15 10:53:01.903 I  xcprepare[2420:1e3e] Checking HTTP file HEAD for https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-15 10:53:01.903 I  xcprepare[2420:1e3e] Making request https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-15 10:53:02.362 I  xcprepare[2420:1e3e] Checking HTTP file HEAD for https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/marker/65e733e798be3c2e53d837db4f9fd7b20e043c9d-Debug-iphonesimulator-14A400-5
2022-11-15 10:53:02.362 I  xcprepare[2420:1e3e] Making request https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/marker/65e733e798be3c2e53d837db4f9fd7b20e043c9d-Debug-iphonesimulator-14A400-5
2022-11-15 10:53:02.429 I  xcprepare[2420:1e3e] ClangWrapperBuilder compiles file at file:///var/folders/bl/wbxjgtzx7j5_mjsmfr3ynlc00000gp/T/xccc.c.
2022-11-15 10:53:03.965 I  xcprepare[2420:1e3e] Clang compilation output:
2022-11-15 10:53:04.467 I  xcprepare[2649:1f78] Extra config override failed with missingConfigurationFile(user.rcinfo -- file:///Users/distiller/project/IOS_PROJECT_NAME/). Skipping extra configuration
2022-11-15 10:53:04.990 I  xcprepare[2650:1f7a] Checking HTTP file HEAD for https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-15 10:53:04.990 I  xcprepare[2650:1f7a] Making request https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-15 10:53:05.114 I  xcprepare[2650:1f7a] Checking HTTP file HEAD for https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-15 10:53:05.114 I  xcprepare[2650:1f7a] Making request https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-15 10:53:05.175 I  xcprepare[2650:1f7a] Checking HTTP file HEAD for https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-15 10:53:05.175 I  xcprepare[2650:1f7a] Making request https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-15 10:53:05.650 I  xcprepare[2650:1f7a] Checking HTTP file HEAD for https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/marker/65e733e798be3c2e53d837db4f9fd7b20e043c9d-Debug-iphonesimulator-14A400-5
2022-11-15 10:53:05.650 I  xcprepare[2650:1f7a] Making request https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/marker/65e733e798be3c2e53d837db4f9fd7b20e043c9d-Debug-iphonesimulator-14A400-5
2022-11-15 10:53:05.712 I  xcprepare[2650:1f7a] ClangWrapperBuilder compiles file at file:///var/folders/bl/wbxjgtzx7j5_mjsmfr3ynlc00000gp/T/xccc.c.
2022-11-15 10:53:06.057 I  xcprepare[2650:1f7a] Clang compilation output:
Filtering the log data using "sender BEGINSWITH "xc""
Timestamp               Ty Process[PID:TID]
2022-11-15 10:53:01.144 I  xcprepare[2420:1e3e] Extra config override failed with missingConfigurationFile(user.rcinfo -- file:///Users/distiller/project/IOS_PROJECT_NAME/). Skipping extra configuration
2022-11-15 10:53:01.631 I  xcprepare[2420:1e3e] Checking HTTP file HEAD for https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-15 10:53:01.631 I  xcprepare[2420:1e3e] Making request https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-15 10:53:01.840 I  xcprepare[2420:1e3e] Checking HTTP file HEAD for https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-15 10:53:01.840 I  xcprepare[2420:1e3e] Making request https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-15 10:53:01.903 I  xcprepare[2420:1e3e] Checking HTTP file HEAD for https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-15 10:53:01.903 I  xcprepare[2420:1e3e] Making request https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-15 10:53:02.362 I  xcprepare[2420:1e3e] Checking HTTP file HEAD for https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/marker/65e733e798be3c2e53d837db4f9fd7b20e043c9d-Debug-iphonesimulator-14A400-5
2022-11-15 10:53:02.362 I  xcprepare[2420:1e3e] Making request https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/marker/65e733e798be3c2e53d837db4f9fd7b20e043c9d-Debug-iphonesimulator-14A400-5
2022-11-15 10:53:02.429 I  xcprepare[2420:1e3e] ClangWrapperBuilder compiles file at file:///var/folders/bl/wbxjgtzx7j5_mjsmfr3ynlc00000gp/T/xccc.c.
2022-11-15 10:53:03.965 I  xcprepare[2420:1e3e] Clang compilation output:
2022-11-15 10:53:04.467 I  xcprepare[2649:1f78] Extra config override failed with missingConfigurationFile(user.rcinfo -- file:///Users/distiller/project/IOS_PROJECT_NAME/). Skipping extra configuration
2022-11-15 10:53:04.990 I  xcprepare[2650:1f7a] Checking HTTP file HEAD for https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-15 10:53:04.990 I  xcprepare[2650:1f7a] Making request https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-15 10:53:05.114 I  xcprepare[2650:1f7a] Checking HTTP file HEAD for https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-15 10:53:05.114 I  xcprepare[2650:1f7a] Making request https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-15 10:53:05.175 I  xcprepare[2650:1f7a] Checking HTTP file HEAD for https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-15 10:53:05.175 I  xcprepare[2650:1f7a] Making request https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/bucket-health/
2022-11-15 10:53:05.650 I  xcprepare[2650:1f7a] Checking HTTP file HEAD for https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/marker/65e733e798be3c2e53d837db4f9fd7b20e043c9d-Debug-iphonesimulator-14A400-5
2022-11-15 10:53:05.650 I  xcprepare[2650:1f7a] Making request https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/marker/65e733e798be3c2e53d837db4f9fd7b20e043c9d-Debug-iphonesimulator-14A400-5
2022-11-15 10:53:05.712 I  xcprepare[2650:1f7a] ClangWrapperBuilder compiles file at file:///var/folders/bl/wbxjgtzx7j5_mjsmfr3ynlc00000gp/T/xccc.c.
2022-11-15 10:53:06.057 I  xcprepare[2650:1f7a] Clang compilation output:
2022-11-15 10:58:04.436 E  xcprebuild[5432:41e2] (Store) Couldn't verify if should disable RC for 65e733e798be3c2e53d837db4f9fd7b20e043c9d.
2022-11-15 10:58:04.441 I  xcprebuild[5432:41e2] (Store) Found url to remapp: file:///Users/distiller/project/DerivedData/Build/Products/. Remapping: /Users/distiller/project/DerivedData/Build/Products
2022-11-15 10:58:04.441 I  xcprebuild[5432:41e2] (Store) Found url to remapp: file:///Users/distiller/project/IOS_PROJECT_NAME/. Remapping: /Users/distiller/project/IOS_PROJECT_NAME
2022-11-15 10:58:04.441 I  xcprebuild[5432:41e2] (Store) Found url to remapp: file:///Users/distiller/project/SourcePackages/checkouts/. Remapping: /Users/distiller/project/SourcePackages/checkouts
2022-11-15 10:58:04.441 I  xcprebuild[5432:41e2] (Store) Found url to remapp: file:///Users/distiller/project/DerivedData/Build/Intermediates.noindex/. Remapping: /Users/distiller/project/DerivedData/Build/Intermediates.noindex
2022-11-15 10:58:04.442 I  xcprebuild[5432:41e2] (Store) Making request https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/meta/65e733e798be3c2e53d837db4f9fd7b20e043c9d-Store-Debug-iphonesimulator-14A400-eccc143bfd3c1cd4f4c0d2842b7b3f1f.json
2022-11-15 10:58:04.813 I  xcprebuild[5432:41e2] (Store)     Local fingerprint Fingerprint(raw: "ead1de79f9dd72610f231c2a0bb55180", contextSpecific: "af9a748e0c8d336f78372264a0adbb8d") does not match with remote one cd38140edca6591b75a6560201d8c09e.
2022-11-15 10:58:04.813 I  xcprebuild[5432:41e2] (Store) Remote cache cannot be used
2022-11-15 10:58:04.883 I  xcswiftc[5435:41ed] Swiftc marker doesn't exist
2022-11-15 10:58:04.884 Df xcswiftc[5435:41ed] Fallbacking to compilation using swiftc.
2022-11-15 10:58:39.402 I  xcpostbuild[5516:451e] (Store) Found url to remapp: file:///Users/distiller/project/DerivedData/Build/Products/. Remapping: /Users/distiller/project/DerivedData/Build/Products
2022-11-15 10:58:39.402 I  xcpostbuild[5516:451e] (Store) Found url to remapp: file:///Users/distiller/project/IOS_PROJECT_NAME/. Remapping: /Users/distiller/project/IOS_PROJECT_NAME
2022-11-15 10:58:39.402 I  xcpostbuild[5516:451e] (Store) Found url to remapp: file:///Users/distiller/project/SourcePackages/checkouts/. Remapping: /Users/distiller/project/SourcePackages/checkouts
2022-11-15 10:58:39.402 I  xcpostbuild[5516:451e] (Store) Found url to remapp: file:///Users/distiller/project/DerivedData/Build/Intermediates.noindex/. Remapping: /Users/distiller/project/DerivedData/Build/Intermediates.noindex
2022-11-15 10:58:44.230 E  xcprebuild[5581:4625] (IOS_PROJECT_NAMEUITests) Couldn't verify if should disable RC for 65e733e798be3c2e53d837db4f9fd7b20e043c9d.
2022-11-15 10:58:44.235 I  xcprebuild[5581:4625] (IOS_PROJECT_NAMEUITests) Found url to remapp: file:///Users/distiller/project/DerivedData/Build/Products/. Remapping: /Users/distiller/project/DerivedData/Build/Products
2022-11-15 10:58:44.235 I  xcprebuild[5581:4625] (IOS_PROJECT_NAMEUITests) Found url to remapp: file:///Users/distiller/project/IOS_PROJECT_NAME/. Remapping: /Users/distiller/project/IOS_PROJECT_NAME
2022-11-15 10:58:44.235 I  xcprebuild[5581:4625] (IOS_PROJECT_NAMEUITests) Found url to remapp: file:///Users/distiller/project/DerivedData/Build/Intermediates.noindex/. Remapping: /Users/distiller/project/DerivedData/Build/Intermediates.noindex
2022-11-15 10:58:44.235 I  xcprebuild[5581:4625] (IOS_PROJECT_NAMEUITests) Found url to remapp: file:///Users/distiller/project/SourcePackages/checkouts/. Remapping: /Users/distiller/project/SourcePackages/checkouts
2022-11-15 10:58:44.235 I  xcprebuild[5581:4625] (IOS_PROJECT_NAMEUITests) Making request https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/meta/65e733e798be3c2e53d837db4f9fd7b20e043c9d-IOS_PROJECT_NAMEUITests-Debug-iphonesimulator-14A400-bfd512a3a8ee6ba690a014460dbe617f.json
2022-11-15 10:58:44.641 I  xcprebuild[5581:4625] (IOS_PROJECT_NAMEUITests)     Local fingerprint Fingerprint(raw: "d59a17b2982904e3739c9357ea4280a0", contextSpecific: "ed63b208afce7dfcb01b5bbc61274527") does not match with remote one 1bd8e0fbf59b613c8e6b3269fe322fe4.
2022-11-15 10:58:44.641 I  xcprebuild[5581:4625] (IOS_PROJECT_NAMEUITests) Remote cache cannot be used
2022-11-15 10:58:44.735 I  xcswiftc[5582:462a] Swiftc marker doesn't exist
2022-11-15 10:58:44.735 Df xcswiftc[5582:462a] Fallbacking to compilation using swiftc.
2022-11-15 10:59:00.920 I  xcpostbuild[5597:475e] (IOS_PROJECT_NAMEUITests) Found url to remapp: file:///Users/distiller/project/DerivedData/Build/Products/. Remapping: /Users/distiller/project/DerivedData/Build/Products
2022-11-15 10:59:00.920 I  xcpostbuild[5597:475e] (IOS_PROJECT_NAMEUITests) Found url to remapp: file:///Users/distiller/project/IOS_PROJECT_NAME/. Remapping: /Users/distiller/project/IOS_PROJECT_NAME
2022-11-15 10:59:00.920 I  xcpostbuild[5597:475e] (IOS_PROJECT_NAMEUITests) Found url to remapp: file:///Users/distiller/project/DerivedData/Build/Intermediates.noindex/. Remapping: /Users/distiller/project/DerivedData/Build/Intermediates.noindex
2022-11-15 10:59:00.920 I  xcpostbuild[5597:475e] (IOS_PROJECT_NAMEUITests) Found url to remapp: file:///Users/distiller/project/SourcePackages/checkouts/. Remapping: /Users/distiller/project/SourcePackages/checkouts
polac24 commented 1 year ago

The out_of_band_mappings seems to be working correctly on both producer and consumer sides. The next troubleshooting step would be downloading the file https://IOS_PROJECT_NAME-ios-xcode-remote-cache.storage.googleapis.com/meta/65e733e798be3c2e53d837db4f9fd7b20e043c9d-Store-Debug-iphonesimulator-14A400-eccc143bfd3c1cd4f4c0d2842b7b3f1f.json and compare all files listed as dependencies between producer&consumer.

Because the log doesn't mention that a file is missing (that would be a sign of invalid out-of-band remapping) but the fingerprint is invalid so the byte-level contents don't match.

ghost commented 1 year ago

@polac24

I started comparing and found that there are 2 files with the same commit SHA. Which is confusing. Could it be the reason that it's can not match the fingerprints? Why there are two meta files with the same SHA but different fingerprints? In both consumer and producer modes I run xcodebuild build-for-testing where I produce artefcats to run the tests later on. 

65e733e798be3c2e53d837db4f9fd7b20e043c9d-Store-Debug-iphonesimulator-14A400-eccc143bfd3c1cd4f4c0d2842b7b3f1f.json
65e733e798be3c2e53d837db4f9fd7b20e043c9d-IOS_PROJECT_NAMEUITests-Debug-iphonesimulator-14A400-bfd512a3a8ee6ba690a014460dbe617f.json

These files were created relatively at the same time:

image
polac24 commented 1 year ago

@pp-gtroshin That is entirely normal to have different fingerprints for different targets. A unique meta file is generated for each commit/target/platform/version. Because Store and IOS_PROJECT_NAMEUITests targets have different source files (and dependencies), fingerprints are different.