spotify / XCRemoteCache

Other
827 stars 50 forks source link

Inconsistent producer behavior on M1 Max (missing x86_64 or arm64 files or not) #102

Closed cezarsignori closed 2 years ago

cezarsignori commented 2 years ago

My integration setup

[X] Automatic integration using xcprepare integrate ...

Expected/desired behavior

I expect deterministic and reproducible outcome from building my application in producer mode, particularly, I expect a successful build.

Minimal reproduction of the problem with instructions

Step 0. Generate xcworkspace and xcodeproj files via BUCK with ENVs overridden to arm64 only; Step 1. Integrate RC with the application via xcprepare integrate ...; Step 2. Run rm -rf ~/Library/Developer/Xcode/DerivedData and rm -rf ~/Library/Caches/XCRemoteCache; Step 3. Run rm -rf /tmp/cache/cache on sample docker instance on localhost; Step 4. Build the app normally via Xcode; Step 5. Verify producer logs; Step 6. Repeat steps 2 to 5.

I had one completely successful build without any errors in the logs and multiple other builds with randomly occurring errors.

Producer Logs

Postbuild step failed with error (this time with "The file “arm64” couldn’t be opened because there is no such file.") 
  
zipFail
  

Environment

Post build stats

hit_count: 0
miss_count: 79
local_cache_bytes: 0
indexing_hit_count: 0
indexing_miss_count: 0
   

Others

I am on an Apple M1 Max with macOS Monterey 12.2.1 (21D62).

I am testing RC 0.3.7 on small application with 70 dependencies (across my own modules, git submodules and Pods committed to the repo).

Each xcodeproj has an associated manually created .rcinfo file with the following content:

primary_repo: {xcodeproj_repo}
cache_addresses:
- http://localhost:8080/cache
primary_branch: csignori/xcremotecache-proto
out_of_band_mappings: {
 'TOOLS_BUCK-OUT':'/Users/csignori/Projects/iphone/tools/config/buck-out',
 'CORE_MODULES_BUCK-OUT':'/Users/csignori/Projects/iphone/{root_project}/Modules/CoreModules/buck-out',
 'BDUX_BUCK-OUT':'/Users/csignori/Projects/iphone/{root_project}/Modules/Platform/Bdux/buck-out', 
 'MISC_BUCK-OUT':'/Users/csignori/Projects/iphone/{root_project}/Modules/Miscellaneous/buck-out', 
 'PODS_BUCK-OUT':'/Users/csignori/Projects/iphone/Pods/buck-out',
 'SUBMODULES_BUCK-OUT':'/Users/csignori/Projects/iphone/Submodules/buck-out',
 'ROOT_BUCK-OUT':'/Users/csignori/Projects/iphone/buck-out',
 'PODS':'/Users/csignori/Projects/iphone/Pods',
}

After I generate my Xcode files via BUCK, run xcprepare integrate, build the app and check the producer logs I consistently see the following error multiple times (the number of occurrences vary slightly):

The file “x86_64” couldn’t be opened because there is no such file.

Following instructions provided by @polac24 on #92, I added a build step exporting ENVs.

export ARCHS="arm64"
export ARCHS_STANDARD="arm64 x86_64"
export ARCHS_STANDARD_32_64_BIT="arm64 i386 x86_64"
export ARCHS_STANDARD_32_BIT="i386"
export ARCHS_STANDARD_64_BIT="arm64 x86_64"
export ARCHS_STANDARD_INCLUDING_64_BIT="arm64 x86_64"
export ARCHS_UNIVERSAL_IPHONE_OS="arm64 i386 x86_64"
export NATIVE_ARCH="arm64"
export NATIVE_ARCH_32_BIT="arm"
export NATIVE_ARCH_64_BIT="arm64"
export NATIVE_ARCH_ACTUAL="arm64"
export PLATFORM_PREFERRED_ARCH="x86_64"
export VALID_ARCHS="arm64 arm64e i386 x86_64"
export XCRC_PLATFORM_PREFERRED_ARCH="arm64"
export arch="undefined_arch"
export CURRENT_ARCH="undefined_arch"

PLATFORM_PREFERRED_ARCH is indeed x86_64, but also in every other ENV arm64 is 1st in the list (e.g., ARCHS).

I also checked the other ENVs listed in #42 and for me these are their values:

export CURRENT_VARIANT="normal"
export LINK_FILE_LIST_normal_arm64="/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/DMLProto.build/Debug-iphonesimulator/DMLProto_Showdown.build/Objects-normal/arm64/DMLProto_Showdown.LinkFileList"

And LINK_FILE_LIST_normal_x86_64 is not defined, as expected.

From what I read on #33, #39 and #42, I should not be seeing this issue.

So I forced PLATFORM_PREFERRED_ARCH = arm64 to see if the problem would go away and even though the export run script build step shows the arm64 value, the RC log still shows the x86_64 errors.

I then overrode all those ENVs so they don't contain x86_64 at all.

Now, I either get a successful build (rare) or a number these errors:

The file “arm64” couldn’t be opened because there is no such file.

Note, before I had "x86_64" issues and now I most of the times have "arm64" issues.

polac24 commented 2 years ago

Hello! Because XCRemoteCache does not hardcode arm64 nor x86_64 and as you reported none of ENVs contain it, I bet a path to a non-existing directory is reported in the list of dependencies reported by the compiler in .d.

Assuming a target that fails with an error is DMLProto_Showdown and have arm64 does not exist , you can see which dependencies are reported by the compiler. I would verify that all of them exist:

find /Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/DMLProto.build/Debug-iphonesimulator/DMLProto_Showdown.build/Objects-normal/arm64/ -type f -name "*.d" | xargs grep -ohE "[^ ]+\/arm64\/[^ ]+"

For x86_64 error, that would be:

find /Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/DMLProto.build/Debug-iphonesimulator/DMLProto_Showdown.build/Objects-normal/arm64/ -type f -name "*.d" | xargs grep -ohE "[^ ]+\/x86_64\/[^ ]+"
cezarsignori commented 2 years ago

A successful producer build (only 2 zip fail errors) results in 28 matches for the arm64 grep command (.o, ~partial.swiftmodule/swiftdoc/swiftsourceinfo files). All files from the root module. An no matches for the x86_64 command.

A failed producer build with 6 Postbuild step failed with error: Error Domain=NSCocoaErrorDomain Code=260 "The file “arm64” couldn’t be opened because there is no such file." and 3 zip fail errors result in the same 28 matches for the the arm64 grep command and no matches for the x86_64 command.

The 28 matches are listed below:

/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/DMLProto.build/Debug-iphonesimulator/DMLProto_Showdown.build/Objects-normal/arm64/InsuranceScreenBuilder.o
/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/DMLProto.build/Debug-iphonesimulator/DMLProto_Showdown.build/Objects-normal/arm64/InsuranceScreenBuilder~partial.swiftmodule
/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/DMLProto.build/Debug-iphonesimulator/DMLProto_Showdown.build/Objects-normal/arm64/InsuranceScreenBuilder~partial.swiftdoc
/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/DMLProto.build/Debug-iphonesimulator/DMLProto_Showdown.build/Objects-normal/arm64/InsuranceScreenBuilder~partial.swiftsourceinfo
/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/DMLProto.build/Debug-iphonesimulator/DMLProto_Showdown.build/Objects-normal/arm64/PropertyScreenBuilder.o
/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/DMLProto.build/Debug-iphonesimulator/DMLProto_Showdown.build/Objects-normal/arm64/PropertyScreenBuilder~partial.swiftmodule
/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/DMLProto.build/Debug-iphonesimulator/DMLProto_Showdown.build/Objects-normal/arm64/PropertyScreenBuilder~partial.swiftdoc
/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/DMLProto.build/Debug-iphonesimulator/DMLProto_Showdown.build/Objects-normal/arm64/PropertyScreenBuilder~partial.swiftsourceinfo
/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/DMLProto.build/Debug-iphonesimulator/DMLProto_Showdown.build/Objects-normal/arm64/PBDMLClientFactory.o
/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/DMLProto.build/Debug-iphonesimulator/DMLProto_Showdown.build/Objects-normal/arm64/PBDMLClientFactory~partial.swiftmodule
/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/DMLProto.build/Debug-iphonesimulator/DMLProto_Showdown.build/Objects-normal/arm64/PBDMLClientFactory~partial.swiftdoc
/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/DMLProto.build/Debug-iphonesimulator/DMLProto_Showdown.build/Objects-normal/arm64/PBDMLClientFactory~partial.swiftsourceinfo
/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/DMLProto.build/Debug-iphonesimulator/DMLProto_Showdown.build/Objects-normal/arm64/ServiceRegistry.o
/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/DMLProto.build/Debug-iphonesimulator/DMLProto_Showdown.build/Objects-normal/arm64/ServiceRegistry~partial.swiftmodule
/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/DMLProto.build/Debug-iphonesimulator/DMLProto_Showdown.build/Objects-normal/arm64/ServiceRegistry~partial.swiftdoc
/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/DMLProto.build/Debug-iphonesimulator/DMLProto_Showdown.build/Objects-normal/arm64/ServiceRegistry~partial.swiftsourceinfo
/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/DMLProto.build/Debug-iphonesimulator/DMLProto_Showdown.build/Objects-normal/arm64/AppDelegate.o
/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/DMLProto.build/Debug-iphonesimulator/DMLProto_Showdown.build/Objects-normal/arm64/AppDelegate~partial.swiftmodule
/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/DMLProto.build/Debug-iphonesimulator/DMLProto_Showdown.build/Objects-normal/arm64/AppDelegate~partial.swiftdoc
/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/DMLProto.build/Debug-iphonesimulator/DMLProto_Showdown.build/Objects-normal/arm64/AppDelegate~partial.swiftsourceinfo
/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/DMLProto.build/Debug-iphonesimulator/DMLProto_Showdown.build/Objects-normal/arm64/ServiceProvider+DMLClient.o
/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/DMLProto.build/Debug-iphonesimulator/DMLProto_Showdown.build/Objects-normal/arm64/ServiceProvider+DMLClient~partial.swiftmodule
/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/DMLProto.build/Debug-iphonesimulator/DMLProto_Showdown.build/Objects-normal/arm64/ServiceProvider+DMLClient~partial.swiftdoc
/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/DMLProto.build/Debug-iphonesimulator/DMLProto_Showdown.build/Objects-normal/arm64/ServiceProvider+DMLClient~partial.swiftsourceinfo
/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/DMLProto.build/Debug-iphonesimulator/DMLProto_Showdown.build/Objects-normal/arm64/SearchScreenBuilder.o
/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/DMLProto.build/Debug-iphonesimulator/DMLProto_Showdown.build/Objects-normal/arm64/SearchScreenBuilder~partial.swiftmodule
/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/DMLProto.build/Debug-iphonesimulator/DMLProto_Showdown.build/Objects-normal/arm64/SearchScreenBuilder~partial.swiftdoc
/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Build/Intermediates.noindex/DMLProto.build/Debug-iphonesimulator/DMLProto_Showdown.build/Objects-normal/arm64/SearchScreenBuilder~partial.swiftsourceinfo

Now, the 6 targets that failed with the arm64 does not exist error do not show up on those 28 matches.

For example, the OCMock target fails with:

2022-03-09 10:45:30.993 E  xcpostbuild[57264:662681] (OCMock) Postbuild step failed with error: Error Domain=NSCocoaErrorDomain Code=260 "The file “arm64” couldn’t be opened because there is no such file." UserInfo={NSURL=file:///Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Index/Build/Intermediates.noindex/OCMock.build/Debug-iphonesimulator/OCMock.build/Objects-normal/arm64, NSFilePath=/Users/csignori/Library/Developer/Xcode/DerivedData/DMLProto_Showdown-fnyfbyrumovpfnfvglkwmudrshgr/Index/Build/Intermediates.noindex/OCMock.build/Debug-iphonesimulator/OCMock.build/Objects-normal/arm64, NSUnderlyingError=0x126e04b70 {Error Domain=NSPOSIXErrorDomain Code=2 "No such file or directory"}}
cezarsignori commented 2 years ago

I haven't seen this issue anymore (across multiple RC updates and adoption progress on our app). We did have issues with Indexing but setting ARCHS=arm64 ENV solved them. I'm closing this ticket for now. We can re-open if follow-up is needed.