wojciech-kulik / xcodebuild.nvim

Neovim plugin to Build, Debug, and Test applications created for Apple devices (iOS, macOS, watchOS, etc.)
MIT License
675 stars 17 forks source link

Slow incremental builds using `xcodebuild` #201

Open wojciech-kulik opened 1 week ago

wojciech-kulik commented 1 week ago

Recently, I discovered that adding CODE_SIGNING_ALLOWED=NO" to xcodebuild commands significantly speeds up incremental builds. However, the problem is that you can't run the app built this way, because obviously it's unsigned.

I suspect that xcodebuild signing process is for some reason very slow, so if we could figure out a way to sign in afterwards, maybe we could speed up builds. Unfortunately, I haven't yet found a way to sign the app.

So far I tried:

codesign --force --deep --sign "Apple Development: John Snow (XYZ)" /Users/wkulik/Library/Developer/Xcode/DerivedData/XYZ-hblnhsksxjrctzekqmlevcflnsji/Build/Products/Debug-iphonesimulator/XYZ.app

and also signing all frameworks and bundles before signing the whole app, but it doesn't work - the app can't be launched.

wojciech-kulik commented 1 week ago

Root Cause

It turns out that the issue is caused by the way the xcodebuild tool handles signing. It performs multiple requests to developerservices2.apple.com API which take over 20 seconds each time you run a build 🤦‍♂️.

The difference between xcodebuild and Xcode is that Xcode runs those requests on startup and stores the responses, while xcodebuild, on the other hand, calls them every time you invoke it.

On top of that, xcodebuild performs sequentially developerservices2.apple.com/services/v1/capabilities request per each target you have, so the more targets you have, the more significant the difference between Xcode and xcodebuild is going to be. The funny thing is that each request is exactly the same, so it's not even necessary to call them all 🙈.

apple2

What's Next

It's a very inefficient behavior, but unfortunately, I don't think it's possible to fix or configure it on our side. We need to wait for a fix provided by Apple, possibly to cache responses and optimize the signing.

I have already created a ticket in Feedback Assistant: FB15478221. You can also follow the thread here. Initially, I assumed the local signing process was slow for some reason, but the discussion in that thread revealed that network requests were responsible for the delay.

However, there is also good news, you can speed up builds by temporarily blocking that domain. In my case, it reduces an incremental build from 30 seconds down to 10 seconds. Below you will find details on how to do that.

[!CAUTION] Blocking that domain will most likely result in disabling some Xcode functions like automatic registering devices, capabilities, etc. Therefore, it's best to block it when you are working on code and you don't need any modifications related to signing or capabilities.

Workaround 1

Enable workaround:

sudo bash -c "echo '127.0.0.1 developerservices2.apple.com' >>/etc/hosts"

Disable workaround:

sudo sed -i '' '/developerservices2\.apple\.com/d' /etc/hosts

Workaround 2

If you use some tool to sniff network traffic like Proxyman or Charles Proxy, you can block requests to https://developerservices2.apple.com/* and automatically return some error like 999 status code. It will prevent xcodebuild from further calls.

Workaround 3 - plugin integration

Now, xcodebuild.nvim plugin provides automatic integration with this workaround. It blocks the Apple server while xcodebuild is running and unblocks it when the process is finished. Here is the instruction on how to enable it: read more. This feature has been introduced in #202.

Still slower than Xcode 🥺

This workaround improves build time significantly, but there is still some overhead introduced by xcodebuild command.

In my case, incremental build using Xcode takes ~4s, and incremental build using xcodebuild takes ~10s.

Based on the response from Apple (#106) and my profiling, it looks like in the case of my project it takes an additional ~0.3s to initialize the tool, ~3s to load the project, and around ~2s of some extra time needed by xcodebuild.

It seems like xcodebuild is lacking some performance improvements like a project cache. I suspect that the tool was designed more for CI/CD & clean builds rather than incremental builds.

If this issue is important to you I recommend creating a ticket in Feedback Assistant and referencing my ticket FB15478221. The more people report this issue, the greater the chance that it will be fixed.

Almaz5200 commented 1 week ago
Screenshot 2024-10-19 at 3 31 46 PM

Yeah, definetly helped me too, but the change is not as drastic. It went from 27 seconds on average to 18 seconds. I see that for you it makes four of each requests, however I only see one of them. Which Xcode version do you use? Could it be that they improved it at a later version? Mine is 16.0

wojciech-kulik commented 1 week ago

I use Xcode 16.0. It looks like it might be connected with the number of targets in the project. I've got the main target + 3 extensions. The funny thing is that each request is exactly the same... very poor optimization.

Update: I tried a project with a single target and there is only one request then.

Update 2: Also requests to developerservices2.apple.com/services/v1/capabilities are not parallelized, so the more targets you have the more significant the difference between Xcode and xcodebuild is going to be.

Almaz5200 commented 1 week ago

Yeah, and that would explain the fact that you've seen more improvement than I did as well. The project I've tested on doesn't have any extensions

rudrankriyam commented 1 week ago

I found it mentioned in the developer forums and the difference is evident when you turn off the internet and see it does not hang on GatherProvisioningInputs anymore

marcusziade commented 1 week ago

can confirm

wojciech-kulik commented 1 week ago

I'm working to integrate an automatic script to disable developerservices2.apple.com domain during build based on @rudrankriyam's code: https://x.com/rudrankriyam/status/1847734299740811675 to make it simpler to use without the need to manually turn on & off the workaround.

wojciech-kulik commented 6 days ago

The script to automate applying the workaround has been integrated with xcodebuild.nvim.

Here is the instruction explaining how to enable this feature: https://github.com/wojciech-kulik/xcodebuild.nvim/wiki/Tips-&-Tricks#%EF%B8%8Fimprove-build-time

Don't forget to update the plugin first :)!

LionWY commented 6 days ago

follow

kambala-decapitator commented 2 hours ago

Unfortunately, I haven't yet found a way to sign the app

Looks like your codesign command is incorrect - you forgot entitlements for the app. Also, don't pass --deep!

Correct sign workflow is:

  1. For watch app and all extensions: first sign all frameworks, then sign the executable (or its bundle) with respective entitlements
  2. Sign all app frameworks
  3. Sign the main app with respective entitlements

You can also check the codesign command that xcodebuild issues in Xcode build log or in the command line invocation (just don't pass -quiet for the latter).