Experiencing slower builds with Xcode 13.3.1 compared to Xcode 13.2.1 when RC is enabled

[cezarsignori]

Integration setup

XCRemoteCache: Xcode 13.3.1 built master (91505a59d2a3b8db3dc0eef90c6fbdc051141169) HTTP cache server: Local Demo Docker Xcode: 13.2.1 (13C100) and 13.3.1 (13E500a) Manual integration via Buck IgnoreFileSystemDeviceInodeChanges -bool YES EnableSwiftBuildSystemIntegration 0

Problem & Expectation

• The 1st incremental build after a clean build with 100% cache hit (consumer mode) is twice as long with Xcode 13.3.1 than Xcode 13.2.1.
• The clean build time (producer mode) with RC in Xcode 13.3.1 is about 30% higher than with Xcode 13.2.1;
• The clean build time without RC stays the same with Xcode 13.2.1 and Xcode 13.3.1;

The expectation is that build times on RC builds stay virtually the same across both Xcode versions.

It is worth noting clean consumer builds take the same time to build in both Xcode versions. Build times are also the same for 2nd+ incremental builds or re-builds without changes.

Investigation

Using xclogparser to visualize the build log across different Xcode versions with RC in producer mode, we learned that:

• Swift files take longer to compile (up to 4x) in Xcode 13.3.1 (both Swift-only and mixed modules)
• Serial "Merge swiftmodule" step in Xcode 13.2.1 replaced by parallel "Emit swiftmodule" in Xcode 13.3.1

[ahmednafei]

Hi @polac24,

As a follow up on @cezarsignori initial post for this ticket, I just want to highlight that we verified that swift sources compilation time increase in consumer and producer modes is caused by the custom RC swift compiler xcremotecache/xcswiftc

By only disabling the SWIFT_EXEC build setting, swift sources compilation time got back to normal. Although (as expected) all modules swift files are being compiled even in case modules cache hit

[polac24]

Hello @ahmednafei and @cezarsignori, I am not able to reproduce that, here is the sample project: https://github.com/polac24/XCRCPenalty

It builds RxSwift and RxCocoa locally (both Swift targets) with 0% cache hit and for both Xcode 13.4 and 13.2.1, the XCRC penalty is comparable: 15% vs 12%.

Note: I used 13.4 because it is the newest version and ReleaseNote does not indicate any significant changes comparing 13.3.

[cezarsignori]

I just learned that the use of -debug-prefix-map $(SRCROOT)=$(XCRC_FAKE_SRCROOT) is what makes Swift compilation time higher on Xcode 13.3.1 (not to be confused with -fdebug-prefix-map).

In our case to make the difference visible, I clean build with 100% cache hit and then make an incremental build changing a core module. I get 630s without -debug-prefix-map and 1300s with it.

I also was not able to find any performance issues or behavioural changes in xcswiftc across both Xcodes (aside from a few internal swiftc parameters that xcswiftc only forwards).

Strangely, Swift debugging seems to be working just fine without -debug-prefix-map in both pure and mixed modules (-fdebug-prefix-map is still set for all modules).

[polac24]

An interesting finding, maybe you can try adding -Xcc -gmodules right after -debug-prefix-map ..=... to the OTHER_SWIFT_FLAGS?

Some time ago we had to append this redundant flag to get rid of some errors caused by debug-prefix-map (to match generated PCH hash). Maybe that has an effect on performance?

[cezarsignori]

Unfortunately those flags did not help with the issue.

We confirmed that as long as the producer build is performed with debug-prefix-map (about 30% slower with Xcode 13.3.1 and Xcode 13.4), the debugger continues to work just fine for consumer builds that do not have the flag (same build speed as Xcode 13.2.1).

However we do not know yet if that behavior is trustworthy, as we also found out local code changes are not currently reflected in RC builds (see #138). That is regardless of whether debug-prefix-map is specified or not.

[polac24]

debug-prefix-map is required to make the artifacts portable between machines with different absolute paths. I don't think we can get rid of that and +30% penalty - it is a price we have to pay. Of course, unless you have a luxury to force a unique absolute path of a repo - which doesn't sound realistic though.

[ahmednafei]

currently we are enabling the debug-prefix-map for builds with producer mode that runs on CI and paying that +30% penalty there, but disabling that flag for builds with consumer mode that are supposed to be running on local devs machines.

We did a test for consumer builds (with the flag disabled) on different absolute path than producer and looks like debugging is working on consumer side in that case.

So, do you think we are missing something and the flag is also mandatory for consumer builds?

[polac24]

Yes, I think it is mandatory and we cannot do anything about that. Can we close this issue?

[cezarsignori]

Hey! We've been running consumer mode with -fdebug-prefix-map and without -debug-prefix-map for weeks now without issues. I hope it stays that way :P

Once we migrate to a new Xcode version, we can retest and see if that's changed (for better or worse).

As there is nothing to be done in RC side, I think it is fine to close this issue. I do see value in the discussions and findings for the community, as this is not a solved issue (just mitigated). But I guess people can find these learnings in a closed ticket as well.

Thanks!