Significant increase of compilation speed

[ch1e]

We've been utilizing Tart with Cirrus CLI on our self-hosted TeamCity CI infrastructure and have found it to be a valuable addition. However, we're currently encountering a notable performance degradation in compilation speed compared to building on hardware agents. The issue arises during the execution of xcodebuild (via fastlane gym). While the CPU performance, as measured with Geekbench 6 on the VM, remains almost identical to that of the host machine, the compilation time has increased dramatically. We've observed a nearly twofold increase, from approximately 600 seconds on the host machine to around 1100 seconds inside the Tart VM.

Upon researching potential causes, I came across a discussion (https://github.com/cirruslabs/tart/discussions/420) indicating a possible degradation of around 30%. However, our observed degradation is much higher. Currently, we're unable to use the --dirty mode to potentially speed up IO due to the issue outlined in https://github.com/cirruslabs/tart/issues/567

However, I'd like to inquire if there are any other potential ways to address the performance degradation? I would be extremely grateful for any response or assistance in resolving this issue. Thanks!

[fkorotkov]

Hey @ch1e, could you please tell us a big more about your project. How big is it and how you measure the 1100 seconds? It it the whole cirrus run executing including syncing of the project? Or just time of a script that runs xcodebuild?

PS just a few days ago we ran IO benchmarks and results are very promising especially on small files.

[ch1e]

Hey @fkorotkov, thank you for the reply.

Our project is relatively big, with the main part of the app written in React Native and also plenty of native code. So we consider a 10-minute build time to be more or less acceptable. My measurements are quite simple: currently, we have two ways of running iOS builds - directly on a hardware macOS agent or by executing cirrus run and performing the build in a Tart VM (and we are really looking forward to using just Tart because it is much more convenient to support on several agents). I ran the same revision both ways and inspected the timings. Regarding the Fastlane summary, the main issue seems to be in the gym step; all the other steps are perfectly fine. The Tart steps themselves are also very fast; booting the virtual machine takes only 9 seconds, and syncing the working directory takes 13 seconds. So the total time of 1100 seconds is due to the Fastlane gym step only. I compared it with the same step on a hardware agent, which takes around 600s.

[fkorotkov]

Can I also double check with you that you set Tart's VM resources to match the host CPU/memory. By default Tart VMs will only use 4 cores and 8GB of memory. You can override it by defining cpu and memory fields:

task:
  macos_instance:
    image: ghcr.io/cirruslabs/macos-sonoma-xcode:latest
    cpu: 8
    memory: 16G

[ch1e]

Sure! I have this code in .cirrus.yml

persistent_worker:
  isolation:
    tart:
      image: ${IMAGE}
      user: admin
      password: admin
      cpu: 8
      memory: 16G 

task:

As far as I understand, it does the same job: I checked the configuration of the created vm and everything seems to be correct (8 cores and 16g memory)

[fkorotkov]

Thank you for the confirmation! 2x slowness is indeed very weird. We are not familiar with gym so might have a few noob questions. Does it only build an app or runs tests as well? If tests are also executed do have visibility if both build and test phases are slow or just one?

Also do you use one of our images or you build it yourself? If you have your own images please make sure that Spotlight indexing is disabled and that you increased limit for open files.

[ch1e]

Actually, gym is just a more convenient way to run the xcodebuild command (https://docs.fastlane.tools/actions/gym/), and in our case, we don't run any tests, just build.

Regarding images: we use ones based on your sonoma-base or ventura-base, so they should be set up correctly.

Do you think it may be a good idea to prepare an example project with our basic tooling (React Native, Fastlane), try to measure compilation speed on it, and share it with you to provide more context?

[fkorotkov]

That would be wonderful! You'll be a hero by providing a reproducible project. We tend to use some syntactic tests and https://github.com/devMEremenko/XcodeBenchmark for validations.

[ch1e]

Hey @fkorotkov Finally, I was able to create a test project: XcodeBenchmark. I ran builds on two different machines using two methods (natively and through cirrus cli). Here are the results:

On my machine (M1 Pro, 32GB RAM, macOS 14.4):

• Direct run with build.sh: 130-140s
• Using cirrus run: 160-170s

On our CI agent (oakhost Mac mini M1, 16GB RAM, macOS 13.6):

• Direct run with build.sh: 180-190s
• Using cirrus run: 280-300s

To be honest, I'm unsure how to interpret these numbers. While the performance on my local Macbook seems acceptable, the performance on the oakhost Mac mini appears questionable. I'm not sure if, by increasing compilation speed, we can achieve numbers similar to the ones we are currently experiencing in our project. Do you have any ideas? Alternatively, is there any possibility that we have something wrong with the setup on the oakhost machine?

[fkorotkov]

Is there an option to upgrade to Sonoma on the agent? Could you please also make sure Spotlight indexing is disabled on the host:

sudo mdutil -vvv -d /
sudo mdutil -vvv -d /System/Volumes/Data

[ch1e]

We plan to upgrade the OS version, but I am unable to test the same agent with Sonoma right now. Do you think this could be the reason for the issue? I can also confirm that Spotlight indexing is disabled on the host, as it doesn't seem to make any difference in compilation speed in Tart.

[fkorotkov]

Closing it for now since it seems reproducing on a particular host using an old macOS version.