Performance of GitHub-hosted macOS runner

[BrightRan]

Associated community ticket: https://github.community/t/macos-performance/125281

The customer setup a matrix in his workflow to run the same job on different OSs (ubuntu-latest and macos-latest). See the workflow here: https://github.com/gitoleg/bap/actions/runs/187608792/workflow

On some steps in the job, the macOS runner takes far more longer time than the Linux runner (even more than 2 times) to run the steps.

Earlier, I also noticed that some users have reported similar problems. Especially some users who need to pay for runners, they complained that the performance of GitHub-hosted macOS runner is slower and they have to pay more. As we have known, the billing per-minute for macOS runner is 10 times that for Linux runner.

[maxim-lobanov]

Hello @gitoleg, Could you please clarify: did you notice any macOS performance issues recently or they always existed? We are working on some infrastructure changes related to macOS so we need to understand if it could be related

[gitoleg]

Hello @maxim-lobanov, We use Github Actions only for 2 months, maybe even less. And I observe the issue in question during all this time.

[gitoleg]

@maxim-lobanov, Maybe it will help somehow. This run was even worse. The whole workflow takes 1 hour on Ubuntu and more than 5 on macOS! https://github.com/gitoleg/bap/actions/runs/188588845

[MaksimZhukov]

Hello @gitoleg ! Sorry for the late reply! It is expected that MacOS performance is lower than Ubuntu (approximately 20 - 30%), but your build time is really longer than expected. Most likely, it is related to known issue with infrastructure but we can't provide ETA right now. We will keep this issue open until it is resolved. Thank you for your patience! We will keep you informed!

[gitoleg]

@MaksimZhukov Thanks!

[sio]

In the past couple of weeks I've also experienced severe performance degradation on macOS runners.

In one project I've had to increase the timeout value for a function that gets called multiple times by the test suite. Previously timeout was set to sensible 60 seconds since the whole suite runs in 40-50 seconds on Ubuntu, but macOS runners weren't keeping up.

In another project I've moved macOS jobs to another CI provider.

[paresy]

I can confirm that MacOS performance is very flaky and not consistent. Sometimes it is quite fast and sometimes it degrades to 1/4 of the the normal performance. This is very unsatisfying especially on large/long builds due to the high costs.

[martinbonnin]

Sometimes our 30min build takes more than 6h. For an exemple: https://github.com/apollographql/apollo-android/runs/1325891021?check_suite_focus=true This happens randomly but seems more frequent recently. Maybe it's a different thing but it's hard to tell. When this happens, I can't get any log to display. Is it possible to see the logs in real time or maybe even ssh into the vm to debug?

[bdruth]

Chiming in ... running xcodebuild locally: 266s, on macOS runner: 921s ... that's almost a 3.5x slowdown, with a 10x charge penalty ... that's a bit ridiculous, no? Local: MBP 16 (2.3 GHz 8-Core Intel Core i9).

[detrin]

I can confirm that MacOS performance is 2x-2.5x slower than ubuntu-latest, when testing Julia package https://github.com/detrin/OpenQuantumSystems.jl

[syoder]

Some anecdata from a few of my projects:

Step	time on macOS	time on Ubunutu
setting up the job	11s-27s	3s
checking out the repo	10s-12s	2s-5s

I can run my test suite locally in about 1.5m, but it takes ~6m on Github. When you throw in all the other setup and teardown steps, it's ~10m. If it really were only 20%-30% slower than Ubuntu instances, that would be great.

[locus313]

I am also seeing very slow performance on the macos runners. The recent runs take about an hour to run: https://github.com/azerothcore/azerothcore-wotlk/actions/runs/449468989 But when testing the build locally on a mac it takes about 12-16 minutes.

I also noticed the queue time is quite long. sometimes the macos builds are queued for 8+ hours.

[kariharju]

Building for Win(self-hosted for signing), Mac and Linux we are seeing build times ~8min for Linux and over 20min for Mac. On top of this Mac builds still quite often fail to some mysterious network errors, which causes the Mac builds to run even more. Add to this the 10x cost aspect of Mac builds in GHA and the end result is that just about all the costs for us come from Mac builds and we need to start looking to self-hosted Mac machines because this is not sustainable.

[neil-119]

8 minutes to build on a 2018 MacBook Pro.

GitHub Runner:

Literally unusable with that $0.08/minute pricing.

[melroy89]

Still an issue..

[jpike88]

From start to finish, my build pipeline takes about 10-15 minutes on a MacBook Pro.

This is GitHub actions:

The logs show nothing abnormal.

We just wasted money and time on a sub-standard service. Why even bother releasing it in it's current form, even as a beta.

[AlenaSviridenko]

Hi folks, we are sorry for inconveniences, but these are the technical limitations of our macOS runners, their hardware specification can be found in the official docs https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources. Improving performance is on our radar, but unfortunately, no ETA for now.

You can track progress in the related roadmap issue https://github.com/github/roadmap/issues/161. Closing this one, please, feel free to contact us in case of any other questions. Thank you.

[paresy]

@AlenaSviridenko Initially this issue was about inconsistent performance for MacOS runners. You can run the same job multiple times and it will take hugely different amount of times to complete. Has this been fixes?

I know that no one should compare GA with his own 8-Core MB Pro and that the roadmap item suggests to throw more powerful machines (=money) on a problem which we cannot predict. I love the option to do so, but i prefer the underlying inconsistency to be fixed.

In my option the initial version of this issue is still valid and should be left open, even if it doesn't look good in the issue list.

[miketimofeev]

@paresy a few details about performance are available here — https://github.com/actions/virtual-environments/issues/2707#issuecomment-782056996 Basically we can revisit this inconsistency later this year when all the migration is finished.

[smorimoto]

Well, I guess this was probably related to the random access performance of the storage, as shown in another issue that @miketimofeev touched on (#2707). GitHub (and many other CI providers) relies on providers that provide dedicated Mac hardware, but sometimes they provide very old hardware. (IIRC: In fact, Bitrise has had similar problems in the past.)

[smorimoto]

@paresy

i prefer the underlying inconsistency to be fixed

I think we already fixed by #2707 in that sense. (But, obviously, this issue is closed for a completely different reason, and it feels strange to me as well.)

[jpike88]

I've jumped to Bitrise for now, whatever they have going on it works well

[smorimoto]

@jpike88 Yes, that's why I said, "in the past" :) Bitrise had a similar problem in the past due to old machines from MacStadium. When I investigated into the network stuff for another issue, there was MacStadium (https://bgp.he.net/AS54112), so I suspected that there might be a similar issue this time.

[neil-119]

I bought a Mac Mini M1 off eBay for $700 and set it up as an on-prem build agent with Azure Pipelines and it works quite well -- build times back down to 8-10 minutes. Pretty sure it should work for GitHub as well with self-hosted runners.

[smorimoto]

.NET doesn't officially support Apple Silicon yet, and that's a far cry from the point of this discussion.

[taylorkline]

@AlenaSviridenko Is this still on the roadmap for Q1 2022 as github/roadmap#161 implies?

[brianpep]

Would definitely love to see some movement on this issue. We are paying 10x the cost of an Ubuntu runner and getting a tenth of the performance. The main issue I have seen every single time is network timeouts. These timeouts can be seen in the live output when a job runs, but they do not appear in the logs once a job has finished. Have jobs that take 2 minutes on Ubuntu, and over an hour on MacOS, with download steps being the biggest difference.

[miketimofeev]

@brianpep what kind of downloads did you mean? Is it code checkout or some binaries download?

[Ziao]

Agreed. The performance of macos-latest is abysmal. A simple yarn install that would take 15 seconds on my local machine (with node_modules and cache removed) takes upwards of 5 minutes on an action. You cannot charge 10x when performance is 0.1 that of other images. For this reason, we are also looking to simply set up our own physical machine. It'll be cheaper in the (not that) long term.

[smorimoto]

If you can accept arm64, I think Scaleway's product is relatively good.

[sneps85]

Same here, CocoaPods install taking over 10 minutes, where local it takes not 1. Please upgrade your infrastructure so that macos images perform in a more realistic way.

[Sytten]

Community link has been deleted, but definitely still an issue. Our rust builds are 3x slower on mac than other platforms... Really not acceptable considering the price charged.

[mekwall]

Why is this closed when it is still an issue? We are forced to use Bitrise for building our iOS app because it takes more than 3 times longer on GitHub-hosted macOS runners. A build that takes 20 minutes on Bitrise takes over an hour on a GitHub-hosted macOS runner. It just isn't feasible or cost-effective. Please reopen and investigate.

[kopax-polyconseil]

Hello, we are stuck trying to run iOS-Simulateur with appium end 2 end test using this runner. Did anyone succeed ?

I am able only using a self hosted runner, any hint can be appreciated!

[VOvchinnikov]

Our pure python build takes ~13...15 minutes to run on ubuntu-latest runner, but 40+ minutes on macos-latest runner - it would be nice if the issue was actually re-opened as not solved.

The run does not use network, it is purely local test run without any network access necessary (addressing the hypothesis, that it's related to downloads). So this impact comes from the really poor performance of the runner itself.

[sidekick-eimantas]

Control (ubuntu-22.04):

Macos 12:

Macos 11:

Why is this closed if this is still not solved?

[mfkrause]

+1. Larger (and probably more expensive) macOS runners won't resolve the underlying problem. There is absolutely no reason for these VMs being up 500% to 1000% slower when they have double the RAM and a full virtual core more than their Linux counterparts. Adding together the pricing and the performance, macOS runners currently run at about 50-100x the pricing of a Linux runner. Definitely not paying this longterm and if you think we're going to pay even more for a larger macOS runner just so we can maybe get to the performance level of a base Ubuntu runner, you must be out of your mind. We'll buy three M2 Pro Minis at that rate, host the runner ourselves and will save money in the process.

[jozefizso]

We also have performance issues with macOS runners.

MacBook Pro Quad-Core: 60 seconds to build our project MacBook Pro M2 Pro: 14 seconds GitHub macOS 12 runner: 9 minutes

With the current pricing model for macOS runners this makes our builds super overpriced.

[ilia-shipitsin]

@jozefizso , can I ask to open an issue on https://support.github.com ?

this particular issue tracker is related to image generation (i.e. software included and so on), we cannot help much with operations

[sidekick-eimantas]

@jozefizso , can I ask to open an issue on https://support.github.com ?

this particular issue tracker is related to image generation (i.e. software included and so on), we cannot help much with operations

I've had an open issue since November of last year regarding this problem

It is clear GitHub are unable to fix this problem and are trying to sweep it under a rug, lest MacOS runners be labeled by the community as unusable

For those with access, and who can afford them - large macos runners do not seem to have this problem

[jozefizso]

@jozefizso , can I ask to open an issue on support.github.com ?

this particular issue tracker is related to image generation (i.e. software included and so on), we cannot help much with operations

Great, we are waiting a year for macOS 13 build machine.

[synth]

This should not be closed. We are also experiencing insanely slow performance of macOS runners. This is one issue among many that result in extremely degraded trust in Github, sadly.

[Ziao]

No official answer or any form of resolution, and the issue is closed? This is really uncool, Github..

[aviramha]

This affects us as well.

[Lessica]

Debugging my first iOS app CI script and fail, over 40 times. 3x slower than my M1 Pro MacBook Pro but $100 gone. Not affordable, at all.