Optimise and improve GitHub Actions workflow

beeware / Python-Apple-support

A meta-package for building a version of Python that can be embedded into a macOS, iOS, tvOS or watchOS project.

MIT License

1.1k stars 159 forks source link

Optimise and improve GitHub Actions workflow #168

Open freakboy3742 opened 1 year ago

freakboy3742 commented 1 year ago

It currently takes almost 90 minutes to do a CI build of the support packages. This is at least partially due to duplicated build effort. We use a simple matrix build to target each build platform (macOS, iOS, tvOS, watchOS); however, building iOS requires a build of macOS, so macOS is effectively built 4 times; this also means there are duplicated maOS builds of bz2, xz and OpenSSL.

Describe the solution you'd like

An optimised GitHub Actions workflow for builds where the macOS build step is a pre-requisite of the iOS/tvOS/watchOS targets.

If we're making changes to GitHub Actions, it would also be desirable if:

The various build, install, and merge logs were attached as artefacts to the build, so that they can be easily accessed afterwards
The build process included a "sanity check" that confirms the count of binary modules is what is expected. This will help avoid problems like #167.

mhsmith commented 1 year ago

It currently takes almost 90 minutes to do a CI build of the support packages

It's even worse than that, because we can only run 5 concurrent macOS jobs per GitHub account, and we're currently spawning 16 (4 Python versions 4 platforms). So even if each job was reduced to 30 minutes, the total build time would still be over 90 minutes (30 minutes 16/5), during which time all macOS jobs in Briefcase and Toga would be blocked, as happened yesterday.

Other ideas:

Since we're not currently using the watchOS and tvOS platforms, they might as well be disabled.
If it's anything like the Android build, OpenSSL is probably the most time-consuming component. Could we build that only once and then share it between all the Python versions?

freakboy3742 commented 1 year ago

Even though we're not using watchOS and tvOS at present, I'm hesitant to disable them; if they're not being built regularly, then the patches that have been written to support those platforms are going to atrophy rapidly.

The dependency builds are definitely fertile ground for optimization, though. BZip2 and XZ only rarely change; they definitely don't need to be built on every CI build. OpenSSL is updated more often - but the OpenSSL build can/should be decoupled from the Python builds (as the release cadence is completely decoupled).

It might be worth exploring how we can use Releases for multiple products - if tagging bzip2-1.0.5 produced a "BZip2 1.0.5" artefact, and a Python build pulls down that artefact as a pre-requisite, we could optimise a lot of the build process. It would also provide an entry point for producing the binary wheel that is needed to support some third-party packages (as libFFI and OpenSSL are pre-requisites of some packages).