search

sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.34k stars 452 forks source link

GH Actions (macOS): Run a job for "make build-local" first, cache image for job "make build" #32703

Closed mkoeppe closed 2 years ago

mkoeppe commented 2 years ago

We revise the GH Actions workflows to use a 2-stage build: In the first stage, run make build-local, and store SAGE_LOCAL as a build artifact. In the second stage, download the build artifact and run more building and testing.

(+) On top of the artifact containing the full SAGE_LOCAL, we can test several ways to build the Python parts

(+) Tests of optional and experimental packages can be streamlined, as we avoid rebuilding their dependencies that are standard packages.

(+) Splitting the job into two would also help with the configurations for which we scrape at the 6 hour time limit

(-) Unfortunately, because because "needs" cannot depend on "matrix", the jobs for building/testing Python packages would not start before all jobs building SAGE_LOCAL for all platforms are completed

In this ticket, we only change all existing macos workflows to a 2-stage workflow, integrating the separate workflows for optional and experimental packages.

As of this ticket, we rely on the bottleneck of the available parallel jobs on GH Actions to ensure that the 2nd stages of a configuration are run after the 1st stage of that configuration. Experience with this workflow will show whether this suffices.

We also update the macOS/Xcode versions according to what's available on GH Actions and switch the homebrew builds to faster homebrew-usrlocal variants, which can use bottles for all available packages.

Depends on #32113 Depends on #32947

CC: @tobiasdiez @kliem @orlitzky @isuruf

Component: porting

Author: Matthias Koeppe

Branch/Commit: edb4364

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/32703

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 10de67a to 8617b6f

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

8617b6f.github/workflows/tox.yml: Reduce/streamline macos environments
mkoeppe commented 2 years ago

Changed reviewer from https://github.com/mkoeppe/sage/actions/runs/1364372711 to https://github.com/mkoeppe/sage/actions/runs/1597453181

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

2dee3f0.github/workflows/tox.yml (macos): Remove max-parallel
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 8617b6f to 2dee3f0

mkoeppe commented 2 years ago
comment:38

Looks like homebrew's gcc package no longer installs an unversioned gfortran binary

https://github.com/mkoeppe/sage/runs/4572338375?check_suite_focus=true

mkoeppe commented 2 years ago
comment:39

Also conda install fails early

https://github.com/mkoeppe/sage/runs/4572338435?check_suite_focus=true

mkoeppe commented 2 years ago
comment:40

Replying to @mkoeppe:

Also conda install fails early

https://github.com/mkoeppe/sage/runs/4572338435?check_suite_focus=true

(same as #32113 comment:17)

mkoeppe commented 2 years ago

Dependencies: #32113

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 2dee3f0 to 6418579

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

43d481bbump to 0.1.0
f66821bcysignals are needed
ae37b4edeprecate sage.interfaces.primecount, not just remove
d3ae5f4primecount is on conda, too
3a8c7feprimesieve is on conda, too
74b3845allow float inputs for prime_pi
591be34Merge #32894
049a5f6tox.ini: Do not set environment variable CONDARC
f3116a1tox.ini (conda): Force use of conda's python3
6418579Merge #32113
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 6418579 to 933134c

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

933134cbuild/pkgs/gfortran/distros/homebrew.txt: Use gfortran
mkoeppe commented 2 years ago

Changed reviewer from https://github.com/mkoeppe/sage/actions/runs/1597453181 to https://github.com/mkoeppe/sage/actions/runs/1605145773

mkoeppe commented 2 years ago

Changed dependencies from #32113 to #32113, #32947

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

ab4fb99tox.ini: Add ubuntu-jammy, debian-bookworm, linuxmint-20.3, fedora-36
e51225a.github/workflows/tox*.yml: Remove debian-jessie, add ubuntu-jammy, debian-bookworm, linuxmint-20.3, fedora-36
ac47e7d.github/workflows/tox-gcc_spkg.yml: Remove
1aa9328Merge #32947
e99a09esed -i.bak 's/ubunty/ubuntu/g' .github/workflows/*.yml
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 933134c to e99a09e

tobiasdiez commented 2 years ago
comment:48

I know we already touch upon this point, but can you please expand on

Unfortunately, because ​because "needs" cannot depend on "matrix", the jobs for building/testing Python packages would not start before all jobs building SAGE_LOCAL for all platforms are completed

That is, why do you prefer the "custom stage build" over having a "local-macos-stage2" job that depends via "needs" on "local-macos-stage1", where each of these jobs defines an appropriate matrix. In particular, I don't understand the "all platforms" part, as ubuntu and macos are in different jobs, right? It appears to me that with the current design also stage 1 builds are executed before any stage 2 is executed, so the only difference is that you can already have 4 or so stage 2 builds running while the last stage 1 build finishes. I guess for the overall build time this shouldn't make much of a difference anyway, as these 4 runners would be busy with other workflows then (or are free to execute builds of other stuff in the sagemath org).

mkoeppe commented 2 years ago
comment:49

Replying to @tobiasdiez:

That is, why do you prefer the "custom stage build" over having a "local-macos-stage2" job that depends via "needs" on "local-macos-stage1", where each of these jobs defines an appropriate matrix.

To avoid copy-paste

mkoeppe commented 2 years ago
comment:50

Replying to @tobiasdiez:

I don't understand the "all platforms" part, as ubuntu and macos are in different jobs, right?

Yes, that's right, within each job; and in this ticket, only for macos.

tobiasdiez commented 2 years ago
comment:51

Replying to @mkoeppe:

Replying to @tobiasdiez:

That is, why do you prefer the "custom stage build" over having a "local-macos-stage2" job that depends via "needs" on "local-macos-stage1", where each of these jobs defines an appropriate matrix.

To avoid copy-paste

Github recently made it possible to reuse workflows: https://docs.github.com/en/actions/learn-github-actions/reusing-workflows Thus you could extract the current local-macos job into a new workflow and pass the right "tox" command as an argument (determined by the matrix + step).

mkoeppe commented 2 years ago
comment:52

Thanks for the pointer!! I've added this to #29060 for welcome future refactoring - but this won't make it into Sage 9.5

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from e99a09e to edb4364

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

edb4364.github/workflows/tox.yml: Replace homebrew-macos-python3_xcode-standard by homebrew-macos-usrlocal-python3_xcode-standard
mkoeppe commented 2 years ago

Description changed:

--- 
+++ 
@@ -18,8 +18,6 @@

 As of this ticket, we rely on the bottleneck of the available parallel jobs on GH Actions to ensure that the 2nd stages of a configuration are run after the 1st stage of that configuration. Experience with this workflow will show whether this suffices.

-References
-- https://evilmartians.com/chronicles/build-images-on-github-actions-with-docker-layer-caching
+We also update the macOS/Xcode versions according to what's available on GH Actions and switch the `homebrew` builds to faster `homebrew-usrlocal` variants, which can use bottles for all available packages.

-
dimpase commented 2 years ago

Changed reviewer from https://github.com/mkoeppe/sage/actions/runs/1605145773 to Dima Pasechnik

dimpase commented 2 years ago
comment:55

OK,macOS runs on GH look good.

mkoeppe commented 2 years ago
comment:56

Thanks!

vbraun commented 2 years ago

Changed branch from u/mkoeppe/gh_actionsrun_a_job_formake_build_localfirst__cache_image_for_jobmakebuild to edb4364