search

sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.34k stars 452 forks source link

GH Actions (macOS): Run a job for "make build-local" first, cache image for job "make build" #32703

Closed mkoeppe closed 2 years ago

mkoeppe commented 2 years ago

We revise the GH Actions workflows to use a 2-stage build: In the first stage, run make build-local, and store SAGE_LOCAL as a build artifact. In the second stage, download the build artifact and run more building and testing.

(+) On top of the artifact containing the full SAGE_LOCAL, we can test several ways to build the Python parts

(+) Tests of optional and experimental packages can be streamlined, as we avoid rebuilding their dependencies that are standard packages.

(+) Splitting the job into two would also help with the configurations for which we scrape at the 6 hour time limit

(-) Unfortunately, because because "needs" cannot depend on "matrix", the jobs for building/testing Python packages would not start before all jobs building SAGE_LOCAL for all platforms are completed

In this ticket, we only change all existing macos workflows to a 2-stage workflow, integrating the separate workflows for optional and experimental packages.

As of this ticket, we rely on the bottleneck of the available parallel jobs on GH Actions to ensure that the 2nd stages of a configuration are run after the 1st stage of that configuration. Experience with this workflow will show whether this suffices.

We also update the macOS/Xcode versions according to what's available on GH Actions and switch the homebrew builds to faster homebrew-usrlocal variants, which can use bottles for all available packages.

Depends on #32113 Depends on #32947

CC: @tobiasdiez @kliem @orlitzky @isuruf

Component: porting

Author: Matthias Koeppe

Branch/Commit: edb4364

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/32703

mkoeppe commented 2 years ago

Dependencies: #31535

mkoeppe commented 2 years ago

Branch: u/mkoeppe/gh_actionsrun_a_job_formake_build_localfirst__cache_image_for_jobmakebuild

mkoeppe commented 2 years ago

Reviewer: https://github.com/mkoeppe/sage/actions/runs/1350377940

mkoeppe commented 2 years ago

Commit: d08452c

mkoeppe commented 2 years ago

Last 10 new commits:

2101b8cbuild/pkgs/python3/spkg-build.in: Make sure that python finds sqlite3 when determining which extension modules to build
3bbc5d8build/pkgs/python3/spkg-build.in: Set rpath
124b605Merge #32698
bca2141pkgs/sagemath-standard/tox.ini: Use SAGE_VENV or venv symlink to find wheels
bdbab3abuild/make/Makefile.in: Remove SPKG-tox, SPKG-sdist targets
63e47ffMerge #31535
1c40d1c.github/workflows/extract-sage-local.sh: Make script usable on non-cygwin
dee3f41github/workflows/tox.yml: Multi-stage local-macos
bd9fad1.github/workflows/: Remove workflows for testing
d08452cUpdate systems
mkoeppe commented 2 years ago

Changed reviewer from https://github.com/mkoeppe/sage/actions/runs/1350377940 to https://github.com/mkoeppe/sage/actions/runs/1350710395

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

0799c5f.github/workflows/tox.yml: Fixup for macOS tar
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from d08452c to 0799c5f

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

e031622.github/workflows/tox.yml: Fix up upload path
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 0799c5f to e031622

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

a40d706.github/workflows/tox.yml: Do not use /tmp for artifacts to avoid https://github.com/actions/upload-artifact/issues/92
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from e031622 to a40d706

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

dd2bc15.github/workflows/tox.yml: Use env.GITHUB_WORKSPACE
2b46f02.github/workflows/tox.yml: Use sage-local-artifact/ as path
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from a40d706 to 2b46f02

mkoeppe commented 2 years ago

Changed reviewer from https://github.com/mkoeppe/sage/actions/runs/1350710395 to https://github.com/mkoeppe/sage/runs/3935018641?check_suite_focus=true

mkoeppe commented 2 years ago
comment:11

working well now

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 2b46f02 to 4dcd768

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

4dcd768.github/workflows/tox.yml: mkdir
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 4dcd768 to 828463d

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

828463dgithub/workflows/tox.yml: Multi-stage local-macos
tobiasdiez commented 2 years ago
comment:14

If I understand the changes (and the output) correctly, then currently stage 1 and 2 run parallel. For example, some stage 2 runs (e.g https://github.com/mkoeppe/sage/runs/3935018795?check_suite_focus=true) fail since they try to download a non-existing artifact. Maybe something like https://github.com/marketplace/actions/wait-for-check helps.

mkoeppe commented 2 years ago
comment:15

Replying to @tobiasdiez:

If I understand the changes (and the output) correctly, then currently stage 1 and 2 run parallel.

Yes, in theory. But in practice we only get 5 parallel jobs on macOS and we launch many more jobs...

For example, some stage 2 runs (e.g https://github.com/mkoeppe/sage/runs/3935018795?check_suite_focus=true) fail since they try to download a non-existing artifact.

Well, they failed because the previous stage failed.

mkoeppe commented 2 years ago
comment:16

Replying to @tobiasdiez:

Maybe something like https://github.com/marketplace/actions/wait-for-check helps.

Thanks for the pointer, we can look into using something like this

mkoeppe commented 2 years ago
comment:17

Testing with integrated builds for the optional packages now at https://github.com/mkoeppe/sage/actions/runs/1360303527

tobiasdiez commented 2 years ago
comment:18

Replying to @mkoeppe:

Replying to @tobiasdiez:

If I understand the changes (and the output) correctly, then currently stage 1 and 2 run parallel.

Yes, in theory. But in practice we only get 5 parallel jobs on macOS and we launch many more jobs...

Okay, but isn't this a very fragile design? Imagine that you have exactly 5 matrix cases. So all of the stage 1 tasks start at the same time. Now if the last case exists slightly before the other 4, then the stage 2 for the other 4 may be invoked and now fail as their stage 1 is not yet finished. Also this would limit everything to mac, as on linux you have way more runners.

mkoeppe commented 2 years ago
comment:19

Replying to @tobiasdiez:

Okay, but isn't this a very fragile design?

Yes

mkoeppe commented 2 years ago
comment:20

Replying to @tobiasdiez:

on linux you have way more runners.

we also have way more jobs...

tobiasdiez commented 2 years ago
comment:21

Then I would suggest using two jobs that wait using "needs" and accept the disadvantages that go with it.

Unfortunately, because ​because "needs" cannot depend on "matrix", the jobs for building/testing Python packages would not start before all jobs building SAGE_LOCAL for all platforms are completed

I think this is also a more flexible design as the stage two matrix will probably have more options in the future (e.g. editable install, with/without prebuild wheels from pypi, pipenv). In fact, for maximal flexibility do you think it makes sense too split it further into 3 stages:

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 828463d to 9404238

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

313dc82.github/workflows/tox.yml [local-macos]: Run optional packages here
65f3155.github/workflows/: Remove workflows for testing
3aeedb2Update systems
63d0752.github/workflows/tox.yml: Pass SAGE_LOCAL to extract-sage-local.sh
9404238.github/workflows/extract-sage-local.sh: Use short option with touch so it works on macOS; handle var/lib/sage/venv* too
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

d09e275Fixup
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 9404238 to d09e275

mkoeppe commented 2 years ago
comment:24

Yes, in stage 2 we can have all of these variants, next to the runs for the optional packages added in 313dc82

Whether the doctest needs a separate stage remains to be seen; I think having it as part of the stage 2 builds will easily fit into the 6 hour limit in all variants.

mkoeppe commented 2 years ago
comment:25

Optional packages working well now - https://github.com/mkoeppe/sage/actions/runs/1364372711

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from d09e275 to 9940993

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

4ef2baabuild/make/Makefile.in: Add targets SPKG-sdist, SPKG-tox, SPKG-tox-% for script packages
f621e31tox.ini: Add local-conda-environment-src
32f2755pkgs/sagemath-standard/tox.ini: Use SAGE_VENV or venv symlink to find wheels
ea69f60build/make/Makefile.in: Remove SPKG-tox, SPKG-sdist targets
2a9ab2a.github/workflows/extract-sage-local.sh: Make script usable on non-cygwin
e1410degithub/workflows/tox.yml: Multi-stage local-macos
1180c80.github/workflows/tox.yml [local-macos]: Run optional packages here
b28e85d.github/workflows/tox.yml: Pass SAGE_LOCAL to extract-sage-local.sh
9940993.github/workflows/extract-sage-local.sh: Use short option with touch so it works on macOS; handle var/lib/sage/venv* too
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 9940993 to 553dea4

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

553dea4.github/workflows/tox.yml [local-macos]: Run experimental packages here
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

924d4e6.github/workflows/tox.yml [local-macos]: Add tox_packages_factor=maximal
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 553dea4 to 924d4e6

mkoeppe commented 2 years ago

Changed reviewer from https://github.com/mkoeppe/sage/runs/3935018641?check_suite_focus=true to https://github.com/mkoeppe/sage/actions/runs/1364372711

mkoeppe commented 2 years ago

Description changed:

--- 
+++ 
@@ -9,6 +9,10 @@

 (-) Unfortunately, because [because "needs" cannot depend on "matrix"](https://github.community/t/needs-based-on-matrix/132400), the jobs for building/testing Python packages would not start before all jobs building `SAGE_LOCAL` for all platforms are completed

+In this ticket, we change all existing `macos` workflows to a 2-stage workflow, integrating the separate workflows for optional and experimental packages.
+
+As of this ticket, we rely on the bottleneck of the available parallel jobs on GH Actions to ensure that the 2nd stages of a configuration are run after the 1st stage of that configuration. Experience with this workflow will show whether this suffices.
+
 References
 - https://evilmartians.com/chronicles/build-images-on-github-actions-with-docker-layer-caching
mkoeppe commented 2 years ago

Author: Matthias Koeppe

mkoeppe commented 2 years ago

Changed dependencies from #31535 to none

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

ef7e478.github/workflows/extract-sage-local.sh: Make script usable on non-cygwin
d943c03github/workflows/tox.yml: Multi-stage local-macos
5e89943.github/workflows/tox.yml [local-macos]: Run optional packages here
d9d9823.github/workflows/tox.yml: Pass SAGE_LOCAL to extract-sage-local.sh
659d2a8.github/workflows/extract-sage-local.sh: Use short option with touch so it works on macOS; handle var/lib/sage/venv* too
f4bd501.github/workflows/tox.yml [local-macos]: Run experimental packages here
7b15b44.github/workflows/tox.yml [local-macos]: Add tox_packages_factor=maximal
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 924d4e6 to 7b15b44

mkoeppe commented 2 years ago
comment:32

Rebased away from #31535.

mkoeppe commented 2 years ago

Description changed:

--- 
+++ 
@@ -1,15 +1,20 @@
-(+) On top of the built image containing the full SAGE_LOCAL, we can test several ways to build the Python parts
+We revise the GH Actions workflows to use a 2-stage build:
+In the first stage, run `make build-local`, and store `SAGE_LOCAL` as a build artifact. In the second stage, download the build artifact and run more building and testing.
+
+(+) On top of the artifact containing the full SAGE_LOCAL, we can test several ways to build the Python parts
 - Sage distribution, classic
 - Sage distribution with `configure --enable-editable`
 - Sage distribution with `configure --enable-system-site-packages` (#29665)
 - the configurations from pkgs/**sagemath-standard**/tox.ini
 - build/install of modularized distributions such as pkgs/**sage-categories** (#29865), pkgs/**sagemath-standard-no-symbolics** (#32601)

+(+) Tests of optional and experimental packages can be streamlined, as we avoid rebuilding their dependencies that are standard packages.
+
 (+) Splitting the job into two would also help with the configurations for which we scrape at the 6 hour time limit

 (-) Unfortunately, because [because "needs" cannot depend on "matrix"](https://github.community/t/needs-based-on-matrix/132400), the jobs for building/testing Python packages would not start before all jobs building `SAGE_LOCAL` for all platforms are completed

-In this ticket, we change all existing `macos` workflows to a 2-stage workflow, integrating the separate workflows for optional and experimental packages.
+In this ticket, we only change all existing `macos` workflows to a 2-stage workflow, integrating the separate workflows for optional and experimental packages.

 As of this ticket, we rely on the bottleneck of the available parallel jobs on GH Actions to ensure that the 2nd stages of a configuration are run after the 1st stage of that configuration. Experience with this workflow will show whether this suffices.
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 7b15b44 to 10de67a

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

497682f.github/workflows/extract-sage-local.sh: Make script usable on non-cygwin
afbd4fagithub/workflows/tox.yml: Multi-stage local-macos
57884a6.github/workflows/tox.yml [local-macos]: Run optional packages here
49276f2.github/workflows/tox.yml: Pass SAGE_LOCAL to extract-sage-local.sh
c3e2467.github/workflows/extract-sage-local.sh: Use short option with touch so it works on macOS; handle var/lib/sage/venv* too
55c83f5.github/workflows/tox.yml [local-macos]: Run experimental packages here
10de67a.github/workflows/tox.yml [local-macos]: Add tox_packages_factor=maximal