Set up GitHub Actions for building the book from sources

[barisere]

Closes #20

Setup details

We use GitHub Actions to build a PDF each time a commit is pushed to the master branch. The CI setup consists of three parts:

1. The definition of the container action that provides the binaries needed to build the PDF. This is available in the gh-build-action folder. I created it following the tutorial at Creating a Docker container action.
2. The shell scripts that use those binaries to build the PDF. For now I added a second shell script, make_sofp_pdf_simple.sh, that uses the minimal commands @winitzki provided because make_sofp_pdf.sh reports some errors. We can fix that in this PR.
3. The workflow definition (in .github/workflows/build-pdf.yml) that coordinates the process.
4. After a successful build, the pipeline: 4.1. uploads the PDF as a build artefact that you is named as sofp-$COMMIT_SHA.pdf. This file appears under the summary of the CI run as shown below. . 4.2. Creates a new draft release where you can fill in the tag and details of the release, as shown in the following image. This step has a slight problem: it creates a release on every successful run, which may not be ideal, as I noted here.
5. If a CI run fails, GitHub Actions shows us the logs output by the failed step.

Possible improvements

1. The Docker container action can be placed in a separate repository so that maintaining it will be easier. That will allow us update it without triggering a rebuild of the book. That will also allow us version it so that each version of the book uses the dependencies it needs, thereby making builds reproducible.
2. We can create releases only when a new tag is pushed. That will prevent the spurious draft releases.

[winitzki]

Hi @barisere , I have pushed (to master branch) some changes in the bash scripts so that it should run on Linux. Could you please try running the latest make_sofp_pdf.sh and see if there are any errors?

Some points for you to verify:

• The build also needs tar and bzip2, I think. Are they installed in your docker image?
• The build script make_sofp_pdf.sh calls other bash scripts check_and_make_draft.sh, check-consistent-labels.sh, and spelling_check.sh. I hope this is okay.
• The script check_and_make_draft.sh produces sofp-draft.pdf, which should also be uploaded to the release along with sofp.pdf.

Please let me know if there are any problems.

Other than these changes, the PR looks good to me.

[barisere]

@winitzki It seems there's an error in the make_sofp_pdf.sh script. Running it deletes the source files that are required to build the book, so the build fails. To verify that the files were being deleted, I ran the build on a read-only filesystem and saved the output to a text file. I've attached the file in this message. Please check it out.

PS: sorry this has taken so long, I would have finished already. There's an ongoing protest against police brutality and corruption in my country, and I've been shuttling between protesting and working. I'll try to finish before next Sunday.

sofp_ci_run.txt

[winitzki]

Hi @barisere ,

I think the error in the bash script is not that it deletes source files (that would prevent it from running also on my mac) but that it did not change into the sofp-src subdirectory before executing commands. I pushed a fix to the master branch, please update your branch and try again. Thank you for your help!

[barisere]

@winitzki I have made the following updates to the CI workflow.

1. The CI build step runs on every push to master.
2. A new draft release is created only if a new tag is pushed.
3. sofp-draft.pdf and sofp.pdf are added to the release.

It feels hacky to me, but it works as expected. I pushed a new tag to my fork and here's the release it created.

The entire CI run took 6 minutes. How is this so far?

[winitzki]

@barisere Looks good! Is this PR ready to be merged or were you planning on any further work within this PR?

[barisere]

@winitzki This is ready to be merged. I think any further work should be informed by your experience with the setup. I planned to migrate the Docker container to the GitHub registry under your account, but that's not urgent.

[winitzki]

@barisere Thank you very much for your work on this PR. I really appreciate your contribution.

What do I need to do in order to get the docker container? I have a Docker account with username winitzki but I've never used it apart from some testing 4 years ago.

[barisere]

That's great, @winitzki. You can setup Docker Hub to build the image whenever you push to the repository. That's how my current setup works. Let's do that in a separate PR. That'll clean things up some more. I propose we move the Dockerfile and the GitHub Action definition to a separate repository so that Docker Hub builds won't get triggered whenever you push changes to the SOFP book repository. I'm out now, I'll provide more information later.

[barisere]

@winitzki I'm glad that I could contribute. Thanks for making SOFP, it's a treasure.

[winitzki]

@barisere I created a new repository https://github.com/winitzki/sofp-docker-build and copied Dockerfile there. Please let me know how to proceed, or open further PRs against this and the other repository. Thank you!

[barisere]

@winitzki Here is how we can proceed.

1. Copy the rest of the files (entrypoint.sh and action.yml) in the gh-build-action directory to the new repository (I've opened a PR to do that).
2. Configure Docker Hub to build a new Docker image whenever you push to https://github.com/winitzki/sofp-docker-build.
3. Change the following Docker image references in .github/workflows/build.yml to point to the image from your Docker Hub repository. 3.1 L20 3.2 L35. This location will change if #24 gets merged, so grep for every occurence of "feastingmonk/sofp-builder:latest".

Here's a video that explains how to achieve 2 and 3.

I notice that the CI runs failed after you removed make_sofp_pdf_simple.sh. #24 removes the lingering reference to that file from the CI configuration. After that is merged, the CI runs will create new draft releases when you push a new tag. Let's make that the primary workflow for releasing new versions and delete the PDF files from the repo source (they bloat the repo. Cloning this repo transfers a lot of data, so I worked on these from a virtual machine on GCP).

[winitzki]

@barisere Thank you for the instructions and the PRs.

I have merged your PRs and made a Docker Hub repository as you described in the video. Once the new docker image is built, I will try running the CI build using the new docker image.

You mentioned that we should remove the PDF files. Yes, they do bloat the repository. But there is one remaining issue, which is that the PDF files built on my Mac are not formatted the same way as the PDF files built by the docker image. On my Mac, I am using pdfTeX 3.14159265-2.6-1.40.20 (TeX Live 2019) while your Docker image installs Debian's TeX Live, which is likely to be an older version. When I upgraded TeX Live on my Mac, it upgraded some LaTeX class files, and the page formatting changed. As a result, the PDF files built and uploaded to github by CI have several pages more than PDF files built on my Mac. I would like to avoid this. Do you know how we could install a different version of TeX Live, or perhaps start from a different base image instead of debian:buster? For example, use ubuntu:focal instead?

https://www.tug.org/texlive/debian.html says ubuntu focal supports TeX Live 2019. Ubuntu Docker images are listed here: https://hub.docker.com/_/ubuntu

I will try that and let you know if it fails. I'm building the new Docker image now.

[winitzki]

@barisere Another trick I use is to checkout repositories with git clone --depth=1 so that only the latest commit is checked out and not the entire commit history with all previous versions of PDF files. This kind of "shallow" checkout may not allow you to make PRs but it is much faster.

[barisere]

@winitzki Great, let's see if Ubuntu will give us better results. Debian packages move slow.

[winitzki]

@barisere Everything works (but the PDF files are still not the same as on my Mac; I can fix that myself, or just leave it like that, it's not too important to have exactly the same page formatting).

Thank you very much for your help!

[barisere]

Hi @winitzki. How's your experience with the CI setup been? Do you see any possibility for improvement? Also, you can enable discussions for this repository to let readers discuss the book there. Things that aren't really issues (feedback, for example), can go there too.

[winitzki]

Hi @barisere I am so far happy with the results, and I appreciate your help.

The new PDF files are being uploaded to "releases", and I am going to add some more artifacts soon (such as, a generated book cover as PDF).

One thing that might be useful is to build PRs automatically after each commit (but not upload any artifacts). Maybe you know how to achieve this.

I'm not sure how else to improve the process.

Thank you for suggesting "discussions", I didn't know about this new feature. I have enabled discussions. Soon I will merge a big PR and put a link to discussions in the readme. I don't expect a lot of activity in this repository, though.

[barisere]

One thing that might be useful is to build PRs automatically after each commit (but not upload any artifacts). Maybe you know how to achieve this.

I can look into that. I noticed that you're opening PRs against master to trigger a build. Sorry for the late reply, I've been on a tight deadline as the current project I'm working on is approaching its launch date. I'm thrilled that they'll launch anyway. Bear with me a little, I'll find time for this.

[winitzki]

Thank you Jonathan, there's no rush. Let me know how I can help. I'll probably finish working on the current PR in a week or so, and it is not necessary to have CI runs for the current PR. But I just thought it would be good to have CI runs for PRs later.