Jekyll build/deploy workflow

[0xdevalias]

Sets up a GitHub actions/workflow to build and deploy the Jekyll site to Github Pages.

• fixes https://github.com/0xdevalias/devalias.net/issues/27

[0xdevalias]

I'm not actually sure what the best method for testing GitHub actions/workflows is without just deploying to prod.. I found one project claiming to be able to:

• https://github.com/nektos/act

But in attempting to run it in dryrun mode, it looked like it would fail:

⇒  act -n
*DRYRUN* [Build and deploy Jekyll site/jekyll] 🚀  Start image=node:12.6-buster-slim
*DRYRUN* [Build and deploy Jekyll site/jekyll]   🐳  docker run image=node:12.6-buster-slim entrypoint=["/usr/bin/tail" "-f" "/dev/null"] cmd=[]
*DRYRUN* [Build and deploy Jekyll site/jekyll] ⭐  Run 📂 checkout
*DRYRUN* [Build and deploy Jekyll site/jekyll]   ✅  Success - 📂 checkout
*DRYRUN* [Build and deploy Jekyll site/jekyll] ⭐  Run 💎 setup ruby
*DRYRUN* [Build and deploy Jekyll site/jekyll]   ☁  git clone 'https://github.com/ruby/setup-ruby' # ref=v1
*DRYRUN* [Build and deploy Jekyll site/jekyll] Unable to resolve v1: reference not found
*DRYRUN* [Build and deploy Jekyll site/jekyll]   ❌  Failure - 💎 setup ruby
Error: reference not found

I expect if we used an exact version tag it would probably work.. but that seems at odds with how other examples seem to imply GitHub works?

Changing ruby/setup-ruby@v1 to ruby/setup-ruby@v1.40.0 gave the following output, implying it should work correctly:

⇒  act -n
*DRYRUN* [Build and deploy Jekyll site/jekyll] 🚀  Start image=node:12.6-buster-slim
*DRYRUN* [Build and deploy Jekyll site/jekyll]   🐳  docker run image=node:12.6-buster-slim entrypoint=["/usr/bin/tail" "-f" "/dev/null"] cmd=[]
*DRYRUN* [Build and deploy Jekyll site/jekyll] ⭐  Run 📂 checkout
*DRYRUN* [Build and deploy Jekyll site/jekyll]   ✅  Success - 📂 checkout
*DRYRUN* [Build and deploy Jekyll site/jekyll] ⭐  Run 💎 setup ruby
*DRYRUN* [Build and deploy Jekyll site/jekyll]   ☁  git clone 'https://github.com/ruby/setup-ruby' # ref=v1.40.0
*DRYRUN* [Build and deploy Jekyll site/jekyll]   ✅  Success - 💎 setup ruby
*DRYRUN* [Build and deploy Jekyll site/jekyll] ⭐  Run 🔨 install dependencies & build site
*DRYRUN* [Build and deploy Jekyll site/jekyll]   ☁  git clone 'https://github.com/limjh16/jekyll-action-ts' # ref=v2
*DRYRUN* [Build and deploy Jekyll site/jekyll]   ✅  Success - 🔨 install dependencies & build site

Yet when we run it properly, we hit an issue based on the docker image that is being used not matching perfectly to how GitHub actions runners are:

⇒  act
[Build and deploy Jekyll site/jekyll] 🚀  Start image=node:12.6-buster-slim
[Build and deploy Jekyll site/jekyll]   🐳  docker run image=node:12.6-buster-slim entrypoint=["/usr/bin/tail" "-f" "/dev/null"] cmd=[]
[Build and deploy Jekyll site/jekyll]   🐳  docker cp src=/Users/devalias/dev/_sites/devalias.net/. dst=/github/workspace
[Build and deploy Jekyll site/jekyll] ⭐  Run 📂 checkout
[Build and deploy Jekyll site/jekyll]   ✅  Success - 📂 checkout
[Build and deploy Jekyll site/jekyll] ⭐  Run 💎 setup ruby
[Build and deploy Jekyll site/jekyll]   ☁  git clone 'https://github.com/ruby/setup-ruby' # ref=v1.40.0
[Build and deploy Jekyll site/jekyll]   🐳  docker cp src=/Users/devalias/.cache/act/ruby-setup-ruby@v1.40.0 dst=/actions/
[Build and deploy Jekyll site/jekyll]   ❗  ::error::ENOENT: no such file or directory, open '/etc/lsb-release'
[Build and deploy Jekyll site/jekyll]   ❌  Failure - 💎 setup ruby
Error: exit with `FAILURE`: 1

So it seems we may not actually have a good way to be able to test this locally..

[0xdevalias]

The build failed since the runner doesn't have the appropriate gsl libraries installed (required to speed up --lsi)

2020-07-26T03:00:40.4975849Z Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
2020-07-26T03:00:40.4976473Z 
2020-07-26T03:00:40.4977046Z current directory:
2020-07-26T03:00:40.4978110Z /home/runner/work/devalias.net/devalias.net/vendor/bundle/ruby/2.7.0/gems/gsl-2.1.0.3/ext/gsl_native
2020-07-26T03:00:40.4978850Z /home/runner/.rubies/ruby-2.7.1/bin/ruby -I
2020-07-26T03:00:40.4979471Z /home/runner/.rubies/ruby-2.7.1/lib/ruby/2.7.0 -r
2020-07-26T03:00:40.4980071Z ./siteconf20200726-2689-qeqmg6.rb extconf.rb
2020-07-26T03:00:40.4980461Z *** ERROR: missing required library to compile this module: No such file or
2020-07-26T03:00:40.4980973Z directory - gsl-config
2020-07-26T03:00:40.4981343Z *** extconf.rb failed ***
2020-07-26T03:00:40.4981669Z Could not create Makefile due to some reason, probably lack of necessary
2020-07-26T03:00:40.4982013Z libraries and/or headers.  Check the mkmf.log file for more details.  You may
2020-07-26T03:00:40.4982324Z need configuration options.
2020-07-26T03:00:40.4982586Z 
2020-07-26T03:00:40.4982892Z Provided configuration options:
2020-07-26T03:00:40.4983385Z    --with-opt-dir
2020-07-26T03:00:40.4983931Z    --without-opt-dir
2020-07-26T03:00:40.4984471Z    --with-opt-include
2020-07-26T03:00:40.4985036Z    --without-opt-include=${opt-dir}/include
2020-07-26T03:00:40.4985572Z    --with-opt-lib
2020-07-26T03:00:40.4986125Z    --without-opt-lib=${opt-dir}/lib
2020-07-26T03:00:40.4986662Z    --with-make-prog
2020-07-26T03:00:40.4987192Z    --without-make-prog
2020-07-26T03:00:40.4987764Z    --srcdir=.
2020-07-26T03:00:40.4988311Z    --curdir
2020-07-26T03:00:40.4988896Z    --ruby=/home/runner/.rubies/ruby-2.7.1/bin/$(RUBY_BASE_NAME)
2020-07-26T03:00:40.4989442Z    --with-gsl-version
2020-07-26T03:00:40.4989740Z 
2020-07-26T03:00:40.4990056Z extconf failed, exit code 1
2020-07-26T03:00:40.4990316Z 
2020-07-26T03:00:40.4990620Z Gem files will remain installed in
2020-07-26T03:00:40.4991186Z /home/runner/work/devalias.net/devalias.net/vendor/bundle/ruby/2.7.0/gems/gsl-2.1.0.3
2020-07-26T03:00:40.4991576Z for inspection.
2020-07-26T03:00:40.4991888Z Results logged to
2020-07-26T03:00:40.4992519Z /home/runner/work/devalias.net/devalias.net/vendor/bundle/ruby/2.7.0/extensions/x86_64-linux/2.7.0/gsl-2.1.0.3/gem_make.out
2020-07-26T03:00:40.4992877Z 
2020-07-26T03:00:40.4993259Z An error occurred while installing gsl (2.1.0.3), and Bundler cannot continue.
2020-07-26T03:00:40.4994249Z Make sure that `gem install gsl -v '2.1.0.3' --source 'https://rubygems.org/'`
2020-07-26T03:00:40.4994949Z succeeds before bundling.
2020-07-26T03:00:40.4995256Z 
2020-07-26T03:00:40.4995583Z In Gemfile:
2020-07-26T03:00:40.4995896Z   gsl

Apparently we can install ubuntu packages as part of the workflow though, so if we install the gsl libraries, this probably should work:

• https://stackoverflow.com/questions/57982945/how-to-apt-get-install-in-a-github-action
• https://askubuntu.com/questions/890027/installing-gsl-libraries-in-ubuntu-16-04-via-terminal

- name: install system dependencies
  run: sudo apt-get install libgsl-dev

[0xdevalias]

This build shows that installing the libgsl-dev system dependency works, but appears to take ~10sec to complete, at least on the first run. If this remains the same on subsequent builds, we may need to consider what the cost/benefit of this extra timing is, and/or alternatives we could use (eg. caching, prebuilt docker container, etc)

---

We also need to consider the install order, so that gsl knows to use nmatrix:

  Post-install message from gsl:

      gsl can be installed with or without narray support. Please install
      narray before and reinstall gsl if it is missing.

      gsl is also now compatible with NMatrix. Please install nmatrix before
      installing gsl.

The rb-gsl installation notes claim that we can forcibly tell it to use nmatrix:

In order to use rb-gsl with NMatrix you must first set the NMATRIX environment variable and then install rb-gsl: gem install nmatrix export NMATRIX=1 gem install rb-gsl

This will compile rb-gsl with NMatrix specific functions.

---

As currently setup, even using --lsi, it looks as though the site only took ~9sec to generate.. would be interesting to see how that changes if we remove gsl and/or nmatrix:

jekyll build
  /home/runner/.rubies/ruby-2.7.1/bin/bundle exec jekyll build -s /home/runner/work/devalias.net/devalias.net/ --lsi
  Configuration file: /home/runner/work/devalias.net/devalias.net/_config.yml
              Source: /home/runner/work/devalias.net/devalias.net
         Destination: /home/runner/work/devalias.net/devalias.net/_site
   Incremental build: disabled. Enable with --incremental
        Generating... 
     GitHub Metadata: No GitHub API authentication could be found. Some fields may be missing or have incorrect data.
         Jekyll Feed: Generating feed for posts
                      done in 8.475 seconds.
   Auto-regeneration: disabled. Use --watch to enable.
  Took   9.13 seconds

To fix the follow warning, we need to set the appropriate environment variables for the Jekyll GitHub Metadata plugin:

GitHub Metadata: No GitHub API authentication could be found. Some fields may be missing or have incorrect data.

• https://github.com/jekyll/github-metadata/blob/master/docs/authentication.md
  • JEKYLL_GITHUB_TOKEN

Can we use ${{ secrets.GITHUB_TOKEN }} for this? (no, see https://github.com/0xdevalias/devalias.net/pull/86#issuecomment-663934375)

[0xdevalias]

Even when we removed nmatrix, our site build still seemed to take only ~9sec.

Unsure if this is because gsl wasn't actually using it before, or if it just has a negligible impact on our site build time.

---

Also, by using the cached gems, rather than having to re-install them, our total site dependencies/build step went from ~4min 51sec down to just ~16sec.

[0xdevalias]

Even when we removed gsl, our site build still seemed to take only ~8sec.

Unsure if this is because gsl just has a negligible impact on our site build time.

[0xdevalias]

Even when we removed Jekyll's --lsi option, the site build still only seemed to take ~7sec.

So maybe --lsi either isn't working, or just doesn't really have a big impact on our site build time.

In light of this.. i'm thinking we can leave the --lsi option enabled for now, but can probably remove the gsl/nmatrix optimisations we had added. Though we should probably do this in a follow up PR (see https://github.com/0xdevalias/devalias.net/issues/87)

This also most likely renders https://github.com/0xdevalias/devalias.net/issues/83 as irrelevant

(Edit: as was pointed out on #83, I may have disabled related_posts on my site a while ago, which is likely why the changes here weren't doing anything. Theres a lot of good info in #83 and linked to from #87 as well.. so worth exploring that more deeply before just removing it all)

[0xdevalias]

The build using secrets.GITHUB_TOKEN for JEKYLL_GITHUB_TOKEN failed with the following error during the site build:

2020-07-26T04:19:36.2417867Z   Liquid Exception: GET https://api.github.com/gists/1395926: 403 - Resource not accessible by integration // See: https://developer.github.com/v3/gists/#get-a-single-gist in /home/runner/work/devalias.net/devalias.net/_posts/2013-06-17-gists-on-tumblr.md
2020-07-26T04:19:36.2418778Z                     ------------------------------------------------
2020-07-26T04:19:36.2419520Z       Jekyll 4.1.1   Please append `--trace` to the `build` command 
2020-07-26T04:19:36.2420234Z                      for any additional information or backtrace. 
2020-07-26T04:19:36.2420916Z                     ------------------------------------------------
2020-07-26T04:19:36.2422515Z /home/runner/work/devalias.net/devalias.net/vendor/bundle/ruby/2.7.0/gems/octokit-4.18.0/lib/octokit/response/raise_error.rb:16:in `on_complete': GET https://api.github.com/gists/1395926: 403 - Resource not accessible by integration // See: https://developer.github.com/v3/gists/#get-a-single-gist (Octokit::Forbidden)

This appears to be caused by jekyll-gist:

• https://github.com/jekyll/jekyll-gist

  • Pro-tip: If you provide a personal access token with Gist scope, as the environmental variable JEKYLL_GITHUB_TOKEN, Jekyll Gist will use the Gist API to speed up site generation.

This same env var will also be used by the jekyll-github-metadata plugin, as per:

• https://github.com/jekyll/github-metadata/blob/master/docs/authentication.md

---

Looking at the token I currently use for this on my local machine, it seems to only have the repo.public_repo scope enabled.

So perhaps creating a similar token, and configuring it as a GitHub Action secret that we then pass to this env var would work:

• https://docs.github.com/en/actions/configuring-and-managing-workflows/creating-and-storing-encrypted-secrets#creating-encrypted-secrets-for-a-repository
• https://github.com/0xdevalias/devalias.net/settings/secrets

eg. ${{ secrets.JEKYLL_GITHUB_TOKEN_PAT }}

Edit: Thoughts from 2023 now that I know more about GitHub actions.. is there any reason why I can't just use the action token for this?

[0xdevalias]

When we added the deploy step to a test branch, the build seemed to complete successfully, taking only ~6sec to deploy to the test gh-pages branch.

---

The commit message was just deploy: 41e8aad, which differs from our previous message style of Update site: Wed 22 Jul 2020 15:19:29 AEST

actions-gh-pages lets us customise this using the commit_message or full_commit_message arguments, though i'm currently unsure how we would generate a similar formatted date to pass to it:

• https://github.com/peaceiris/actions-gh-pages#%EF%B8%8F-set-custom-commit-message

There appear to be some actions related to getting the current time, unsure if there is a simpler way:

• https://github.com/gerred/current-time
  • uses core.setOutput("time", time)
• https://github.com/srfrnk/current-time
  • uses core.setOutput("time", time)
  • https://github.com/1466587594/get-current-time
  • uses core.setOutput("formattedTime", moment().utcOffset(utcOffset).format(format));
• https://github.com/ajilraju/actions-date
  • appears to use coreutil's date command in an alpine docker container

Since most of these just seem to use core.setOutput, we may be able to do similar by using the workflow command echo syntax, and running directly on the host runner:

• https://docs.github.com/en/actions/reference/workflow-commands-for-github-actions
  • echo "::workflow-command parameter1={data},parameter2={data}::{command value}"
• https://docs.github.com/en/actions/reference/workflow-commands-for-github-actions#setting-an-output-parameter
  • echo "::set-output name=action_fruit::strawberry"
• https://docs.github.com/en/actions/creating-actions/metadata-syntax-for-github-actions#outputs

  • The <output_id> must start with a letter or _ and contain only alphanumeric characters, -, or _.

• https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjobs_idoutputs

  • Job outputs are available to all downstream jobs that depend on this job.

• https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstepsrun

  • Runs command-line programs using the operating system's shell.

• https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstepswith

  • A map of the input parameters defined by the action. Each input parameter is a key/value pair. Input parameters are set as environment variables. The variable is prefixed with INPUT_ and converted to upper case.

Maybe something like:

- id: currentdate
  name: Get the current date and export it for later steps to use
  run: echo "::set-output name=date_str::$(date)"
  #run: echo "::set-output name=date_str::`date`"

[0xdevalias]

The build testing whether we can get a formatted date as step output (without using external actions) seemed to work:

      - id: currentdate
        name: 📅 get current date and time
        run: echo "::set-output name=date_str::$(date)"

      - name: 📅 check if date output was set correctly
        run: echo ${{steps.currentdate.outputs.date_str}}

Therefore, we can use them to customise the commit message for the generated site builds as per https://github.com/0xdevalias/devalias.net/pull/86#issuecomment-663938320:

actions-gh-pages lets us customise this using the commit_message or full_commit_message arguments, though i'm currently unsure how we would generate a similar formatted date to pass to it:

• https://github.com/peaceiris/actions-gh-pages#%EF%B8%8F-set-custom-commit-message

I think we want to use something like commit_message: ${{ steps.currentdate.outputs.date_str }}