search

0xdevalias / devalias.net

Source for devalias.net
http://www.devalias.net/
48 stars 10 forks source link

Dockerize Jekyll + automate site builds #27

Open 0xdevalias opened 4 years ago

0xdevalias commented 4 years ago

Dockerize the blog + setup automated site build/deploy on code push

Refs

According to https://github.com/github/pages-gem/issues/651, the best recommended way to support Jekyll 4 on GitHub pages currently is to use GitHub actions:

See Also

0xdevalias commented 1 year ago

tl;dr:

👋 !

We just announced official support for building/deploying Pages straight from GitHub Actions: https://github.blog/changelog/2022-07-27-github-pages-custom-github-actions-workflows-beta/

No hacks (no .nojekyll, no CNAME files), no need to push content back to a dedicated branch, etc. It's "streamlined" and while it doesn't look like much on the surface, lots of things have been simplified underneath to make it happen.

We wrote starter workflows for Pages for a few static site generators including one for plain Jekyll. By that I mean, a Gemfile is required and so is a _config.yml file but there are no "vendor locks" (in the name of security): you can depend on the latest version of Jekyll, use all plugins and all the themes you want. We don't meddle with your configuration file either because Actions is a secure sandbox.

I am aware that this is not a drop in replacement for what pages-gem provides today (a default theme, configuration file, etc.). This is also not exactly the objective of this new feature but we have to start somewhere! pages-gem today fills two goals: (a) provide the security requirements for a non-Actions build infrastructure, (b) provide default so sites can be built out of markdown files seamlessly without a user needing to even know what Jekyll is in the first place. I am not sure yet that we need pages-gem to answer (b) in a GitHub Actions world. I am hopefully my team or the community will figure out something soon enough.

This whole feature is in public beta today and I am very excited about it 🥳 This is certainly not perfect and will be iterated on but we want to hear your feedback. Here or on the new community site.

Originally posted by @yoannchaudet in https://github.com/github/pages-gem/issues/651#issuecomment-1197448026

0xdevalias commented 2 weeks ago

Notes for doing a local build + deploy currently

Apparently I already started upgrading things back in ~2020.. which I'd completely forgotten about; so there may be useful notes/snippets of long lost wisdom there:

There's also a newer 'update jekyll' issue here, to bring things up to date again:

bundle install (on ruby 3.1.1)

Looks like we can't use ruby 3.1.1 (an arbitrary 3.x version I had installed) without updating some of the dependencies:

⇒ rbenv local 3.1.1

⇒ bundle install
Bundler 2.3.9 is running, but your lockfile was generated with 2.1.4. Installing Bundler 2.1.4 and restarting using that version.
Fetching gem metadata from https://rubygems.org/.
Fetching bundler 2.1.4
Installing bundler 2.1.4
Calling `DidYouMean::SPELL_CHECKERS.merge!(error_name => spell_checker)' has been deprecated. Please call `DidYouMean.correct_error(error_name, spell_checker)' instead.
Fetching gem metadata from https://rubygems.org/.........
listen-3.2.1 requires ruby version >= 2.2.7, ~> 2.2, which is incompatible with the current version, ruby 3.1.1p18

rbenv install ruby 2.7.1

So let's try with the version in the .ruby-version file:

rbenv install ruby 2.7.1

But that errored during compilation:

..snip..

ossl_pkey_dsa.c:394:11: error: incompatible function pointer types assigning to 'int (*)(DSA *, unsigned char **)' (aka 'int (*)(struct dsa_st *, unsigned char **)') from 'int (const DSA *, unsigned char **)' (aka 'int (const struct dsa_st *, unsigned char **)') [-Wincompatible-function-pointer-types]
        i2d_func = i2d_DSA_PUBKEY;
                 ^ ~~~~~~~~~~~~~~

..snip..

78 warnings and 1 error generated.
make[2]: *** [ossl_pkey_dsa.o] Error 1
make[1]: *** [ext/openssl/all] Error 2
make: *** [build-ext] Error 2
!!! Compiling ruby 2.7.1 failed!

Googling for some bits of that error led me to this:

Which led me to looking back at my build logs and seeing this part:

..snip..

openssl@1.1 is keg-only, which means it was not symlinked into /usr/local,
because this is an alternate version of another formula.

If you need to have openssl@1.1 first in your PATH, run:
  echo 'export PATH="/usr/local/opt/openssl@1.1/bin:$PATH"' >> ~/.zshrc

For compilers to find openssl@1.1 you may need to set:
  export LDFLAGS="-L/usr/local/opt/openssl@1.1/lib"
  export CPPFLAGS="-I/usr/local/opt/openssl@1.1/include"

For pkg-config to find openssl@1.1 you may need to set:
  export PKG_CONFIG_PATH="/usr/local/opt/openssl@1.1/lib/pkgconfig"

..snip..

All those env flags except for PATH were empty in my shell, so I tried this:

export PATH="/usr/local/opt/openssl@1.1/bin:$PATH";
export LDFLAGS="-L/usr/local/opt/openssl@1.1/lib";
export CPPFLAGS="-I/usr/local/opt/openssl@1.1/include";
export PKG_CONFIG_PATH="/usr/local/opt/openssl@1.1/lib/pkgconfig";

Then tried the install again, which succeeded ✅:

⇒ rbenv install ruby 2.7.1
Note: This is just a small wrapper around 'ruby-install' to prefill the 'install-dir' params for rbenv support

It looks like you're trying to install
  flavor  : ruby
  version : 2.7.1

We will install with the following command:
  ruby-install  --install-dir '/Users/devalias/.rbenv/versions/2.7.1' 'ruby' '2.7.1'

Is that ok? [Y/n]: Y

..snip..

>>> Successfully installed ruby 2.7.1 into /Users/devalias/.rbenv/versions/2.7.1

If we wanted to, we could then cleanup the env vars again with something like:

export PATH=${PATH#/usr/local/opt/openssl@1.1/bin:}
unset LDFLAGS
unset CPPFLAGS
unset PKG_CONFIG_PATH

Or just restart our shell to get the defaults back again.

bundle install (on ruby 2.7.1)

Ensuring we are still using ruby 2.7.1 as per the .ruby-version file:

⇒ rbenv local 2.7.1

We try bundle install again:

⇒ bundle install

..snip..

Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /Users/devalias/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/gsl-2.1.0.3/ext/gsl_native
/Users/devalias/.rbenv/versions/2.7.1/bin/ruby -I /Users/devalias/.rbenv/versions/2.7.1/lib/ruby/2.7.0 -r ./siteconf20240620-69761-1fcewq5.rb extconf.rb
*** ERROR: missing required library to compile this module: No such file or directory - gsl-config
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
    --with-opt-dir
    --with-opt-include
    --without-opt-include=${opt-dir}/include
    --with-opt-lib
    --without-opt-lib=${opt-dir}/lib
    --with-make-prog
    --without-make-prog
    --srcdir=.
    --curdir
    --ruby=/Users/devalias/.rbenv/versions/2.7.1/bin/$(RUBY_BASE_NAME)
    --with-gsl-version

extconf failed, exit code 1

Gem files will remain installed in /Users/devalias/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/gsl-2.1.0.3 for inspection.
Results logged to /Users/devalias/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/extensions/x86_64-darwin-23/2.7.0-static/gsl-2.1.0.3/gem_make.out

An error occurred while installing gsl (2.1.0.3), and Bundler cannot continue.
Make sure that `gem install gsl -v '2.1.0.3' --source 'https://rubygems.org/'` succeeds before bundling.

In Gemfile:
  gsl

But get an error while installing gsl:

ERROR: missing required library to compile this module: No such file or directory - gsl-config

Which it seems we have seen/resolved before:

But this time seem to be hitting new issues with it:

These issues might be relevant here:

Since it sounds like we're not even using the site.related_posts feature that --lsi / gsl / etc are meant to be speeding up (Ref); there's probably no harm in just commenting out those gems for now, at least as a temporary hack to get things working:

  # ..snip..
- gem 'nmatrix'
- gem 'gsl' # Note: you need to install a compatible version (eg. 2.1) of gsl first: brew install gsl@2.1
- gem 'classifier-reborn'
+ # gem 'nmatrix'
+ # gem 'gsl' # Note: you need to install a compatible version (eg. 2.1) of gsl first: brew install gsl@2.1
+ # gem 'classifier-reborn'
  # ..snip..

Which then allows us to complete our bundle install:

⇒ bundle install
Fetching gem metadata from https://rubygems.org/.........

..snip..

Bundle complete! 15 Gemfile dependencies, 48 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.

The following PR commits these changes more permanently (at least for now):

Building the site

Now to remember how to build/deploy the site.. which seems to be scattered across a number of half documented places:

In my 'upgrade to Jekyll 4.x' PR, I seem to have changed the suggested commands in PUBLISHING.md for Build/Deploy to use ./bin/build and ./bin/deploy.. so I guess they're probably the main ones I should be using now:

Running bin/build:

⇒ ./bin/build
Configuration file: /Users/devalias/dev/0xdevalias/devalias.net/_config.yml
            Source: /Users/devalias/dev/0xdevalias/devalias.net
       Destination: /Users/devalias/dev/0xdevalias/devalias.net/_site
 Incremental build: disabled. Enable with --incremental
      Generating...
   GitHub Metadata: No GitHub API authentication could be found. Some fields may be missing or have incorrect data.
       Jekyll Feed: Generating feed for posts

Build Process Summary:

| PHASE      |    TIME |
+------------+---------+
| RESET      |  7.8864 |
| READ       |  0.1378 |
| GENERATE   |  0.4316 |
| RENDER     | 22.9618 |
| CLEANUP    |  0.0262 |
| WRITE      |  1.1139 |
+------------+---------+
| TOTAL TIME | 32.5577 |

Site Render Stats:

| Filename                                                                   | Count |     Bytes |   Time |
+----------------------------------------------------------------------------+-------+-----------+--------+
| _layouts/default.html                                                      |   647 |  8684.62K | 16.305 |
| _includes/head/head.html                                                   |   647 |  2574.81K |  1.860 |
| _includes/loop.html                                                        |   537 |  1937.34K |  1.308 |
| _includes/post-meta.html                                                   |   931 |  1061.73K |  0.719 |
| _includes/navigation.html                                                  |   647 |  1009.04K |  0.381 |
| _posts/2013-06-17-gists-on-tumblr.md                                       |     1 |     0.27K |  0.361 |
| _posts/2013-07-13-nmap-sh-saving-precious-seconds.md                       |     1 |     0.52K |  0.358 |
| _posts/2013-08-03-rails-lessons-learned-the-hard-way-1-db-migrate.md       |     1 |     0.48K |  0.353 |
| _posts/2013-08-10-erpscan-automator-because-manual-is-meh.md               |     1 |     0.54K |  0.348 |
| _posts/2014-04-02-hacking-unicoins-for-fun-and-profit.md                   |     1 |     1.03K |  0.347 |
| _posts/2015-04-20-dogedraw-now-with-more-nyaan.md                          |     1 |     0.79K |  0.345 |
| _posts/2014-05-15-java-scala-future-promise-map-headsplode.md              |     1 |     0.50K |  0.344 |
| _posts/2013-11-21-vfeed-wrapper-helper-scripts-for-speed-and-efficiency.md |     1 |     0.91K |  0.328 |
| _posts/2013-08-29-reversing-powershell-securestring-for-fun-and-profit.md  |     1 |     0.86K |  0.320 |
| _layouts/post.html                                                         |    77 |  1159.14K |  0.253 |
| _includes/header.html                                                      |   647 |   563.42K |  0.160 |
| sitemap.xml                                                                |     1 |    47.27K |  0.103 |
| _includes/tag_pagination.html                                              |  1010 |   220.11K |  0.084 |
| feed.xml                                                                   |     1 |   167.83K |  0.075 |
| _layouts/atom.xml                                                          |   494 |  5956.01K |  0.062 |
| _includes/author-social-icons.html                                         |    93 |   261.40K |  0.046 |
| _includes/head/styles.html                                                 |   647 |   236.31K |  0.033 |
| _includes/scripts/analytics.html                                           |   648 |   202.50K |  0.024 |
| tag/hypernova/index.html                                                   |     1 |     4.96K |  0.020 |
| 3/index.html                                                               |     1 |    10.80K |  0.020 |
| tag/bootzooka/index.html                                                   |     1 |     4.96K |  0.016 |
| _includes/social-share.html                                                |    77 |   116.78K |  0.016 |
| tag/greatfet/index.html                                                    |     1 |     7.81K |  0.015 |
| tag/redux/index.html                                                       |     1 |     3.76K |  0.014 |
| tag/cashay/index.html                                                      |     1 |     3.77K |  0.013 |
| tag/debug/index.html                                                       |     1 |     3.32K |  0.012 |
| _layouts/page.html                                                         |    33 |   123.26K |  0.012 |
| 5/index.html                                                               |     1 |    12.06K |  0.012 |
| _includes/author-meta.html                                                 |    93 |    37.62K |  0.011 |
| index.html                                                                 |     1 |    12.13K |  0.011 |
| tag/backend/index.html                                                     |     1 |     4.96K |  0.011 |
| 4/index.html                                                               |     1 |    11.29K |  0.010 |
| tag/dev/index.html                                                         |     1 |    12.71K |  0.010 |
| author/devalias/11/index.html                                              |     1 |    13.84K |  0.010 |
| tag/hack/index.html                                                        |     1 |    10.91K |  0.010 |
| tag/pentest/index.html                                                     |     1 |    10.98K |  0.010 |
| tag/laser/index.html                                                       |     1 |     3.59K |  0.010 |
| tag/akka-http/index.html                                                   |     1 |     7.05K |  0.010 |
| tag/babel/index.html                                                       |     1 |     4.95K |  0.009 |
| tag/coffeescript/index.html                                                |     1 |     4.97K |  0.009 |
| tag/amazon/index.html                                                      |     1 |     3.71K |  0.009 |
| tag/python/index.html                                                      |     1 |     6.58K |  0.009 |
| 9/index.html                                                               |     1 |    10.91K |  0.009 |
| tag/sources.list/index.html                                                |     1 |     3.31K |  0.009 |
| tag/usb/index.html                                                         |     1 |     6.38K |  0.009 |
+----------------------------------------------------------------------------+-------+-----------+--------+
| TOTAL (for 50 files)                                                       |  7263 | 24544.81K | 24.836 |

                    done in 32.599 seconds.
 Auto-regeneration: disabled. Use --watch to enable.

There's a warning that seems to be from jekyll-github-metadata:

GitHub Metadata: No GitHub API authentication could be found. Some fields may be missing or have incorrect data.

It seems we need to configure an access token for it:

Which I have now configured, and added a placeholder for to my dotfiles:

Re-running bin/build:

⇒ ./bin/build
Configuration file: /Users/devalias/dev/0xdevalias/devalias.net/_config.yml
            Source: /Users/devalias/dev/0xdevalias/devalias.net
       Destination: /Users/devalias/dev/0xdevalias/devalias.net/_site
 Incremental build: disabled. Enable with --incremental
      Generating...
       Jekyll Feed: Generating feed for posts

Build Process Summary:

| PHASE      |    TIME |
+------------+---------+
| RESET      |  8.3194 |
| READ       |  0.1158 |
| GENERATE   |  0.4404 |
| RENDER     | 23.9001 |
| CLEANUP    |  0.1186 |
| WRITE      |  0.5646 |
+------------+---------+
| TOTAL TIME | 33.4589 |

Site Render Stats:

| Filename                                                                   | Count |     Bytes |   Time |
+----------------------------------------------------------------------------+-------+-----------+--------+
| _layouts/default.html                                                      |   647 |  8698.90K | 16.015 |
| _includes/head/head.html                                                   |   647 |  2574.81K |  1.592 |
| _includes/loop.html                                                        |   537 |  1936.73K |  1.187 |
| _posts/2013-06-17-gists-on-tumblr.md                                       |     1 |     0.82K |  1.001 |
| _includes/post-meta.html                                                   |   931 |  1061.07K |  0.655 |
| _posts/2014-05-15-java-scala-future-promise-map-headsplode.md              |     1 |     2.37K |  0.514 |
| _posts/2014-04-02-hacking-unicoins-for-fun-and-profit.md                   |     1 |     2.28K |  0.500 |
| _posts/2013-08-10-erpscan-automator-because-manual-is-meh.md               |     1 |     2.90K |  0.479 |
| _posts/2013-11-21-vfeed-wrapper-helper-scripts-for-speed-and-efficiency.md |     1 |     1.65K |  0.475 |
| _posts/2015-04-20-dogedraw-now-with-more-nyaan.md                          |     1 |     4.38K |  0.462 |
| _posts/2013-07-13-nmap-sh-saving-precious-seconds.md                       |     1 |     1.28K |  0.459 |
| _posts/2013-08-03-rails-lessons-learned-the-hard-way-1-db-migrate.md       |     1 |     0.53K |  0.431 |
| _posts/2013-08-29-reversing-powershell-securestring-for-fun-and-profit.md  |     1 |     2.24K |  0.427 |
| _includes/navigation.html                                                  |   647 |  1009.04K |  0.359 |
| _layouts/post.html                                                         |    77 |  1174.03K |  0.254 |
| _includes/header.html                                                      |   647 |   563.42K |  0.153 |
| sitemap.xml                                                                |     1 |    47.27K |  0.078 |
| _includes/tag_pagination.html                                              |  1010 |   220.11K |  0.077 |
| _layouts/atom.xml                                                          |   494 |  6078.07K |  0.061 |
| feed.xml                                                                   |     1 |   167.83K |  0.049 |
| _includes/author-social-icons.html                                         |    93 |   261.40K |  0.048 |
| _includes/head/styles.html                                                 |   647 |   236.31K |  0.032 |
| _includes/scripts/analytics.html                                           |   648 |   202.50K |  0.022 |
| tag/sectalks/rss.xml                                                       |     1 |    21.37K |  0.018 |
| _includes/social-share.html                                                |    77 |   116.78K |  0.015 |
| _layouts/page.html                                                         |    33 |   123.26K |  0.014 |
| 2/index.html                                                               |     1 |    10.69K |  0.014 |
| tag/kali/index.html                                                        |     1 |     5.65K |  0.013 |
| index.html                                                                 |     1 |    12.13K |  0.012 |
| tag/wireshark/index.html                                                   |     1 |     4.75K |  0.012 |
| _includes/author-meta.html                                                 |    93 |    37.62K |  0.011 |
| author/devalias/index.html                                                 |     1 |    15.47K |  0.011 |
| tag/bulletproof/index.html                                                 |     1 |    11.68K |  0.011 |
| tag/npm/index.html                                                         |     1 |     3.30K |  0.011 |
| tag/bash/index.html                                                        |     1 |     6.32K |  0.010 |
| tag/facebook/index.html                                                    |     1 |     6.34K |  0.009 |
| tag/bulletproof/2/index.html                                               |     1 |     8.87K |  0.009 |
| tag/dev/index.html                                                         |     1 |    12.71K |  0.009 |
| author/devalias/3/index.html                                               |     1 |    14.18K |  0.009 |
| _includes/author-image.html                                                |    93 |    18.62K |  0.008 |
| tag/problem/index.html                                                     |     1 |     3.16K |  0.008 |
| tag/bulletproof-quarterly/index.html                                       |     1 |    11.73K |  0.008 |
| tag/rest/index.html                                                        |     1 |     3.76K |  0.008 |
| _includes/head/indie-auth.html                                             |   647 |   171.23K |  0.008 |
| tag/babel/index.html                                                       |     1 |     4.95K |  0.008 |
| tag/apt-get/index.html                                                     |     1 |     5.66K |  0.007 |
| tag/gh-pages/index.html                                                    |     1 |     3.06K |  0.007 |
| tag/security/2/index.html                                                  |     1 |     9.64K |  0.007 |
| author/devalias/5/index.html                                               |     1 |    15.43K |  0.007 |
| tag/relay/index.html                                                       |     1 |     3.76K |  0.007 |
+----------------------------------------------------------------------------+-------+-----------+--------+
| TOTAL (for 50 files)                                                       |  8001 | 24912.03K | 25.601 |

                    done in 33.501 seconds.
 Auto-regeneration: disabled. Use --watch to enable.

That seemed to fix the warning! ✅

Deploying the site

Now that we've built the site, time to deploy it!

As we figured out earlier, bin/deploy is probably what we want to use here:

So it sounds like running bin/deploy will deploy the site we just built with bin/build (that ended up in ./_site) to GitHub pages.. let's find out!

From memory, the way we used to deploy this is that _site was actually a separately cloned git repository mapped to the gh-pages branch of the repo (which was originally made as a disconnected branch so it doesn't share a parent commit with any of the 'source code' for the site); but looking at it currently on this local machine, there doesn't seem to be a separate _site/.git:

⇒ ls -la _site/.git
gls: cannot access '_site/.git': No such file or directory

We can see that we have it .gitignore'd for the main branch:

https://github.com/0xdevalias/devalias.net/blob/58ea83d1a4936fcb1dc7d34eaff2bbb20f9ebdaf/.gitignore#L4-L8

And there is seemingly no .gitignore on the gh-pages branch; which is how we would expect it to be:

https://github.com/0xdevalias/devalias.net/blob/gh-pages/.gitignore

Based on that.. I would kind of expect bin/deploy to fail currently.. or at the very least, to do things we're sort of not expecting, since the git add -A would add to the main repo, not the other clone.. Though since we have all the contents of that folder .gitignore'd, I suspect it may just end up trying to make an empty commit, and then probably failing..

I dug into things a bit, and refactored bin/build and bin/deploy to be more robust to ensuring the _site git branch is cloned/setup properly; as well as not failing if there are no commits to add, etc.

Following those instructions, we got a deploy to work:

Which we can see the deploy run for here:

We did some manual checks, and everything looked good, so were able to finalise that PR/build/deployment: