multi-kernel, cross-compiling, bash based Hook & (default+foreign) kernels build (incl GHA matrix)

[rpardini]

multi-kernel, cross-compiling, bash based Hook & (default+external) kernels build (incl GHA matrix)

Original RFC below; please check commits for the (many) updates done after the first drop. Original RFC kept for reference, below the line.

---

RFC: multi-kernel, cross-compiling, bash based Hook & (default+external) kernels build (incl GHA matrix)

very early stage RFC

This is a rewrite of the build system. The produced default artifacts (aarch64/x86_64) should be equivalent, save for an updated 5.10.213+ kernel and arm64 fixes. It's missing, at least, documentation and linters, possibly more, that I removed and intend to rewrite. But since it's a large-ish change, I'd like to collect some feedback before continuing.

Main topics

• Makefile build replaced with bash. I'm not too proud of the bash, I will definitely clean it up a lot -- it's the concepts I'd need confirmation on.
• Replaces the emulated Alpine kernel build with a Debian based cross-compiling build
  • Much faster building. Emulating x86_64 on arm64 is very slow and vice-versa.
• Replaces kernel .config's with the defconfig versions, via Kbuild's make savedefconfig
• Replaces Git-SHA1-based image versioning ("current revision") with content-based hashing.
  • This way, there's much higher cache reuse, and new versions are pushed only when components actually changed (caveat emptor)
  • Should allow people to develop Hook without having to build a kernel, depending on CI frequency and luck.
• Introduces multiple "flavors" of hook. Instead of restricted to 2 hardcoded flavors (x86_64 and aarch64, built from source), we can now define multiple flavors, each with an ID and version/configure/build methods.
  • the hook-default-amd64 and hook-default-arm64 kernels are equivalent to the two original.
  • the armbian- prefixed kernels are actually Armbian kernels for more exotic arm64 SBCs, or Armbian's generic UEFI kernels for both arches. Those are very fast to "build" since Armbian publishes their .deb packages in OCI images, and here we just download and massage them into the format required by Linuxkit.
• hook.yaml is replaced with hook.template.yaml which is templated via a limited-var invocation of envsubst; only the kernel image and the arch is actually different per-flavor.
• Introduced a distributed GitHub Actions build workflow. The bash build system produces JSON objects that drive the matrix stages:
  • One matrix is per-arch, and builds all the containers whose source is hosted in this repo (bootkit, docker, mdev)
  • Second matrix is per-flavor(/kernel), and builds the kernel
  • Third matrix, depending on the other two, is per-flavor(/kernel), and builds Hook itself (via LinuxKit) and prepares a .tar.gz into GH artifacts
• Auto-updating of the kernel via kernel.org's JSON endpoint (ofc only works for LTS or recent-enough stable kernels). Could opt-out/use a fixed version.
• Auto updating of Armbian kernels via OCI tag listing via skopeo. Can opt-out/use a fixed version.
• DTB-producing Kernel builds (aarch64) produce a dtbs.tar.gz artifact together with the initrd and vmlinuz.

Flavors (/kernels)

Hook's own kernels

ID	Current version	Description
hook-default-arm64	5.10.213	Hook's own aarch64 kernel
hook-default-amd64	5.10.213	Hook's own x86_64 kernel

Armbian kernels

• External kernels, taken from Armbian's OCI repos. Those are "exotic" kernels for certain SoC's.
  • edge: release candidates or stable but rarely LTS, more aggressive patching
  • current: LTS kernels, stable-ish patching

ID	Current version	Description
armbian-bcm2711-current	6.6.22	bcm2711 (Broadcom) current, from RaspberryPi Foundation with many Armbian fixes for CNCF-landscape projects; for the RaspberryPi 3b+/4b/5
armbian-meson64-edge	6.7.10	meson64 (Amlogic) edge Khadas VIM3/3L, Radxa Zero/2, LibreComputer Potatos, and many more
armbian-rockchip64-edge	6.7.10	rockchip64 (Rockchip) edge, for many rk356x/3399 SoCs. Not for rk3588!
armbian-uefi-arm64-edge	6.8.1	Armbian generic edge UEFI kernel
armbian-uefi-x86-edge	6.8.1	Armbian generic edge UEFI kernel

Proof of working-ness?

In my fork:

• https://github.com/rpardini/tinkerbell-hook/actions/runs/8396946361 A full build workflow, with all misses:
  • Default kernels build (in GHA default runners) in 15-21 minutes each
  • Armbian kernels are done in less than 2 minutes each
  • LK containers in 1 minute for x86_64, 4 minutes for arm64 (done under qemu)
  • Most Hook builds in around 1 minute
  • Full build done in <24m.
• https://github.com/rpardini/tinkerbell-hook/actions/runs/8396643610/job/22998447592 A full build run with all cache hits
  • Everything done in <4 minutes
• https://github.com/rpardini/tinkerbell-hook/releases/tag/20240322-2128 artifacts for testing (initramfs+vmlinuz+dtbs artifact for each flavor)
• https://github.com/rpardini?tab=packages&repo_name=tinkerbell-hook OCI packages in ghcr.io (quay.io had a meltdown today)

Future possibilities:

• it would be fairly simple to add Debian/Ubuntu kernels as well as Armbian firmware.

• Many, many more Armbian kernels could be added, but save for Allwinner and the Rockchip -rkr vendor kernel, I think they might be too niche. Users should have an easy time adding it themselves if they need, though.

• Better support for u-boot's "pxelinux" booting requires changes outside of Hook (namely in Smee/ipxedust) which I'll PR eventually.

• Certain arm64 SoCs require changes in iPXE (nap.h) -- same, I'll PR those to ipxedust repo.

• All these Hook flavors are used in a "showcase" Helm chart based on stack that I will also PR to the charts repo.

TO-DO

• Find a better name for "flavor". Naming things is hard. Hats? Swords?
• Update README.md and CONTRIBUTING.md; For now all I have is really
  • bash build.sh config-kernel <flavor> & follow instructions to configure kernel; only works for default flavors
  • bash build.sh build-kernel <flavor> builds the kernel
  • bash build.sh build <flavor> builds Hook with that kernel
• Restore golang & shellcheck linting
• Update to Linuxkit 1.2.0 and new linuxkit pkgs. I tried, but there's some incompatible changes that we need to figure out.
• Consider using actuated for native arm64 building? -- https://actuated.dev/blog/arm-ci-cncf-ampere

---

Thanks for reading this far. I'm looking forward to your feedback!

[chrisdoherty4]

Thanks @rpardini. Awesome OP. I've got a busy week so probably won't get to it until next weekend.

[rpardini]

won't get to it until next weekend.

No problem, meanwhile I keep on working -- I just added a 2nd drop.

2nd drop of rpardini's take on multi-hook

• bump linuxkit/openntpd and remove bind
  • now it works, which allows us to use RTC-defient hardware (eg RPi)
• introduce build-run-qemu and run-qemu to run linuxkit run qemu
  • works on both amd64 & arm64 as long as KVM is available
  • still mostly Debian/Ubuntu host only, needs a Fedora treatment
  • with TINK_SERVER & MAC env variables set: run a complete test
  • without: run just basic Linuxkit services & kernel, enough to test certain things
• refactor a lot of the bash into functions, handle shortcircuits, etc
• better bash logging
• introduce DO_PUSH checking to avoid pushing on local builds
• check for Docker working & having buildx available before starting

[rpardini]

Pushed: forced initial no-offset-limit NTP sync via busybox (fixes RaspberryPi & others without an RTC), support (WiP) for rk3588 devices & a fix for DockerHub rate limits being hit pulling linuxkit/* pkgs during Hook Linuxkit build.

Also, an initial PR for the showcase chart, which demonstrates much of what is being done here, in the charts repo: https://github.com/tinkerbell/charts/pull/89

[jacobweinstock]

Hey @rpardini. Thanks for this. I'm playing around with it and the x86_64 build works great. I have some non-technical concerns though. It's not clear how to cross compile HookOS for aarch64. Also, it seems like there are some envs involved for different build.sh commands but it isnt clear what all the env options available are and how to use them properly.

As this is a significant change from the status quo and in order for this to land and be maintainable we're going to need to be able to understand this better. Can you provide docs for all functionality? Most users arent going to need to or want to rebuild kernels so docs around all the options for building the final HookOS are the most important to me. Again, thanks for all this work! I think we really needed it and I want to see it land.

[jacobweinstock]

I think i found it. /build.sh linuxkit hook-default-arm64 works great! We'll need docs to explain this. Will also need docs around how to customize a kernel.

[jacobweinstock]

Also, i see in the RFC.md that you mention actuated. We have dedicated self hosted GitHub runners for both x86 and aarch64. They can be referenced in github actions via runs-on: [self-hosted, Linux, ARM64] and runs-on: [self-hosted, Linux, X64]

[jacobweinstock]

FYI, just got Hook 5.15 and 6.6 kernels built and booted into them! I'm liking this!

[rpardini]

Super thanks for the review!

Can you provide docs for all functionality?

... /build.sh linuxkit hook-default-arm64 ... We'll need docs to explain this. Will also need docs around how to customize a kernel.

Definitely! Docs are a big challenge. I hope to massage the RFC.md into README.md over time. I'm not too sure about the general CLI interface though: it takes environment variables as well as a command and optional flavor/kernel-id. I guess settling on some sane terminology also needed.

To customize a kernel: bash build.sh config-kernel hook-default-arm64 -- it will prepare a shell in Docker context where you can run make menuconfig -- it outputs some instructions, which evidently must be improved. Do you think we should have a more direct shortcut to make menuconfig && make savedefconfig & cp defconfig ... so all can done in a single step, too?

They can be referenced in github actions via runs-on: [self-hosted, Linux, ARM64] and runs-on: [self-hosted, Linux, X64]

Perfect. I'd like to keep the ability to cross-build on GH Hosted runners, for people without self-hosted runners.

What I propose is adding environment variables like RUNNER_ARM64="self-hosted, Linux, ARM64" and RUNNER_AMD64="self-hosted, Linux, X64" -- if found, the gha-matrix command will use them as values for a new runner: JSON field. If one or both are not set, it defaults to ubuntu-latest. Finally, the field is used in the GHA job matrix as runs-on: ${{matrix.runner}}. Then we can have a conditional step on the gh_org== tinkerbell to set the value; that way forks default to free runners. wdyt?

[rpardini]

Another large drop. I didn't squash this time, so I might have missed some sign-off's. I wrote some docs

Ref the GitHub actions runners: see the same workflow running against an org with self hosted arm64 runners, and another fork with just plain gh-hosted amd64 runners. Ended up with finer-grained control than proposed above.

[rpardini]

got Hook 5.15 and 6.6 kernels built and booted into them!

In the last drop I added hook-latest-lts-amd64 and hook-latest-lts-arm64, using 6.6; the defconfig was simply copied over from 5.10 and olddefconfig'ed. Works well in QEMU, but has some trouble on nvidia hardware (apparently ref nouveau / framebuffer). Could you push your 6.6 defconfig's if you have them?

[jacobweinstock]

Hey @rpardini , I believe i have found the issue with Docker not starting with Linuxkit v1.2.0. We need to enable cgroups v2 in the DinD container.

  - name: hook-docker
    image: "${HOOK_CONTAINER_DOCKER_IMAGE}"
    capabilities:
      - all
    net: host
    pid: host
    mounts:
      - type: cgroup2
        options: [ "rw", "nosuid", "noexec", "nodev", "relatime" ]
        destination: /sys/fs/cgroup

[jacobweinstock]

Hey @rpardini, if you want to fix the DCO check then I think we can merge this and I can PR follow-ups to get the other stuff. I've been using these changes for a few weeks and am quite pleased with it all.

[rpardini]

Hey @jacobweinstock -- I've fixed the DCOs, cleaned up a few commit messages, changed the default OCI coordinates & CI params to Tinkerbell org, and rebased onto main.

I've not touched the existing CI workflows, though; feel free to push to my branch if fixes are needed before merge.

Thanks so much for the reviews!