Closed rpardini closed 3 months ago
Thanks @rpardini. Awesome OP. I've got a busy week so probably won't get to it until next weekend.
won't get to it until next weekend.
No problem, meanwhile I keep on working -- I just added a 2nd drop.
build-run-qemu
and run-qemu
to run linuxkit run qemu
Pushed: forced initial no-offset-limit NTP sync via busybox (fixes RaspberryPi & others without an RTC), support (WiP) for rk3588
devices & a fix for DockerHub rate limits being hit pulling linuxkit/*
pkgs during Hook Linuxkit build.
Also, an initial PR for the showcase
chart, which demonstrates much of what is being done here, in the charts
repo: https://github.com/tinkerbell/charts/pull/89
Hey @rpardini. Thanks for this. I'm playing around with it and the x86_64 build works great. I have some non-technical concerns though. It's not clear how to cross compile HookOS for aarch64. Also, it seems like there are some envs involved for different build.sh commands but it isnt clear what all the env options available are and how to use them properly.
As this is a significant change from the status quo and in order for this to land and be maintainable we're going to need to be able to understand this better. Can you provide docs for all functionality? Most users arent going to need to or want to rebuild kernels so docs around all the options for building the final HookOS are the most important to me. Again, thanks for all this work! I think we really needed it and I want to see it land.
I think i found it. /build.sh linuxkit hook-default-arm64
works great! We'll need docs to explain this. Will also need docs around how to customize a kernel.
Also, i see in the RFC.md that you mention actuated
. We have dedicated self hosted GitHub runners for both x86 and aarch64. They can be referenced in github actions via runs-on: [self-hosted, Linux, ARM64]
and runs-on: [self-hosted, Linux, X64]
FYI, just got Hook 5.15 and 6.6 kernels built and booted into them! I'm liking this!
Super thanks for the review!
Can you provide docs for all functionality?
...
/build.sh linuxkit hook-default-arm64
... We'll need docs to explain this. Will also need docs around how to customize a kernel.
Definitely! Docs are a big challenge. I hope to massage the RFC.md into README.md over time. I'm not too sure about the general CLI interface though: it takes environment variables as well as a command and optional flavor/kernel-id. I guess settling on some sane terminology also needed.
To customize a kernel: bash build.sh config-kernel hook-default-arm64
-- it will prepare a shell in Docker context where you can run make menuconfig
-- it outputs some instructions, which evidently must be improved. Do you think we should have a more direct shortcut to make menuconfig && make savedefconfig & cp defconfig ...
so all can done in a single step, too?
They can be referenced in github actions via
runs-on: [self-hosted, Linux, ARM64]
andruns-on: [self-hosted, Linux, X64]
Perfect. I'd like to keep the ability to cross-build on GH Hosted runners, for people without self-hosted runners.
What I propose is adding environment variables like RUNNER_ARM64="self-hosted, Linux, ARM64"
and RUNNER_AMD64="self-hosted, Linux, X64"
-- if found, the gha-matrix
command will use them as values for a new runner:
JSON field. If one or both are not set, it defaults to ubuntu-latest
. Finally, the field is used in the GHA job matrix as runs-on: ${{matrix.runner}}
. Then we can have a conditional step on the gh_org== tinkerbell
to set the value; that way forks default to free runners. wdyt?
Another large drop. I didn't squash this time, so I might have missed some sign-off's. I wrote some docs
Ref the GitHub actions runners: see the same workflow running against an org with self hosted arm64 runners, and another fork with just plain gh-hosted amd64 runners. Ended up with finer-grained control than proposed above.
got Hook 5.15 and 6.6 kernels built and booted into them!
In the last drop I added hook-latest-lts-amd64
and hook-latest-lts-arm64
, using 6.6; the defconfig was simply copied over from 5.10 and olddefconfig
'ed. Works well in QEMU, but has some trouble on nvidia hardware (apparently ref nouveau
/ framebuffer). Could you push your 6.6 defconfig's if you have them?
Hey @rpardini , I believe i have found the issue with Docker not starting with Linuxkit v1.2.0. We need to enable cgroups v2 in the DinD container.
- name: hook-docker
image: "${HOOK_CONTAINER_DOCKER_IMAGE}"
capabilities:
- all
net: host
pid: host
mounts:
- type: cgroup2
options: [ "rw", "nosuid", "noexec", "nodev", "relatime" ]
destination: /sys/fs/cgroup
Hey @rpardini, if you want to fix the DCO check then I think we can merge this and I can PR follow-ups to get the other stuff. I've been using these changes for a few weeks and am quite pleased with it all.
Hey @jacobweinstock -- I've fixed the DCOs, cleaned up a few commit messages, changed the default OCI coordinates & CI params to Tinkerbell org, and rebased onto main.
I've not touched the existing CI workflows, though; feel free to push to my branch if fixes are needed before merge.
Thanks so much for the reviews!
multi-kernel, cross-compiling, bash based Hook & (default+external) kernels build (incl GHA matrix)
Original RFC below; please check commits for the (many) updates done after the first drop. Original RFC kept for reference, below the line.
RFC: multi-kernel, cross-compiling, bash based Hook & (default+external) kernels build (incl GHA matrix)
This is a rewrite of the build system. The produced default artifacts (aarch64/x86_64) should be equivalent, save for an updated 5.10.213+ kernel and arm64 fixes. It's missing, at least, documentation and linters, possibly more, that I removed and intend to rewrite. But since it's a large-ish change, I'd like to collect some feedback before continuing.
Main topics
defconfig
versions, via Kbuild'smake savedefconfig
hook-default-amd64
andhook-default-arm64
kernels are equivalent to the two original.armbian-
prefixed kernels are actually Armbian kernels for more exotic arm64 SBCs, or Armbian's generic UEFI kernels for both arches. Those are very fast to "build" since Armbian publishes their .deb packages in OCI images, and here we just download and massage them into the format required by Linuxkit.hook.yaml
is replaced withhook.template.yaml
which is templated via a limited-var invocation ofenvsubst
; only the kernel image and the arch is actually different per-flavor.skopeo
. Can opt-out/use a fixed version.dtbs.tar.gz
artifact together with the initrd and vmlinuz.Flavors (/kernels)
Hook's own kernels
hook-default-arm64
hook-default-amd64
Armbian kernels
edge
: release candidates or stable but rarely LTS, more aggressive patchingcurrent
: LTS kernels, stable-ish patchingarmbian-bcm2711-current
armbian-meson64-edge
armbian-rockchip64-edge
armbian-uefi-arm64-edge
armbian-uefi-x86-edge
Proof of working-ness?
In my fork:
Future possibilities:
it would be fairly simple to add Debian/Ubuntu kernels as well as Armbian firmware.
Many, many more Armbian kernels could be added, but save for Allwinner and the Rockchip
-rkr
vendor kernel, I think they might be too niche. Users should have an easy time adding it themselves if they need, though.Better support for u-boot's "pxelinux" booting requires changes outside of Hook (namely in Smee/ipxedust) which I'll PR eventually.
Certain arm64 SoCs require changes in iPXE (nap.h) -- same, I'll PR those to ipxedust repo.
All these Hook flavors are used in a "showcase" Helm chart based on stack that I will also PR to the charts repo.
TO-DO
bash build.sh config-kernel <flavor>
& follow instructions to configure kernel; only works for default flavorsbash build.sh build-kernel <flavor>
builds the kernelbash build.sh build <flavor>
builds Hook with that kernelactuated
for native arm64 building? -- https://actuated.dev/blog/arm-ci-cncf-ampereThanks for reading this far. I'm looking forward to your feedback!