This PR revamps the multi-kernel test setup in line with concerns recently
brought up by obs-profiling, extends it to support arm64 tests, and sets the
whole thing up to run in CI via GitHub actions.
Background
Concerns were brought up by obs-profiling about the initial iteration of the
testing framework locking them into something that wasn't adequate with regards
to cross platform compatibility.
Specifically, obs-profiling wanted the core of the framework to be very easy to
run on different architectures generally (be it generating a test initramfs for
x86 on an arm64 Mac or running a test in an arm64 guest on an x86 host).
Practically speaking, this means writing the core components of the framework
in Go, as it's super easy to cross compile for different architectures and
platforms.
While these concerns were less of a problem for AWP, obs-profiling still wanted
to be able to share common logic pertaining to running BPF tests with us as to
follow the DRY principle. The concern was that either obs-profiling would have
to create something else from scratch, thus performing redundant work, or use a
framework that wasn't quite what they were looking for.
Upon discussion with @florianl, the common logic that both teams will need is
really just limited to initramfs generation and test binary running (i.e. the
init process in the VM). Anything beyond that (e.g. specifics of the tests,
driver scripts to invoke QEMU) becomes pretty team specific.
As a result, we decided that both teams can just go install his
Bluebox project to provide this core
functionality of initramfs generation and a VM init process. Bluebox is written
in Go and easily compiles to run on arm64/x86 hosts or generate initramfs's for
arm64/x86 guests. Anything beyond what Bluebox provides is team-specific and
can be implemented by the team writing the tests.
This will maximise the amount of shared code between our testing setups (which
is good from a DRY standpoint) while also allowing each team to build what
works best for them on top of Bluebox.
This PR provides the AWP-side realiziation of that work, and also sets up the
whole thing to run in CI via GitHub actions.
Changes Made
All initramfs generation logic has been moved into Bluebox. Thus the init/
directory and parts of run_tests.sh that were responsible for invoking cpio
to generate an initramfs have been removed. That job is now done by Bluebox.
The make container make target has been revamped to build a container that
can cross-compile the repo for x86_64 and arm64. See
docker/Dockerfile.builder. Building the repo for different architectures is
now as simple as make ARCH=x86_64 or make ARCH=aarch64. Debug builds are
analgous, the target name is just build-debug.
The build_mainline_kernels.sh script has been updated to build all arm64 and
x86_64 kernel images we need, and output kernel binaries with debug info, so
kernel code can be stepped through with QEMU in case of a gnarly BPF error (see
the debugging section in the updated testing/README.md). These binaries are
stored in the GCS bucket under the debug_binaries directory of each
architecture directory.
CI
As mentioned, a GitHub actions workflow has been added to run tests on multiple
kernels. It pulls kernel images from a GCS bucket and runs tests against them.
For this initial iteration, only mainline kernels (built with the
build_mainline_kernels.sh script) are run. Distro-specific kernels can be
easily added to the test matrix later via an extra line in the yaml.
Kernel images are cached via the cache action and only re-downloaded from GCS
if the containing directory changes to save bandwith and GCS costs.
It takes ~45s per kernel to run the kernel tests in a GitHub actions runner.
x86_64 and arm64 tests are run in parallel, and each encompass 8 kernels
currently (v5.11 - v5.18), so it takes ~6m to run them all. Unfortunately the
GitHub actions runners don't support KVM, which would drastically speed this
up. In the future, we can look at a self-hosted runner with KVM support if we
want to be really aggressive about testing on many, many kernels.
Whether tests pass or fail, a summary of the run and individual results are
archived in GitHub actions (see the summary page). results-mainline-{arch}
contains the console output for the tests run on each kernel and
run-summary-{arch}.txt contains a summary of the pass/fail status of each
test on that particular architecture.
Bugs
A handful of bugs were discovered and fixed to get the tests running on all
kernels (see commits). ~~You'll note only one test is still failing --
TestFileRename on v5.11/aarch64. I'm opening this PR to get feedback now
while I continue to try to fix that, as fixing it has become a bit of a time
sink.~~ (Thank you @mmat11 for helping me squash the last bug 🚀 )
This PR revamps the multi-kernel test setup in line with concerns recently brought up by obs-profiling, extends it to support arm64 tests, and sets the whole thing up to run in CI via GitHub actions.
Background
Concerns were brought up by obs-profiling about the initial iteration of the testing framework locking them into something that wasn't adequate with regards to cross platform compatibility.
Specifically, obs-profiling wanted the core of the framework to be very easy to run on different architectures generally (be it generating a test initramfs for x86 on an arm64 Mac or running a test in an arm64 guest on an x86 host). Practically speaking, this means writing the core components of the framework in Go, as it's super easy to cross compile for different architectures and platforms.
While these concerns were less of a problem for AWP, obs-profiling still wanted to be able to share common logic pertaining to running BPF tests with us as to follow the DRY principle. The concern was that either obs-profiling would have to create something else from scratch, thus performing redundant work, or use a framework that wasn't quite what they were looking for.
Upon discussion with @florianl, the common logic that both teams will need is really just limited to initramfs generation and test binary running (i.e. the init process in the VM). Anything beyond that (e.g. specifics of the tests, driver scripts to invoke QEMU) becomes pretty team specific.
As a result, we decided that both teams can just
go installhis Bluebox project to provide this core functionality of initramfs generation and a VM init process. Bluebox is written in Go and easily compiles to run on arm64/x86 hosts or generate initramfs's for arm64/x86 guests. Anything beyond what Bluebox provides is team-specific and can be implemented by the team writing the tests.This will maximise the amount of shared code between our testing setups (which is good from a DRY standpoint) while also allowing each team to build what works best for them on top of Bluebox.
This PR provides the AWP-side realiziation of that work, and also sets up the whole thing to run in CI via GitHub actions.
Changes Made
All initramfs generation logic has been moved into Bluebox. Thus the
init/directory and parts ofrun_tests.shthat were responsible for invokingcpioto generate an initramfs have been removed. That job is now done by Bluebox.The
make containermake target has been revamped to build a container that can cross-compile the repo for x86_64 and arm64. Seedocker/Dockerfile.builder. Building the repo for different architectures is now as simple asmake ARCH=x86_64ormake ARCH=aarch64. Debug builds are analgous, the target name is justbuild-debug.The
build_mainline_kernels.shscript has been updated to build all arm64 and x86_64 kernel images we need, and output kernel binaries with debug info, so kernel code can be stepped through with QEMU in case of a gnarly BPF error (see the debugging section in the updatedtesting/README.md). These binaries are stored in the GCS bucket under thedebug_binariesdirectory of each architecture directory.CI
As mentioned, a GitHub actions workflow has been added to run tests on multiple kernels. It pulls kernel images from a GCS bucket and runs tests against them.
For this initial iteration, only mainline kernels (built with the
build_mainline_kernels.shscript) are run. Distro-specific kernels can be easily added to the test matrix later via an extra line in the yaml.Kernel images are cached via the cache action and only re-downloaded from GCS if the containing directory changes to save bandwith and GCS costs.
It takes ~45s per kernel to run the kernel tests in a GitHub actions runner. x86_64 and arm64 tests are run in parallel, and each encompass 8 kernels currently (v5.11 - v5.18), so it takes ~6m to run them all. Unfortunately the GitHub actions runners don't support KVM, which would drastically speed this up. In the future, we can look at a self-hosted runner with KVM support if we want to be really aggressive about testing on many, many kernels.
Whether tests pass or fail, a summary of the run and individual results are archived in GitHub actions (see the summary page).
results-mainline-{arch}contains the console output for the tests run on each kernel andrun-summary-{arch}.txtcontains a summary of the pass/fail status of each test on that particular architecture.Bugs
A handful of bugs were discovered and fixed to get the tests running on all kernels (see commits). ~~You'll note only one test is still failing --
TestFileRenameon v5.11/aarch64. I'm opening this PR to get feedback now while I continue to try to fix that, as fixing it has become a bit of a time sink.~~ (Thank you @mmat11 for helping me squash the last bug 🚀 )CC: @norrietaylor @SeanHeelan @m-sample