Kernel Version Testing Framework CI implementation

[FedeDP]

This issue will serve as a tracker for the implementation of the "Kernel Version Testing" framework for the Falco libs. The framework aims to test the compatibility of the drivers (kmod, bpf, and the modern bpf) across different kernels and distributions by running scap-open and drivers_test functions. The testing environment will leverage Firecracker microVMs with OCI artifacts for storing kernels and images. The entire testing process will be orchestrated using an Ansible playbook, and the output will be a Markdown matrix detailing the test results.

It is, for all intents and purposes, an implementation of the detailed proposal merged ~ a week ago: https://github.com/falcosecurity/libs/blob/master/proposals/20230530-driver-kernel-testing-framework.md

Tasks

1. Set up Firecracker microVMs:

   • [x] Research and evaluate the suitability of Firecracker for the testing requirements.
   • [x] Develop a strategy for managing microVM creation, configuration, and cleanup within the testing framework.

2. Implement OCI artifact storage:

   • [x] Identify a suitable repository or storage solution for storing kernel versions and images as OCI artifacts (eg: falcosecurity/drivers_testing_{kernel,rootfs})
   • [x] Define the process for retrieving the required artifacts during the testing workflow.

3. Design and implement the Ansible playbook:

   • [x] Define the necessary Ansible roles and tasks to automate microVM provisioning and configuration.
   • [x] Integrate the playbook with the OCI artifact retrieval process to ensure consistent environment setup.
   • [x] Implement logic to capture test results and generate detailed reports.

4. Generate the Markdown matrix:

   • [x] Design a structured format for the Markdown matrix, including kernel version, distribution, and test results.
   • [x] Develop a script or tool to generate the Markdown matrix based on the captured test results.

5. Integration with CI/CD pipeline:

   • [x] metal EC2 node is needed to run Firecracker vms
   • [x] Self-hosted github actions runner on the ec2 node
   • [x] Master CI will have a workflow to run the kernel version testing framework

First Impl (ETA end of July)

Given all of the above, first implementation should be considered done when:

• [x] We have a CI running tests for x86
• [x] The tests will only run scap-open to validate the drivers build and bpf verifiers
• [x] We have at least an arm64 entry in the matrix (even if untested because we have no arm64 metal nodes) that we were able to test locally, so that we know that the framework is able to work on arm64 nodes too
• [x] We cover ~15/20 kernels from various distros (see https://github.com/falcosecurity/libs/blob/master/proposals/20230530-driver-kernel-testing-framework.md#appendix-1)
• [x] We upload the output matrix as github action artifact

Second iteration

• [x] Add a gh pages to show the result matrix
• [x] Add full support for arm64 (ie15-20 additional images to be tested)
• [ ] Switch to drivers_test executable instead of scap-open to also verify drivers correct behavior

Ideas

• [ ] Automatically fetch needed info (kernel images, modules and so on) from kernel-crawler
• [ ] Automatically build input test matrix (ie: list of images to be tested) given weekly kernel-crawler output (ie: add eg: 1 image per each crawled distro each week, enlarging our input test matrix)

Additional Notes

Please use this issue to track the progress of individual tasks related to the implementation of the Kernel Version Testing framework. Update the task checkboxes as you make progress, and provide relevant details, code snippets, or documentation in the comments.

Feel free to ask any questions or seek clarification regarding the tasks!

[FedeDP]

Note: points 1-3 are partially solved by https://github.com/alacuku/e2e-falco-tests, a PoC repo created for this very same reason. It just needs minor updates and changes (as of now, the repo expects the drivers_test binary to be copied to microvms; instead, we need to clone and build scap-open and drivers inside the vms).

So, in the near future (ie: this week and the next one) the aim is to provide new images to be tested and implement the new Ansible tasks to build drivers and scap-open on each of them. Once we reach:

• images parity with the proposal, meaning a matrix that is big enough
• microvms that are capable of building all the artifacts and running the tests

We will need to:

• manage matrix output in a meaningful way (read: build the matrix output)
• build the CI around all of this

[FedeDP]

See the donation proposal for @alacuku repo: https://github.com/falcosecurity/evolution/issues/282

[FedeDP]

Status update: we were able to test on both x86 and aarch64 metal nodes the current https://github.com/alacuku/e2e-falco-tests implementation; it works fine. @therealbobo is working on adding more and more images to our matrix grid + has forked ignite repo to add a small patch: https://github.com/therealbobo/ignite; @alacuku has worked on improving its e2e-falco-tests repo, by improving the main playbook making it much more customizable and implementing a scap-open playbook that builds drivers + scap-open and tests all 3 drivers with scap-open, see: https://github.com/alacuku/e2e-falco-tests/blob/main/scap-open-test.yml and its role: https://github.com/alacuku/e2e-falco-tests/blob/main/roles/scap-open-test/tasks/main.yml.

With this, what we still need is to add a github action self-hosted runner on our nodes, and build the CI. Finally, we also need to improve the output matrix UX :)

[leogr]

/assign @FedeDP :heart:

[FedeDP]

Hey since friday, we have github pages support for the kernel test matrix :) It looks awesome IMHO! See: https://alacuku.github.io/e2e-falco-tests/

I am going to move all the CI + gh pages stuff to libs push: master very soon, even today hopefully; so that when we finally donate the alacuku/e2e-falco-tests repository to falcosecurity, we have already everything in place and we only need a couple of small fixes (namely: change the repository name being cloned in CI actions/checkout!).

[FedeDP]

I will move remaining tasks to a new tracking issue!

[FedeDP]

See https://github.com/falcosecurity/libs/issues/1224